Crashes when using aov integrator with specfilm

[ewan-schafer]

Summary

Trying to use the AOV integrator with specfilm causes SIGABRT / SIGBUS / SIGSEV.

System configuration

System information:
OS: 14.4.1
CPU: b'Apple M2 Max'
Python: 3.11.7 (main, Dec 15 2023, 12:09:56) [Clang 14.0.6 ]
LLVM: 17.0.3

Dr.Jit: 0.4.4
Mitsuba: 3.5.0
Is custom build? False
Compiled with: AppleClang 14.0.0.14000029
Variants:
scalar_rgb
scalar_spectral
llvm_ad_rgb

Description

I have been trying to use the scalar_spectral variant of mitsuba3 to render an image with custom spectral channels and also a depth layer.

Using the AOV integrator with specfilm fails with one of the following exit codes: SIGABRT / SIGBUS / SIGSEV (it isn't always the same one).

Using hdrfilm does not cause the issue.

The documentation for specfilm implies that OpenEXR format is used under the hood, so I assumed that the behaviour would be the same.

As an alternative, I've been able to get the result that I want by doing two separate renders with different integrators and films. But I couldn't see anything in the docs to suggest that my original approach wasn't intended usage.

Steps to reproduce

Below is a minimal example that reproduces the issue on my machine:

import mitsuba as mi

mi.set_variant('scalar_spectral')

scene = mi.load_dict({
    'type': 'scene',
    'integrator': {
        'type': 'aov',
        'aovs': 'd:depth',
        'image': {'type': 'path'},
        },
    'emitter': {
        'type': 'directional',
        'direction': [-1, 0, 0],
        'irradiance': {
            'type': 'regular',
            'wavelength_min': 300,
            'wavelength_max': 1500,
            'values': '1.0, 1.0'
        }
    },
    'target': {'type': 'sphere'},
    'sensor': {
        'type': 'orthographic',
        'to_world': mi.ScalarTransform4f.look_at(
            origin=[0, 2, 0],
            target=[0, 0, 0],
            up=[0, 0, 1]
        ),
        'film': {
            'type': 'specfilm',
            'width': 256,
            'height': 256,
            'ch1': {
                'type': 'regular',
                'wavelength_min': 500,
                'wavelength_max': 600,
                'values': '1.0, 1.0'
            }
        }
    }
})

result = mi.render(scene, spp=128)

[merlinND]

Hello @ewan-schafer,

I can reproduce on my machine but I couldn't find the root cause of the issue from a quick look.
It triggers when combining the depth AOV with specfilm. Switching to 'aovs': 'd:depth,a:albedo', the crash no longer occurs.

@mcrescas, do you know what might be going wrong with depth AOV + specfilm?

[merlinND]

Another observation: if you add more channels to the specfilm, the crash no longer occurs (>= 3 channels).

Could it be that the sub-integrators' develop step each uses the same sensor's film, which in the case of the specfilm, may not have enough channels? → segfault / memory corruption.

mitsuba3/src/integrators/aov.cpp

Lines 363 to 367 in 19bf5a4

	std::vector<TensorXf> inner_images;
	for (auto& integrator : m_integrators) {
	auto image = integrator->render(scene, sensor, seed, spp, develop, evaluate);
	inner_images.push_back(image);
	}

Or the other way around, the AOV integrator always assumes there are 4 (RGBA) "base" channels, whereas the specfilm may have fewer.

mitsuba3/src/integrators/aov.cpp

Lines 118 to 124 in 19bf5a4

	m_integrators.push_back(integrator);
	m_aov_types.push_back(Type::IntegratorRGBA);
	m_aov_names.push_back(kv.first + ".R");
	m_aov_names.push_back(kv.first + ".G");
	m_aov_names.push_back(kv.first + ".B");
	m_aov_names.push_back(kv.first + ".A");
	m_integrator_aovs_count+= 4;

(I'm not familiar with the details, these are just guesses).

[mcrescas]

Hi 👋 ,

It seems that the problem is what @merlinND mentioned in his last message. AOV integrator expects that the integrators passed to it output RGBA images (in spectral mode, that means that the spectrum computed will be transformed to that space). If you are lucky enough, the number of channels will match and no segfault will happen.

The fix would be to change the logic to avoid assumptions about the data output of the spectral integrators (which we do in the other render functions). We will fix it.

In the mean time, I would recommend you to instantiate the scene one time using mi.load_dict(...) and then instantiate the aov integrator alone (with only aov channels, no subintegrators) with another call to aov_integrator = mi.load_dict({'type', 'aov', ....}). After that, you can callmi.render(...) to generate the specfilm image and then with mi.render(... , integrator=aov_integrator) to obtain the aovs.

Hope it helps