After upgrade from 4.8 to 5.0 RAW files from Fujifilm GFX 100 / 100S cameras are all purple after import.

[tin-]

Describe the bug

I've been using previous release 4.8 to process RAF files from Fujifilm GFX and it worked as expected.
After fresh installation of darktable 5.0 (Windows 10 PC) and import of GFX files images show up normal in the lighttable library, but after import to darkroom for editing they turn heavy purple with all default settings. Applying color calibration or playing with colorspace does not make it any workable. Turning OpenCL on or off does not change the result. I've tried files from the two cameras - GFX 100 and GFX 100S, and results are identical.

Same issue was reported here as well:
https://discuss.pixls.us/t/purple-images-with-fuji-gfx-in-dt-5-0/47164

Steps to reproduce

1. Use RAW file from GFX, such as one listed in https://discuss.pixls.us/t/purple-images-with-fuji-gfx-in-dt-5-0/47164
2. Import to darktable 5.0 (windows 10, didn't test on other OS)
3. Open file in darkroom. Image will be heavy purple.

Expected behavior

Should open with proper colors, as the old 4.8 does.

Logfile | Screenshot | Screencast

No response

Commit

No response

Where did you obtain darktable from?

downloaded from www.darktable.org

darktable version

5.0.0

What OS are you using?

Windows

What is the version of your OS?

Windows 10 1709 (16299.125)

Describe your system?

Intel Core i7-7980XE, 96GB memory, EVGA GeForce 1080 Ti

Are you using OpenCL GPU in darktable?

Yes

If yes, what is the GPU card and driver?

No response

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

No response

[jenshannoschwalm]

@kmilos did you encounter this before? white points wrong? bayer pattern? RGGB / BGGR ?

[kmilos]

I guess this could be related and a side effect of changing the rawspeed crop to handle #17861

Sadly, not much testing was done as RPU is down.

I thought whatever crop rawspeed supplies, the CFA pattern is automatically adjusted in demosaic...

[jenshannoschwalm]

Right. So the initial value could be bad or it's the black points.

[kmilos]

Unfortunately I won't be able to for a while but these need to be checked since after the change the offsets are probably no longer zero:

https://github.com/darktable-org/rawspeed/blob/d9c9d35b8856ff779a3ac0c50bd96fccaadb1c21/src/librawspeed/common/RawImage.cpp#L193-L194

(Although even offsets shouldn't really affect Bayer CFA...)

[jenshannoschwalm]

No chance to get me into rawspeed code and indeed that should not be the problem. Also it seems it's the same issue :-)

[robgrub]

I got the same problem, using MacOS Sequoia 15.1.1

[jenshannoschwalm]

@kmilos I checked brute-force using libraw for all raf files. works for this camera. For some reason completely failing on other raf files ...

[SteveLamont]

I'm experiencing the same issue with RAF images which loaded in a previous (4.6.1) version.
For what it's worth, RawTherapee loads the images fine.
If you'd like sample images from a GFX100S, please let me know and I will provide.

[jenshannoschwalm]

@ralfbrown @kmilos i tried to use libraw on all RAF images and was somewhat surprised. The image in question was good, others had a failure like the one mentioned in this issue. We also changed the loader mechanism, any bell ringing?

[ralfbrown]

The only thing the changed loader mechanism might be doing is switching whether rawspeed or LibRaw gets used for an image. There have been cases where different firmware versions result in different data being recorded, and perhaps rawspeed and LibRaw disagree on which way to interprest things....

I just checked, and 5.0.0 and master both show purple for GFX 100 and GFX 100 II samples from RPU (my local copy) and are fine for the other GFX models. I clearly didn't see that when making the camera styles back in June or so. BTW, 5.0.0 is using rawspeed as loader for both purple images. Interestingly, when I tried swapping the CFA colors for the GFX 100 in cameras.xml (because it crops an odd number of sensels on top and bottom), darktable told me the file was corrupt as a result of rawspeed throwing an "Unexpected Bayer phase" exception....

[jenshannoschwalm]

Yes, I found that too.

[kmilos]

because it crops an odd number

Looks like this was a dormant bug that's now exposed by this new odd crop only... Likely there is some double shifting/readjustment of the CFA pattern (perhaps the contract is not clear, and happens once in RawSpeed, and another in dt). Unfortunately, I'm still days, if not weeks away from a debugging session...

P.S. exiftool -a -u -s -G1 should print all the active sensor and vendor crop sizes for modern RAFs, that RawSpeed tries to parse as well...

[kmilos]

@jenshannoschwalm It's additionally confusing as the (manual) odd crop for 5D Mark Ii should be working (I think you did a commit to address this in HDR merge, but demosaic was already working ok for that odd crop on single frames...).

[jenshannoschwalm]

@kmilos @LebedevRI i don't know the rawspeed internals and it's working at all, but i can confirm:

1. Adding the removed <Crop x="0" y="4" width="-146" height="-6"/> lines bring back correct colors.
2. I can't find a problem with odd/even iop/rawspeed croppers, that has and is working correct
3. I would have expected defining it as a BGGR pattern would solve the issue but as @ralfbrown has already reported above this results in an "Unexpected Bayer phase"

[kmilos]

Ah, it looks like the adjusted CFA pattern gets reset here after the crop (in applyCorrections() above):

https://github.com/darktable-org/rawspeed/blob/d9c9d35b8856ff779a3ac0c50bd96fccaadb1c21/src/librawspeed/decoders/RafDecoder.cpp#L345

Obviously, there is no effect if the crop is even/zero.

Edit: please test darktable-org/rawspeed#770

[dllu]

I can confirm that darktable-org/rawspeed#770 fixes it. Thanks @kmilos !

[jenshannoschwalm]

Good to me too :-) Checked for (new) failures on all raf sample files i have ...

[sengebusch]

f.y.i.: same problem for Fuji GFX 100 ii.

I was not able to try the patch yet as I am using binaries usually.

[kmilos]

Possibly the alternative workaround w/o rebuilding until the fix lands is to try

1. Quit dt; expose the raw black/white point module crop editing controls (set plugins/darkroom/rawprepare/allow_editing_crop to true in your darktablerc file); start dt
2. Make the raw black/white point module crop even, e.g. change the top/bottom from 7/11 to 6/12

[jenshannoschwalm]

@kmilos did you try that and found it working? For me it didn't and I also think it should not as we always can freely change the raw croppers and the pattern is changed accordingly.

[kmilos]

did you try that and found it working? For me it didn't

Apologies, I guess you're right, it would indeed come in too late, i.e. after RawSpeed has already scrambled the pattern... So it must be done in cameras.xml then: instead of the legacy 4.8 values, adding <Crop x="8" y="6" width="-152" height="-12"/> would be the closest to vendor's intent, followed by a reset of the module in dt.

[sengebusch]

in dt 4.8 I found <Crop x="0" y="0" width="-146" height="-6"/> in cameras.xml for GFX 100xx cameras.
This lines seems to be missing in dt 5.0 for GFX at all. Do you know, why they were removed?

I downgraded dt on my working horse from 5.0 to 4.8 using a back up,
but I will install a fresh system in parallel with dt 5.0 this evening and test.

[kmilos]

Do you know, why they were removed?

Already mentioned earlier: #18073 (comment) (and highlighted in the 5.0.0 release notes)

[jenshannoschwalm]

@kmilos still hoping for your proper rawspeed fix getting merged for 5.0.1

[kmilos]

Sure, we just need a ping from @TurboGit for the freeze date (hoping to potentially catch up w/ a couple of more camera models until then)...

[SteveLamont]

On 1/16/25 06:12, sengebusch wrote: in dt 4.8 I found |<Crop x="0" y="0" width="-146" height="-6"/>| in | cameras.xml| for GFX 100xx cameras.

I can confirm that adding that line to cameras.xml, for the GFX 100S, at least, produces what appears to my eye to be a properly rendered image. Perhaps it is as simple as that? Regards. spl

[kmilos]

Perhaps it is as simple as that?

That is just a temporary workaround for GFX 100 series users shooting uncropped MF only and wishing to continue editing in 5.0.0. This is not an ideal long-term fix as it doesn't work properly if one switches to a FF crop mode on GFX bodies (as mentioned above). A proper fix not requiring any intervention should hopefully land in the 5.0.1 patch release.

[TurboGit]

@kmilos : For bug-fix release there is no freeze date. We push safe bug-fix in darktable 5.0.x branch until we find the status ok for publishing it. So if you have a fix for 5.0.x do it now.

[sengebusch]

I used <Crop x="0" y="0" width="-146" height="-6"/> in cameras.xml for my GFX 100 II compressed mode
and reprocessed some images with dt 5.0. The result is the same as with dt 4.8, so this works for now.

F.y.i.:
I also reprocessed RAW images for Fuji X-H2, Nikon D5300, Nikon D850 and Sony RX 100VA with dt 5.0
and all results are also the same as with dt 4.8, so rawspeed seems to work fine for this cameras
without changes.

[kmilos]

We push safe bug-fix in darktable 5.0.x branch until we find the status ok for publishing it.

I'd rather do a single PR for the rawspeed submodule update, thanks.

[paperdigits]

RPU is back up, btw.

[kmilos]

Implemented in #18242

[CritLoren]

Would be nice if this kind of continuously updated file was decoupled from the main app, as to not require waiting for a full stable update, using a snapshot, or setting up a temporary fix to get around the issue

[paperdigits]

I feel like decoupling the file would be extremely confusing, and the application already sows enough confusion as it is.

…

On February 9, 2025 9:58:42 AM PST, CritLoren ***@***.***> wrote: Would be nice if this kind of continuously updated file was decoupled from the main app, as to not require waiting for a full stable update, using a snapshot, or setting up a temporary fix to get around the issue -- Reply to this email directly or view it on GitHub: #18073 (comment) You are receiving this because you commented. Message ID: ***@***.***>

[LebedevRI]

The cameras.xml is, technically, decoupled, but

1. this wouldn't have helped here since a code change was needed too
2. i'm not sure how well this update works, because then the actual decoder version would need to be tracked per-camera entry, and that isn't being done.