Panasonic LX7 garbage pixels

[kofa73]

Describe the bug/issue
Some LX7 images have garbage pixels on the edge. The affected files, when processed using dcraw, have no garbage data.

To Reproduce

1. Download the files from https://tech.kovacs-telekes.org/files/2023-04-01-lx7-garbage-pixels/
2. Import and open them in the darkroom.
3. Check for garbage on the right

Expected behavior
No garbage pixels.

Screenshots
2023-03-30_17-06-32_P1000043.RW2
No garbage pixels - size shown in image information: 3792 (3920) x 2536:

Camera's JPG: 3776 x 2520
dcraw PPM: 3792 x 2536

2023-03-30_17-06-32_P1000044.RW2
Garbage pixels - size shown in image information: 3792 (3920) x 2476:

Camera's JPG: 3776 x 2520
dcraw PPM: 3704 x 2476

Images sizes from camera manual:
Aspect ratio 4:3 -> 3648 x 2736
Aspect ratio 3:2 -> 3776 x 2520
Aspect ratio 16:9 -> 3968 x 2232
Aspect ratio 1:1 -> 2736 x 2736

Additional info
I had optical zoom set to maximum; I may have accidentally slipped into the digital zoom range with the corrupted image. I've tried taking images with digital zoom to verify, but have been unable to reproduce the issue.

Platform
_Please fill as much information as possible in the list given below. Please state "unknown" where you do not know the answer and remove any sections that are not applicable _

• darktable version : 4.3.0+1501~gf103dd65ab
• OS : Linux 5.19.0-38
• Linux - Distro : Ubuntu 22.10
• Memory : 64 GB
• Graphics card : NVidia 1060/6GB
• Graphics driver : 470.161.03
• OpenCL installed : yes
• OpenCL activated : yes/no (tested both)
• Xorg : 7.7+23ubuntu2
• Desktop : KDE Plasma 5.25.5-0ubuntu1
• GTK+ : 3.24.34-3ubuntu2
• gcc : 12.2.0-3ubuntu1
• cflags : N/A
• CMAKE_BUILD_TYPE : Release

[kofa73]

The garbage pixels correspond to the area extending over the 3:2 crop set in the camera. Two columns are visible: one seems to be repeated image data; the other, garbage:

[kmilos]

I cannot reproduce using https://raw.pixls.us/ samples (in any of the 4:3, 3:2, 16:9 nor 1:1 modes) and dt 4.2.1 on Windows.

The default crop for 3:2 and 16:9 is e.g.

[kofa73]

It does not occur with all files. It may have something to do with digital zoom, as the LX7 crops the raw image if you use digital zoom. As an example, an image of a map was over 10 MB without digital zoom, and only 3.2 MB with maximum digital zoom.

[kmilos]

Perhaps try resetting the rawprepare ("raw black/white point") module?

You can also switch plugins/darkroom/rawprepare/allow_editing_crop=false to true in your user darktablerc and then verify the crop settings in the rawprepare module are as expected (compare to cameras.xml).

[kmilos]

As an example, an image of a map was over 10 MB without digital zoom, and only 3.2 MB with maximum digital zoom.

Ah, that could make sense, cameras.xml and rawprepare would not know how to handle that. They might even get confused and assign a (horizontal) crop from a different aspect ratio setting depending on what exact cropped dimensions are actually written out by the camera...

https://github.com/darktable-org/rawspeed/blob/5d8388d670fe88a71df364493bbed76a16b772fa/src/librawspeed/decoders/Rw2Decoder.cpp#L309-L339

Your best bet is to enable that hidden option then and adjust the rawprepare crop manually.

Of course, if you manage to find a pattern and a flaw in the RawSpeed mode guessing logic, that might be worth potentially fixing.

[kofa73]

As an example, an image of a map was over 10 MB without digital zoom, and only 3.2 MB with maximum digital zoom.

Ah, that could make sense, cameras.xml and rawprepare would not know how to handle that.

Actually, they do (something does, anyway). Here is the image info with a maximally zoomed-in image:

rawprepare's crop setting:

For the garbled image, 2023-03-30_17-06-32_P1000044.RW2, the correct value for the right crop seems to be 214.

[kmilos]

Actually, they do (something does, anyway).

It comes from the code snippet I linked to. For those values it grabs the 16:9 no-zoom setting from cameras.xml:

https://github.com/darktable-org/rawspeed/blob/bd03fd4878b5296d1f35cf40e554cfcff131bdb4/data/cameras.xml#L10333

[jsmucr]

darktable-org/rawspeed#490

[kmilos]

@kofa73 @jsmucr This should now be working in master, can you please confirm? (Might need to remove and reimport, or reset the raw black/point module...)

[kofa73]

Unfortunately it's still failing for 2023-03-30_17-06-32_P1000044.RW2.
4.5.0+1219~geb2d8420a0

This is after

git checkout master
git clean -d -f -x ;
git pull --rebase
git submodule update
./build.sh --prefix /home/kofa/darktable-master --build-type Release --install -DCUSTOM_CFLAGS=ON -DCMAKE_CXX_FLAGS="-O3 -march=native" -DCMAKE_C_FLAGS="-O3 -march=native"

[kmilos]

How about removing the crop lines for LX7 from cameras.xml?

[kofa73]

Thanks! The garbage is gone if I remove the crop lines.

[kmilos]

Thanks for checking. How does it affect normal files w/o digital zoom? Does it change the expected crop considerably when you reset the rawprepare module?

[kofa73]

I tried a wide-angle shot I exported earlier and now with current master + crop lines removed; the image changed by a few pixels only (I'm not exactly sure if it was the same stack).
Resetting rawprepare cropped in 8 pixels from all sides, so not an issue.
I cannot test more today. Post here if you want me to test more (aspect ratios etc.).

[kmilos]

Sounds like it's safe to make the change - old edits are preserved, the difference on new ones seems small enough.

[LebedevRI]

@kofa73 could you please contribute one such problematic sample to RPU?

[kofa73]

I've uploaded some (all aspect ratios with full "intelligent zoom" - Panasonic uses the term "digital zoom" when they don't just crop, but also enlarge the middle parts of the image).
Interestingly, some aspect ratios used to have artefacts when "intelligent zoom" was used partially (not at maximum zoom), but not when fully "zoomed in".
With the crop-lines removed (using vendor crop), I do not get any issues anymore. A pleasant surprise is that with some aspect ratios, I actually get more visible image surface with the crop-lines removed. E.g. with the aspect ratio 3:2, the cropped (i-zoomed) raw is 1904x1276 px (1.492 : 1); the old darktable crop was 1776x1276 (1:39 : 1, more like 4:3 than 3:2) (0, 0, 128, 0 for left, top, right, bottom crop); with the vendor crop, I get 1888 x 1260 (1.498 : 1) (8 px from all sides; all clear -- and I could even set 0 for all sides, there is no garbage at all).

[LebedevRI]

@kofa73 thank you for contributing the samples!