Idea: Allow use of libjpegli for jpeg import/export

[xiota]

libjpegli was formerly part of the libjxl project. Images are processed in 32-bit float representation, similarly to jxl. This would be useful to expand editing latitude of imported images. libjpeg-turbo also appears to have an option for expanded bit depth, but it must be enabled during build and is disabled by default.

The new library would need to be built statically, to prevent conflict with system libjpeg-turbo. Workflows that already build libjxl may work with adjusted build options. For others, libjpegli sources can be separated from the rest of libjxl and built as a subproject. Which jpeg library to use could be selected during configure for conditional compilation.

I have tested separating libjpegli from libjxl and compiling it as a submodule of a test project that uses it to convert jpegs to and from other formats. This code could be adapted to RawTherapee. (libjpegli is no longer part of libjxl and no longer any need to manually separate it.)

Thoughts? Opinions?

[kmilos]

libjpeg-turbo appears to have an option for expanded bit depth, but it must be enabled during build and is disabled by default.

No longer true since 3.x, it supports multiple sample bit depths OOTB. Code using it does have to be adapted though. However, I don't think you can ask for a higher bit-depth sample to what was encoded...

This would be useful to expand editing latitude of imported images.

Not sure I get this part - isn't RT pipeline float already?

Are you saying that, independent of RT, these are different:

8-bit jpeg -> libjpeg(-turbo) decode -> cast to float (w/ 1/255.f scale)
8-bit jpeg -> libjpegli decode direct to float

and the latter is somehow "better"? Any demos of this improvement?

[xiota]

No longer true since 3.x...

I'll look into it more. Has RawTherapee already been adapted?

Are you saying that, independent of RT, these are different...

jpeg is nominally 8-bits, but because of the transformation, rounding, quantization, etc, fewer bits are used. Then when the process is reversed, there's more rounding, etc to make it fit back into 8-bits. But if more bits are allowed in the process, the gaps between the 8-bit values (converted to float) can be filled in.

It can be seen with GIMP. Open a jpeg photo. Convert to 16-bit. Then look at the histogram. Convert the same photo to 16-bit pnm with djpegli. Look at the histogram.

This works because the transformations are based on sums of smooth curves.

[kmilos]

Has RawTherapee already been adapted?

I doubt it, because there are virtually no 12-bit or 16-bit JPEGs out there....

Thanks for the histograms, I get the quantization issues. I guess what I'm saying is - since the plain old JPEG was already heavily quantized, do you really gain anything in terms of noticeable image quality with theoretically more levels when decoding? I suppose it might only become apparent when doing some considerable level stretching when editing...

[xiota]

jpeg is quantized in a different domain, after transformation. The transformation is based on smooth curves, so when the process is reversed, the gaps are filled back in. They're not the exact same values, but it's better than a bunch of spikes.

It's noticeable in GIMP with curve and level adjustments. If RawTherapee has some tricks to improve the image, it may not matter as much. Depends on the users' needs.

[xiota]

It's kind of like the raw vs jpeg debate. Don't need raw if someone nails exposure every time. So why does anyone need a raw processor? (So people who bought into the jpeg side before realizing their mistake have a collection of jpegs they wish were raws... This might help them get a bit more out of them.)

[kmilos]

get a bit more

Sure. Just wondering how much is that "bit" 😉

[xiota]

Sure. Just wondering how much is that "bit" 😉

Suppose it's 2-4 extra bits. I don't really know what that means. How much does each bit allow to increase exposure or any other setting?

[kmilos]

Might get interesting w/ Ultra HDRs (JPEG + gain map). (see also https://github.com/google/libultrahdr)

[Lawrence37]

How much does each bit allow to increase exposure or any other setting?

Well, it's simple to test. Save an underexposed image as a jpeg. Open it the standard way and with higher precision. Correct the exposure for both versions and compare.

[jonnyawsom3]

Sure. Just wondering how much is that "bit" 😉

Suppose it's 2-4 extra bits. I don't really know what that means. How much does each bit allow to increase exposure or any other setting?

On the JXL Discord server it was said to be roughly 10.5 bits in a standard 8 bit Jpeg, when encoded and decoded with jpegli from a 16 bit source.
The reason for the half-a-bit is because of how the jpeg implementation works, it can technically allow more bits in smooth gradients (where it's needed) but has a lower limit on sharp areas.
Jpegli is currently the only implementation that uses this advantage, so the gains are much more noticeable when both encoded and decoded though it, rather than decoding an old file where it has less data to work with (But will still fill in the gaps).

[jonnyawsom3]

Just remembered about this issue and realised I have some tests results to share using SSIMULACRA2 against an 'original' PNG created from a DNG and downsized to reduce noise. (With bonus JXL transcode results)

jpg 84.09, djpegli 8-bit 85.37, djpegli 16-bit 85.48, Transcoded JXL input 85.75, djxl 8-bit 85.58, djxl 16-bit 85.78
As expected the original JPEG is lowest, djpegli adds an extra point, JXL is 0.4, then 16-bit adds 0.1 for JPEG and 0.2 for JXL.

This is using an 8-bit PNG as input for a most-common example, but I can re-run with 16-bit to gain the benefits of jpegli encoding too

[jonnyawsom3]

Did a fairly simple test by opening one of my DNGs, processing it to a 16-bit PNG, encoding that to various formats, and then adding +6 to Exposure Compensation to show the differences. Filesize and SSIMULACRA2 scores are next to names.

Original PNG 55.5 MB

Current JPEG 3.12 MB 53.75

Jpegli Decoded JPEG 3.12 MB 65.47

Jpegli Encoded and Decoded JPEG 2.31 MB 62.45 (Score seems wrong, visually identical to Original other than noise)

JXL bonus 4.05 MB 76.29

So overall, quite a large improvement, especially if encoded with jpegli since it can make use of that '10.5 bit' encoding previously discussed. The key areas are inside the engine, the shadow of the landing gear and the runway to the right that loose their blue tint due to quantization/rounding. JPEGs were both encoded at Quality 96 with 4:2:0 Subsampling, the JXL was encoded at higher quality but the key here is the decoding and effective bitdepth, not the quality itself.

[jonnyawsom3]

Another quick test comparing quality 100 JPEGs, in case naïve people thought that it meant lossless and saved files using it.

Quality 100 4:4:4 Jpegli encoded JPEG 6.03 MB 65.82

Quality 100 4:4:4 Jpegli encoded and decoded JPEG 6.03 MB 82.62

Once again, much better results decoding with jpegli, even beating my old JXL test which was actually a 16-bit file

[Lawrence37]

Those are good results. I would also like to see the results of high bit depth encoding with 8 bit decoding, which would be the scenario when a JPEG produced by RawTherapee is viewed in an application that only does 8 bit decoding. Do you know why the first libjpegli encoded and decoded image is smaller than the others? The image appears softer too.

[jonnyawsom3]

8-bit decoding is shown in the second to last image I uploaded, the last being the same file decoded with jpegli.

If you mean the PNG files that I uploaded, the lower quality jpeg encoding and decoding means the final result is more quantized, and easier to compress as a PNG.

[Lawrence37]

The second to last image is encoded with higher quality settings than the first series of JPEGs so they can't be compared.

The smaller and softer image I'm referring to is "Jpegli Encoded and Decoded JPEG 2.31 MB 62.45".

[jonnyawsom3]

I can make more comparisons tomorrow if I have time, currently away from home.

The smaller image is because jpegli uses perceptual based encoding, so filesize is smaller at the same visual quality. Though, Adaptive Quantization has been noted to be worse when above quality 90, I'll have to double check when I'm home.

[jonnyawsom3]

Those are good results. I would also like to see the results of high bit depth encoding with 8 bit decoding, which would be the scenario when a JPEG produced by RawTherapee is viewed in an application that only does 8 bit decoding.

Here's the jpegli encoded file used for Jpegli Encoded and Decoded JPEG 2.31 MB 62.45

I tested and Github doesn't touch the file, so the browser will display it using an 8-bit decoder, but downloading and using djpegli --bitdepth=16 will decode with the higher bitdepth.

And the exposure boosted decode of that file in RawTherapee currently

[jonnyawsom3]

The second to last image is encoded with higher quality settings than the first series of JPEGs so they can't be compared.

The smaller and softer image I'm referring to is "Jpegli Encoded and Decoded JPEG 2.31 MB 62.45".

I disabled the Adaptive Quantization on this file, seems to have improved the smoothing with a small filesize increase.

Jpegli No AQ Encoded and Decoded JPEG 2.65 MB 69.43

This should look a lot better

[Lawrence37]

It seems like the high bit depth decoding is responsible for the majority of the improvement. Encoding and decoding in high bit depth still has a noticeable improvement, but once 8-bit decoding is used, it doesn't look much better than 8-bit encoding and decoding. Is this what you notice as well?

[jonnyawsom3]

If I understand you correctly, yes. The improvements here are from jpegli's higher effective bitdepth. Other improvements such as quality are better compared at the original exposure. The high bit depth encoding has graceful fallback to standard 8bit decoding without jpegli, also loosing the higher bitdepth.

Standard JPEG with jpegli decoding: Higher quality output
Jpegli JPEG with standard decoding: Smaller filesize but equal quality
Both jpegli encoding and decoding: Smaller filesize and even higher quality

[Lawrence37]

Another though crossed my mind. Is the libjpegli compression different from libjpeg-turbo? If so and if libjpeg-turbo supports high bit-depth encoding/decoding, then it would be interesting to see if there is any difference between the libraries. If libjpegli offers better compression at the same quality, then it could be worthwhile to switch. Otherwise, it would be much simpler to use libjpeg-turbo's high bit-depth feature.

[xiota]

My not-at-all accurate mental model is that compression is similar to drawing with a French curve. If the French curve is "good enough", a few points can be rendered into a reasonable approximation of a "16-bit" curve, even if the original was only "8-bit".

I wonder whether a jpeg that shows banding at 8-bits would still do so when decompressed to higher bit depths with jpegli.

Based on #7125 (comment), libjpeg-turbo may not be able to decode to higher bit-depth than what was originally encoded? If that is (still) the case, libjpegli would still be needed (for bit depth expansion).

A potential feature aside from importing jpegs is bit-depth expansion of other 8-bit images (hinted at by tests using 8-bit png originals).

Although some distros include cjpegli/djpegli with the libjxl package, as far as I'm aware, they do not provide the libjpegli library. So packagers would need to compile it separately or it would have to be included as a subproject. Another option is to call the command-line program if available.

[xiota]

Does anyone know what happens if an 8-bit image is transformed with DCT to a higher bit-depth array. Then maybe add some noise, instead of quantizing, before transforming into a 16- or 32-bit image? Would the results be any better than trying to convert image data directly from 8-bit to 16-bit?

[jonsneyers]

There are two things that need to be distinguished:

• the nominal bit depth of the jpeg, which can be 8 bit or 12 bit, though de facto only 8 bit is supported by most jpeg decoders. Older versions of libjpeg-turbo would have this choice as a compile flag, so in practice it supported only 8-bit. Current libjpeg-turbo supports both. Jpegli only uses nominally 8-bit jpegs, since 12-bit jpeg will not decode in most deployments so it's almost like a new image format (even though it has been part of the 1992 standard all along, just was never supported by popular decoders until recently). Internally, 8-bit jpeg uses 12-bit DCT coefficients while 12-bit jpeg uses 16-bit DCT coefficients. The 4 extra bits are needed to make the DCT transform relatively close to reversible; without any extra bits you would get precision issues and extra quantization would quickly get very coarse; I assume they defined the number of extra precision bits to be 4 as a good compromise that still fits in int16_t.
• the precision of the encode/decode pipeline: in particular not just the DCT itself but also the intermediate YCbCr values (you can do iDCT with floats but still quantize the result to 8-bit YCbCr before doing chroma upsampling and conversion to RGB). As far as I understand, libjpeg-turbo focuses on speed, not precision. It is a conforming implementation, but conformance is defined quite loosely in the jpeg standard, allowing rather low precision. Moreover, the jpeg standard itself does not even define what the components mean, so anything related to YCbCr upsampling and conversion to RGB is outside the scope of the conformance testing (and this is where libjpeg-turbo loses a lot of precision, considering that 8-bit YCbCr corresponds only to 7-bit RGB). Jpegli implements the whole pipeline in floats, which allows it to get a reasonable fidelity even for HDR images while staying within the syntax of nominally 8-bit jpeg.

Effectively 8-bit jpegs have 12-bit precision for the DC coefficients, and in slow gradients you only need the lowest frequency AC coefficients so you can still get around 10-11 bits of precision for those parts of the image — provided your pipeline does not do any intermediate quantization. So the banding/blocking that you would expect when doing HDR in only 8-bit can be avoided since the effective precision is higher.
In 'busy' parts of the image (with lots of high frequency detail) the effective precision will be lower (only around 8 bits), but these are also regions where banding will not be an issue.