Handle image rotation for JPEG output

[farindk]

Image rotation is not working properly in all cases when outputting in JPEG format.
The problem is that is HEIF image is rotated, but the copied EXIF data causes the JPEG image to rotate again. Usually libheif disables HEIF image rotation when the output is JPEG and there is EXIF data, but with tiles images, this is not implemented. It is also a bit hacky anyways because when transformations are ignored, also other transformations (e.g. cropping) are not processed.

The right way to implement this would be to remove the Orientation tag from the output Exif data.

[jcupitt]

Hello, libvips is currently fighting this issue too. See also #117 for previous battles with orientation.

The situation seems to be that orientation is recorded in two places: as the set of transforms, and in EXIF. This means there are three cases:

1. The device adds transforms to flip the image upright and does not record the orientation in EXIF.
2. The device does not set any transforms, but does set the EXIF orientation tag. Downstream processing somewhere will need to read the tag and handle orientation.
3. Some Android phones seem to set transforms, and ALSO set EXIF. In this case, the EXIF orientation tag must be removed to prevent accidental double rotation later.

The problem for us is that there seems to be no easy way to separate cases 2. and 3., so libvips doesn't know whether to strip the EXIF orientation or not.

How about adding a little API so libheif users can tell if any load transforms were applied? It would save libheif having to add an EXIF parser as a dependency.

[farindk]

Hi John,
the HEIF standard says that the EXIF rotation is informational only and should not be used to actually rotate the image. This is different from JPEG where the EXIF data is used to rotate the image.

Actually, carrying out rotation based on EXIF according your case 2 is wrong with HEIF images.
I have written an HEIF->JPEG converter (not part of libheif), and the way to handle this correctly was to remove all Orientation, and resolution related tags from the EXIF before it was copied to the JPEG.

An API as you propose is not so simple, because in theory, HEIF transformations can be a chain of lots of different things. Rotate -> crop -> overlay with another image -> rotate back - > flip -> then put this into a grid with other images -> rotate the whole thing again -> ...

[jcupitt]

Hi Dirk, thank you for replying so quickly.

I'll remove our case 2 then, though I think that means some malformed images will not rotate correctly. I suppose I can at least blame someone else!

[farindk]

Yes, blaming someone else for case 2 is the way to go :-)

In fact, it's difficult to say how to correctly transform malformed images...
If "correcting" malformed images means that errors have to be introduced into the processing that will in consequence introduce errors to correct images, it is not worth it.

[jhonderson]

My workaround was to use:
convert input_file.jpg -rotate 270 output_file.jpg

After convert the image.

[ziriax]

To remove the orientation from the converted JPEG, I patched the heif-convert example in this repository using the attached patch, to be applied on top of v1.9.1 90b7f4f

It uses libexif to parse the metadata, I didn't find a way to modify the metadata using this library. I guess a PR is not welcome since the dependency on the libexif?

orientation.zip

[janokruta]

@farindk Can you share source which proves this?

the HEIF standard says that the EXIF rotation is informational only and should not be used to actually rotate the image

I found the opposite information here: https://nokiatech.github.io/heif/technical.html#table-7

Name: Image Rotation (‘irot’)
Type: Transformative
Description: Rotation by 90, 180, or 270 degrees.

[farindk]

Your reference mentions the 'irot' box. This is the mandatory information. But this is not part of the EXIF data. The EXIF data is separate and also specifies a rotation (orientation). That EXIF metadata is informational only.

See the excerpt from the standard:

[Kayvlim]

I was affected by this behavior today:

• Converting a HEIC file to PNG results in a correctly oriented image.
• Converting a HEIC file to JPG results in a correctly oriented image with EXIF information stating that the image is rotated.

GIMP (and at least Dolphin in KDE Plasma) is able to interpret the orientation tag and asks me whether to rotate the image. Keeping the original in GIMP is the right choice. Dolphin doesn't ask; it always rotates.

If anyone needs a "fix" that doesn't involve reprocessing the image, what I did was modify the EXIF information (using exif) for every image I converted from HEIC to JPG, and manually set Orientation = top-left.

mkdir ../modified
for image in *.jpg; do
  exif --ifd=0 --tag=0x112 --set-value=1 "$image" -o ../modified/"$image"
done

[xiaoyjy]

My trick way:

mv /usr/bin/heif-convert /usr/bin/heif-convert-bug

then edit /usr/bin/heif-convert paste and save

#!/bin/bash

lastnm="${@: -1}"
lastnm=`printf "%q" "$lastnm"`
suffix=${lastnm##*.}

args=()
for arg in "$@"; do
    args[${#args[@]}]=`printf "%q" "$arg"`
done

if [ "$suffix" = "jpg" ] || [ "$suffix" = "jpeg" ]; then
    eval heif-convert-bug ${args[@]}.png
    eval convert $lastnm.png $lastnm
    eval rm $lastnm.png
else
    eval heif-convert-bug ${args[@]}
fi

chmod +x /usr/bin/heif-convert

[Batwam]

I have noticed the same issue with converted files being rotated while the Exif value seems to be as per the original picture. I am having to add exiftool -n -Orientation=1 file.jpg after each conversion. Has a solution been implemented?

[farindk]

I have completely rewritten the orientation handling when converting from/to JPEG.
Now, when converting from JPEG to HEIF, irot/imir boxes are generated to match the JPEG orientation.
When converting from HEIF to JPEG, the image orientation is normalized, and the EXIF Orientation tag is set to "Normal".
8dc9fe1
4d9f13b

[lastzero]

Now, when converting from JPEG to HEIF, irot/imir boxes are generated to match the JPEG orientation.
When converting from HEIF to JPEG, the image orientation is normalized, and the EXIF Orientation tag is set to "Normal".

Unfortunately, this test image is displayed with the wrong orientation using the latest release v1.15.2:

• https://github.com/photoprism/photoprism/blob/develop/assets/examples/iphone_7.heic

The only solution for us is to use the old version v1.13.0. It seems to generate the same output JPEG, but does not reset the orientation, resulting in a correctly displayed image.

[farindk]

@lastzero Hm I'm sure I already wrote an answer to this last week, but it's not here. Maybe I forgot to press 'send' ...

I had a look at the iphone_7.heic file and it is decoded correctly with libheif > 1.13.0. "Correctly" means in this case that the wrong orientation is the correct output. The coded image has size 4032x3024 (landscape) and there is no 'irot' box that would rotate it.
I see that there is a rotation tag in the Exif data, but for HEIF, the Exif data is only informal and should not be used for carrying out the transformation.

There was a lot of confusion about this in the past and a lot of software was/is handling it incorrectly. The current behavior of libheif is correct.

If you want to emulate the old (wrong) behavior, you have to get the orientation tag from the Exif and compare it with the transformations in the HEIF (libheif v1.16.1 can now output the transformation properties). If they disagree, you can apply the Exif rotation. But be warned: this is a wrong behavior.

[lastzero]

Thank you for your reply, @farindk!

Inconsistent use of media metadata is a common phenomenon, unfortunately. We do our best to hide the complexity from our users, for example by automatically converting formats and normalizing data. However, since we don't have direct HEIC support in Go (which is why we very much appreciate your work), it's not clear to me how we can best work around the "wrong behavior":

• With v1.13.0, all of our file examples are displayed correctly and users have not reported any issues.
• In earlier versions, images were sometimes rotated twice, which we were able to fix with a wrapper script that set the Exif orientation to 1 if there was an additional rotation specified in the video metadata.
• What is the best way to reliably create properly displayed JPEGs in later versions with what we have available? Use the same hack as in earlier versions and detect the libheif version beforehand? 🤔

[farindk]

I'm afraid there is no simple solution. When you say that

With v1.13.0, all of our file examples are displayed correctly

this is not actually true, since that version just made the same common errors. The current version (v1.16.0) handles it correctly.

If you define "correct" as the orientation that the image is intended to be, you might have to take a look at other Exif metadata and determine from what software it was generated and maybe the generation timestamp and then from that modify the orientation in the hope that it will be correct and not make things worse.

In my view, the best approach is to show images correctly according to the specification. If some of the images are then displayed wrong, there should be an easy way for the user to fix the file. This will put pressure on other software authors to write correct files.

[lastzero]

Thanks! I will check again. From what I remember, this mistake was made by Apple, which makes it difficult for small teams like ours to publicly blame them or successfully ask for improvements. Also, there appear to be many affected users and pictures, making it time consuming for us to provide support. I'm completely with you otherwise.

[farindk]

You could check Exif for Make=Apple and also the iOS version from Software and then handle it accordingly:

Make                            : Apple
Camera Model Name               : iPhone XS Max
Orientation                     : Rotate 90 CW
Software                        : 12.2

I have no overview which iOS versions behave in which way. You probably have a larger data basis than me.

[lastzero]

Our archive of sample files is not exhaustive. Feel free to look around and use what seems helpful to you:

• https://dl.photoprism.app/samples/

I personally use Android and have very little time for HEIC in particular due to many other issues. If I learn more I will let you know!