Wrong image rotation/orientation applied to output

[Dwynr]

Hi,

converting a HEIC image shot on an iPhone 13 to either a JPEG or PNG results into the image being flipped (wrongly rotated) for some images.
Looking at the EXIF data of the converted JPEG it looks like it just sets the orientation to 3 for every image.

I'll also attach the test HEIC file if you want to debug it (it's zipped because Github does not allow HEIC files)

heic.heic.zip

[catdad]

Hmm... so this is interesting. So here's what I found:

• this image show in the correct (that is, in human terms) orientation on my iPad (5th gen Air on 15.6.1)
• this image shows in the wrong orientation in Windows Photos
• this image shows in the wrong orientation in Affinity Designer
• this image does have orientation set to seemingly the correct value in its exif
• this image is missing most of its exif data

I tried to repro this with my own iPad. I took photos in all 4 orientations (again, in human terms, I know there's actually 8 orientations, because of course there is). All of those photos had orientation set appropriately in their exif, and also converted correctly, outputting a jpeg image in the correct orientation (all jpeg images are missing exif data, as that is not copied over by this lib). This part of the conversion is all handled natively by libheif, and the heic-converter wrapper never even tries to do anything about it. This makes it a bit interesting, because while all images have the correct orientation exif value, I can't rotate all images, since libheif seems to already be rotating most of them (certainly all images I have other than the one you uploaded).

Now, the fact that your image is missing most of its exif data is interesting. How did you generate this image. It doesn't seem to just be a normal iPhone image. Did you use a novel way of getting this image from your iPhone to your computer? Did you purposefully strip the exif data (I get it, it can have GPS data in there and whatnot, you wouldn't want to share that on the internet)? If so, how did you do that? Essentially, there are two options that we need to narrow down:

• too much exif data was removed, and however it was removed did a bad job -- it probably has a bug
• libheif depends on more than just orientation for some reason -- it probably has a bug

[Dwynr]

The image attached is just a default demo image that Apple provides with the iOS Simulators. I did not attach the images I made due to privacy reasons. I was testing earlier today and I could reproduce it again - though not every time. I took a bunch of images with my iPhone and converted them all. It seems to happen completely randomly.

If you want I can send you the original iPhone images (RAW, including all EXIF data) in private for debugging purposes.

[catdad]

Haha, it's very Apple to not follow specs and make it everyone else's problem.

Feel free to email the images to [email protected]. You can always turn off saving gps data to images before taking test photos... if you want... I promise I won't share them myself.

[Dwynr]

Sent. :)

[catdad]

Well... this is fun. For posterity, I am reading the exif information using exiftool. Of the 3 images you sent, the 2 that render correctly have 135 fields in their exif data. The one that renders sideways has only 61 fields. It looks like the sample image from the emulator that you sent. Which is just weird, right? That's definitely an Apple bug. Why would the phone sometimes write a lot of exif data and sometimes just a little bit of exif data? Of course, who do you even report that to? Apple famously doesn't make mistakes.

For posterity, I tested these images in all the apps I mentioned before, and I see the same thing. The image with less exif data renders sideways in all of them. It even renders sideways in the OneDrive preview that you sent me (which... maybe no one is surprised, they all use libheif for the rendering).

But, of course, I promised fun. All 3 of the images have the orientation field set correctly. Orientation is part of the Tiff tags and the IFD1 tags. The 2 images that render correctly have an additional rotation field set, while the third image is missing that field. Now... rotation doesn't seem to be part of any exif spec I am aware of. It looks like it's part of the MIE spec, which is different? It appears the spec was written by Phil Harvey, the same guy who wrote exiftool, so that's nice. I know very little about all this actually.

Anyway, I digress. I have no idea why your images are different. Again, that seems like an Apple problem (why would you write two different images Apple??). Did you know if you did something subtle or different when taking these? Taking consecutive images super fast? Some king of burst mode? Does Apple's camera even have burst mode? I mostly want to be able to take repro images of my own.

[catdad]

Of course, after I did all that, I simply went to the source and found that libheif already has an issue with this that explains something completely different (i.e. more correct and less wild guessing): strukturag/libheif#227.

I was on the right track though. It looks like in the HEIF spec, EXIF data (including orientation) is considered informational and should not change how images are rendered (in contrast to JPEG, where orientation is expected to be applied at render-time to correct the image). HEIF has a specific way to define complex transformations, and an image is considered malformed if it doesn't have those (or does not include any one specific transformation that a thinking human would go "hey, that's wrong"). It seems to happen with tiled images? Which, that third image is tiled while the first two are not.

In the words of that bug... "Yes, blaming someone else... is the way to go :-)"

[Dwynr]

Alright got it, seems like Apple is ruining the show..

What do you recommend as a workaround? The one in the issue you mentioned?

[catdad]

That's your call really. Rotating the jpeg is trivial, it's detecting that there's a problem that is the hard part.

If you are just converting them for personal use and it is not a lot of images, you can just look at your results and fix the ones didn't work out. Pretty much every photo app nowadays supports rotation.

If you are wanting to automate something for general use, that's when it gets hard. Technically, as Apple has written the file, it is not supposed to be rotated. And who's to say that the thing Apple is doing wrong when generating this completely legal image is not something that some other software does correctly and expect it to work as intended following the spec 🤷‍♀️.

exiftool can extract exif data, but of course, it's not native javascript so using it depends on your use case. There are a bunch of exif libraries on npm, though not all of them work on all different types of images. You could look for images that specify orientation without specifying rotation? Look for images that specify tile information?

When you get a jpeg, you could write exif data to it? Or use heic-decode and sharp or jimp to rotate the image before encoding to jpeg?

I think no one is solving this one in an automated way because Apple's bug can also be someone else actually meaning to do that same thing but doing it correctly, in which case a "fix" actually breaks the image. Apple's answer would be to only view photos taken on an Apple camera and only view them on an Apple device.