fix(heic): Make auto-transform of camera-rotated images optional

Policy change about ImageInput automatic transformation of images with
camera orientation data, as follows:

* Some ImageInput format readers may have the capability to
  automatically reorient images to the preferred display orientation
  upon input, based on the camera Orientation metadata. Whether a
  reader does so will tend to depend on the capabilities of the
  underlying format library. If it can support reorientation easily,
  it is expected to do the reorientation automatically by default.

* If the pixel data are re-oriented, the "Orientation" metadata should
  be set to 1 to indicate that the image returned to the caller is in
  the preferred display orientation) and "oiio:OriginalOrientation"
  metadata should be set to indicate the original camera orientation
  (prior to the automatic re-orientation).

* Open-with-configuration hint "oiio:reorient" (default: 1), if set to
  0, is a request to NOT do the reorientation. In that case,
  "Orientation" metadata should be set correctly to the actual
  orientation.

The heif reader was previously always re-orientating images as the
default behavior of the underlying libheif. That remains the default,
but this patch makes the following changes to libheif behavior to
conform to the policy above: (a) changes to code to allow it to
disable the reorienting behavior if hint "oiio:reorient" is 0; (b)
correct setting of Orientation and oiio:OriginalOrientation metadata;
(c) make this more clean in the documentation.

Tech details: The trick is that using the C++ API wrapper of libheif,
there is no way to pass the decoder options struct that is where you
need to say not to auto-transform.  Needed to drop down to the lower
level C API for this one spot.

Fixes #4123

A subsequent PR will revisit the RAW reader to make it also conform to
the new policy.

Signed-off-by: Larry Gritz <[email protected]>

src/cmake/testing.cmake

@@ -260,8 +260,10 @@ macro (oiio_add_all_tests)
                     IMAGEDIR openexr-images
                     URL http://github.com/AcademySoftwareFoundation/openexr-images)
     oiio_add_tests (heif
-                    FOUNDVAR Libheif_FOUND ENABLEVAR ENABLE_Libheif
-                    URL https://github.com/nokiatech/heif/tree/gh-pages/content)
+                    FOUNDVAR Libheif_FOUND
+                    ENABLEVAR ENABLE_Libheif
+                    IMAGEDIR oiio-images/heif
+                    URL http://github.com/AcademySoftwareFoundation/openexr-images)
     oiio_add_tests (ico
                     ENABLEVAR ENABLE_ICO
                     IMAGEDIR oiio-images URL "Recent checkout of oiio-images")

src/doc/builtinplugins.rst

@@ -685,6 +685,11 @@ preferred except when legacy file access is required.
    * - ``oiio:Gamma``
      - float
      - the gamma correction specified in the RGBE header (if it's gamma corrected).
+   * - ``heif:Orientation``
+     - int
+     - If the configuration option ``heif:reorient`` is nonzero and
+       reorientation was performed, this will be set to the original
+       orientation in the file.
 
 
 **Configuration settings for HDR input**
@@ -704,6 +709,14 @@ options are supported:
      - ptr
      - Pointer to a ``Filesystem::IOProxy`` that will handle the I/O, for
        example by reading from memory rather than the file system.
+   * - ``oiio:reorient``
+     - int
+     - The default of 1 means to let libheif auto-reorient the image to
+       undo the camera's orientation (this will set a "heif:Orientation"
+       metadata to the Exif orientation code indicating the original
+       orientation of the image). If this hint is set to 0, the pixels will be
+       left in their orientation as stored in the file, and the "Orientation"
+       metadata will reflect that.
 
 **Configuration settings for HDR output**
 
@@ -750,6 +763,15 @@ currently supported for reading, but not yet writing. All pixel data is
 uint8, though we hope to add support for HDR (more than 8 bits) in the
 future.
 
+The default behavior of the HEIF reader is to reorient the image to the
+orientation indicated by the camera, and to report the "Orientation" metadata
+as 1 (indicating that the image should be displayed as returned) and set the
+"oiio:OriginalOrientation" metadata to what was originally stored in the file.
+If you want to read the image without automatic reorientation, you can set the
+configuration option "oiio:reorient" to 0, in which case the pixels will be
+left in their orientation as stored in the file, and the "Orientation"
+metadata will reflect that.
+
 **Configuration settings for HEIF input**
 
 When opening an HEIF ImageInput with a *configuration* (see
@@ -769,6 +791,12 @@ attributes are supported:
        cause the reader to leave alpha unassociated (versus the default of
        premultiplying color channels by alpha if the alpha channel is
        unassociated).
+   * - ``oiio:reorient``
+     - int
+     - If nonzero, asks libheif to reorient any images (and report them as
+       having Orientation 1). If zero, then libheif will not reorient the
+       image and the Orientation metadata will be set to reflect the camera
+       orientation.
 
 **Configuration settings for HEIF output**
 

src/doc/imageinput.rst

@@ -923,6 +923,10 @@ hints are supported by each reader) are:
        cause the reader to leave alpha unassociated (versus the default of
        premultiplying color channels by alpha if the alpha channel is
        unassociated).
+   * - ``oiio:reorient``
+     - int
+     - If zero, disables any automatic reorientation that the reader may
+       ordinarily do to present te pixels in the preferred display orientation.
 
 Examples:
 

src/doc/stdmetadata.rst

@@ -77,21 +77,21 @@ Display hints
 
 .. option:: "Orientation" : int
 
-    y default, image pixels are ordered from the top of the display to the
-    ottom, and within each scanline, from left to right (i.e., the same
-    rdering as English text and scan progression on a CRT).  But the
-    "Orientation"` field can suggest that it should be displayed with
-    different orientation, according to the TIFF/EXIF conventions:
+    By default, image pixels are ordered from the top of the display to the
+    bottom, and within each scanline, from left to right (i.e., the same
+    ordering as English text and scan progression on a CRT).  But the
+    `"Orientation"` field can suggest that it should be displayed with
+    a different orientation, according to the TIFF/EXIF conventions:
 
     ===  ==========================================================================
-     0   normal (top to bottom, left to right)
-     1   flipped horizontally (top to botom, right to left)
-     2   rotated :math:`180^\circ` (bottom to top, right to left)
-     3   flipped vertically (bottom to top, left to right)
-     4   transposed (left to right, top to bottom)
-     5   rotated :math:`90^\circ` clockwise (right to left, top to bottom)
-     6   transverse (right to left, bottom to top)
-     7   rotated :math:`90^\circ` counter-clockwise (left to right, bottom to top)
+     1   normal (top to bottom, left to right)
+     2   flipped horizontally (top to bottom, right to left)
+     3   rotated :math:`180^\circ` (bottom to top, right to left)
+     4   flipped vertically (bottom to top, left to right)
+     5   transposed (left to right, top to bottom)
+     6   rotated :math:`90^\circ` clockwise (right to left, top to bottom)
+     7   transverse (right to left, bottom to top)
+     8   rotated :math:`90^\circ` counter-clockwise (left to right, bottom to top)
     ===  ==========================================================================
 
 .. option:: "PixelAspectRatio" : float

src/heif.imageio/heifinput.cpp

@@ -8,6 +8,14 @@
 
 #include <libheif/heif_cxx.h>
 
+#define MAKE_LIBHEIF_VERSION(a, b, c, d) \
+    (((a) << 24) | ((b) << 16) | ((c) << 8) | (d))
+
+#if LIBHEIF_NUMERIC_VERSION >= MAKE_LIBHEIF_VERSION(1, 17, 0, 0)
+#    include <libheif/heif_properties.h>
+#endif
+
+
 
 // This plugin utilises libheif:
 //   https://github.com/strukturag/libheif
@@ -18,6 +26,7 @@
 //     https://github.com/nokiatech/heif/tree/gh-pages/content
 
 
+
 OIIO_PLUGIN_NAMESPACE_BEGIN
 
 class HeifInput final : public ImageInput {
@@ -50,6 +59,7 @@ class HeifInput final : public ImageInput {
     bool m_associated_alpha        = true;
     bool m_keep_unassociated_alpha = false;
     bool m_do_associate            = false;
+    bool m_reorient                = true;
     std::unique_ptr<heif::Context> m_ctx;
     heif_item_id m_primary_id;             // id of primary image
     std::vector<heif_item_id> m_item_ids;  // ids of all other images
@@ -137,6 +147,7 @@ HeifInput::open(const std::string& name, ImageSpec& newspec,
 
     m_keep_unassociated_alpha
         = (config.get_int_attribute("oiio:UnassociatedAlpha") != 0);
+    m_reorient = config.get_int_attribute("oiio:reorient", 1);
 
     try {
         m_ctx->read_from_file(name);
@@ -200,14 +211,14 @@ HeifInput::seek_subimage(int subimage, int miplevel)
         return false;
     }
 
+    auto id     = (subimage == 0) ? m_primary_id : m_item_ids[subimage - 1];
+    m_ihandle   = m_ctx->get_image_handle(id);
+    m_has_alpha = m_ihandle.has_alpha_channel();
+    auto chroma = m_has_alpha ? heif_chroma_interleaved_RGBA
+                              : heif_chroma_interleaved_RGB;
+#if 0
     try {
-        auto id     = (subimage == 0) ? m_primary_id : m_item_ids[subimage - 1];
-        m_ihandle   = m_ctx->get_image_handle(id);
-        m_has_alpha = m_ihandle.has_alpha_channel();
-        auto chroma = m_has_alpha ? heif_chroma_interleaved_RGBA
-                                  : heif_chroma_interleaved_RGB;
-        m_himage    = m_ihandle.decode_image(heif_colorspace_RGB, chroma);
-
+        m_himage = m_ihandle.decode_image(heif_colorspace_RGB, chroma);
     } catch (const heif::Error& err) {
         std::string e = err.get_message();
         errorf("%s", e.empty() ? "unknown exception" : e.c_str());
@@ -217,6 +228,23 @@ HeifInput::seek_subimage(int subimage, int miplevel)
         errorf("%s", e.empty() ? "unknown exception" : e.c_str());
         return false;
     }
+#else
+    std::unique_ptr<heif_decoding_options, void (*)(heif_decoding_options*)>
+        options(heif_decoding_options_alloc(), heif_decoding_options_free);
+    options->ignore_transformations = !m_reorient;
+    // print("Got decoding options version {}\n", options->version);
+    struct heif_image* img_tmp = nullptr;
+    struct heif_error herr = heif_decode_image(m_ihandle.get_raw_image_handle(),
+                                               &img_tmp, heif_colorspace_RGB,
+                                               chroma, options.get());
+    if (img_tmp)
+        m_himage = heif::Image(img_tmp);
+    if (herr.code != heif_error_Ok || !img_tmp) {
+        errorfmt("Could not decode image ({})", herr.message);
+        m_ctx.reset();
+        return false;
+    }
+#endif
 
     int bits = m_himage.get_bits_per_pixel(heif_channel_interleaved);
     m_spec = ImageSpec(m_ihandle.get_width(), m_ihandle.get_height(), bits / 8,
@@ -270,21 +298,77 @@ HeifInput::seek_subimage(int subimage, int miplevel)
             decode_xmp(metacontents, m_spec);
         } else {
 #ifdef DEBUG
-            std::cout << "Don't know how to decode meta " << m
-                      << " type=" << m_ihandle.get_metadata_type(m)
-                      << " contenttype='"
-                      << m_ihandle.get_metadata_content_type(m) << "'\n";
-            std::cout << "---\n"
-                      << string_view((const char*)&metacontents[0],
-                                     metacontents.size())
-                      << "\n---\n";
+            print(
+                "Don't know how to decode meta {} type='{}' contenttype='{}'\n",
+                m, m_ihandle.get_metadata_type(m),
+                m_ihandle.get_metadata_content_type(m));
+            print("---\n{}\n---\n",
+                  string_view((const char*)metacontents.data(),
+                              metacontents.size()));
 #endif
         }
     }
 
+    // Try to discover the orientation. The Exif is unreliable. We have to go
+    // through the transformation properties ourselves. A tricky bit is that
+    // the C++ API doesn't give us a direct way to get the context ptr, we
+    // need to resort to some casting trickery, with knowledge that the C++
+    // heif::Context class consists solely of a std::shared_ptr to a
+    // heif_context.
+    // NO int orientation = m_spec.get_int_attribute("Orientation", 1);
+    int orientation = 1;
+    const heif_context* raw_ctx
+        = reinterpret_cast<std::shared_ptr<heif_context>*>(m_ctx.get())->get();
+    int xpcount = heif_item_get_transformation_properties(raw_ctx, id, nullptr,
+                                                          100);
+    orientation = 1;
+    xpcount     = std::min(xpcount, 100);  // clamp to some reasonable limit
+    std::vector<heif_property_id> xprops(xpcount);
+    heif_item_get_transformation_properties(raw_ctx, id, xprops.data(),
+                                            xpcount);
+    for (int i = 0; i < xpcount; ++i) {
+        auto type = heif_item_get_property_type(raw_ctx, id, xprops[i]);
+        if (type == heif_item_property_type_transform_rotation) {
+            int rot = heif_item_get_property_transform_rotation_ccw(raw_ctx, id,
+                                                                    xprops[i]);
+            // cw[] maps to one additional clockwise 90 degree turn
+            static const int cw[] = { 0, 6, 7, 8, 5, 2, 3, 4, 1 };
+            for (int i = 0; i < rot / 90; ++i)
+                orientation = cw[orientation];
+        } else if (type == heif_item_property_type_transform_mirror) {
+            int mirror = heif_item_get_property_transform_mirror(raw_ctx, id,
+                                                                 xprops[i]);
+            //                                1  2  3  4  5  6  7  8
+            static const int mirrorh[] = { 0, 2, 1, 4, 3, 6, 5, 8, 7 };
+            static const int mirrorv[] = { 0, 4, 3, 2, 1, 8, 7, 6, 5 };
+            if (mirror == heif_transform_mirror_direction_vertical) {
+                orientation = mirrorv[orientation];
+            } else if (mirror == heif_transform_mirror_direction_horizontal) {
+                orientation = mirrorh[orientation];
+            }
+        }
+    }
+
     // Erase the orientation metadata because libheif appears to be doing
     // the rotation-to-canonical-direction for us.
-    m_spec.erase_attribute("Orientation");
+    if (orientation != 1) {
+        if (m_reorient) {
+            // If libheif auto-reoriented, record the original orientation in
+            // "oiio:OriginalOrientation" and set the "Orientation" attribute
+            // to 1 since we're presenting the image to the caller in the
+            // usual orientation.
+            m_spec.attribute("oiio:OriginalOrientation", orientation);
+            m_spec.attribute("Orientation", 1);
+        } else {
+            // libheif supplies oriented width & height, so if we are NOT
+            // auto-reorienting and it's one of the orientations that swaps
+            // width and height, we need to do that swap ourselves.
+            if (orientation == 5 || orientation == 6 || orientation == 8) {
+                std::swap(m_spec.width, m_spec.height);
+                std::swap(m_spec.full_width, m_spec.full_height);
+            }
+        }
+    }
 
     m_subimage = subimage;
     return true;