Add GIF decoder #8406

NicolasHug · 2024-05-03T13:05:42Z

Reviewers: DON'T PANIC this PR is a lot simpler than it looks. It's not 2.6k locs, only ~300 loc.

This PR adds the ability to decode GIF files via the decode_gif() function.

We are vendoring GIFLIB i.e. I copy/pasted the relevant giflib files into torchvision/csrc/io/image/cpu/giflib - this is why the PR looks big, but there's no need to review these files. GIFLIB is MIT licensed so we can do that. Vendoring avoids the dependency nightmare that libjpeg[-turbo] and libpng have been. I don't think giflib even exists on conda (except on conda-forge but even if it did, vendoring would still be easier.

Benchmark

TL;DR Sometimes it's faster than PIL (2x at most), somtimes it's slower (2x at most)

Benchmark script:

import torch
from time import perf_counter_ns


def bench(f, *args, num_exp=100, warmup=0, **kwargs):

    for _ in range(warmup):
        f(*args, **kwargs)

    times = []
    for _ in range(num_exp):
        start = perf_counter_ns()
        f(*args, **kwargs)
        end = perf_counter_ns()
        times.append(end - start)
    return torch.tensor(times).float()

def report_stats(times, unit="ms"):
    mul = {
        "ns": 1,
        "µs": 1e-3,
        "ms": 1e-6,
        "s": 1e-9,
    }[unit]
    times = times * mul
    std = times.std().item()
    med = times.median().item()
    print(f"{med = :.2f}{unit} +- {std:.2f}")
    return med


from PIL import Image, ImageSequence
from torchvision import io
from pathlib import Path

# files from https://sourceforge.net/p/giflib/code/ci/master/tree/pic/
paths = list(Path("/home/nicolashug/dev/giflib-code/pic/").glob("*.gif"))
paths += [Path("/home/nicolashug/grace.gif")]  # grace_hopper from torchvision/test/assets

def read_image_pil(path):
    for img in ImageSequence.Iterator(Image.open(path)):
        img.convert("RGB")

for path in paths:
    print()
    print(f"{path.name:<26} size={list(io.read_image(path).shape)}")
    print("tv : ", end="")
    times = bench(io.read_image, path, warmup=10)
    tv_med = report_stats(times)
    print("pil: ", end="")
    times = bench(read_image_pil, path, warmup=10)
    pil_med = report_stats(times)
    print(f"tv is {pil_med / tv_med:.2f}x faster")

Benchmark results:

fire.gif                  size=[33, 3, 60, 30]
tv : med = 0.62ms +- 0.03
pil: med = 1.24ms +- 0.11
tv is 2.01x faster

gifgrid.gif               size=[3, 100, 100]
tv : med = 0.07ms +- 0.01
pil: med = 0.09ms +- 0.01
tv is 1.25x faster

porsche.gif               size=[3, 200, 320]
tv : med = 0.43ms +- 0.04
pil: med = 0.28ms +- 0.04
tv is 0.65x faster

solid2.gif                size=[3, 400, 640]
tv : med = 1.62ms +- 0.15
pil: med = 0.90ms +- 0.09
tv is 0.55x faster

treescap-interlaced.gif   size=[3, 40, 40]
tv : med = 0.03ms +- 0.00
pil: med = 0.07ms +- 0.01
tv is 2.53x faster

treescap.gif              size=[3, 40, 40]
tv : med = 0.03ms +- 0.01
pil: med = 0.07ms +- 0.01
tv is 2.55x faster

welcome2.gif              size=[6, 3, 48, 290]
tv : med = 0.76ms +- 0.08
pil: med = 1.07ms +- 0.10
tv is 1.41x faster

x-trans.gif               size=[3, 100, 100]
tv : med = 0.08ms +- 0.01
pil: med = 0.11ms +- 0.01
tv is 1.33x faster

grace.gif                 size=[3, 606, 517]
tv : med = 3.41ms +- 0.17
pil: med = 2.64ms +- 0.11
tv is 0.77x faster

pytorch-bot · 2024-05-03T13:05:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/8406

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit b8e9427 with merge base 1644fff ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

NicolasHug · 2024-05-06T12:06:58Z

.github/scripts/unittest.sh

@@ -9,7 +9,7 @@ eval "$($(which conda) shell.bash hook)" && conda deactivate && conda activate c

 echo '::group::Install testing utilities'
 # TODO: remove the <8 constraint on pytest when https://github.com/pytorch/vision/issues/8238 is closed
-pip install --progress-bar=off "pytest<8" pytest-mock pytest-cov expecttest!=0.2.0
+pip install --progress-bar=off "pytest<8" pytest-mock pytest-cov expecttest!=0.2.0 requests


Needed for newly added test.

NicolasHug · 2024-05-06T12:07:57Z

setup.py

-                libraries=image_link_flags,
-                extra_compile_args=extra_compile_args,
-            )
+    ext_modules.append(


We are now always building the image library even if png and jpeg haven't been found, since we can always build libgif anyways. Maybe we'll add a WITH_LIBGIF flag or something similar, but this can come later.

Would be nice to add a TORCHVISION_USE_GIF env var similar to all other image backends for consistency.

Curious why you'd need it @adamjstewart ?
Those are useful for the other libraries as we dynamically link against them and they introduce dependencies, but since we vendor libgif we don't have the same issues with the gif decoder.

I'm coming from the perspective of a maintainer of a from-source package manager: Spack. We need to be able to build packages like torchvision from source on all kinds of weird systems (Linux aarch64, ppc64le, etc.) and with compilers (OneAPI, AOCC, Fujitsu, etc.) for which they are not widely tested. This requires patching these packages to fix compilation errors. For example, our giflib recipe already has multiple patches required for certain systems. Vendoring results in us having to patch multiple locations instead of one. I'm not opposed to it being optionally vendored, but would love support for externally-installed copies too.

it would be useful to have an option to use the shared library. We maintain things at conda-forge for shared linking which would make this easier to integrated.

One bug i'm running into (with the current startegy) is that clang for example likes to complain that -std-c++17 is added to compilation of a c file....

Adam, you can likely take my small patches to unvendor
conda-forge/torchvision-feedstock#94

Would love to see these merged upstream so we don't need to maintain a separate set of patches.

they would definitely need more work to be submitted here

NicolasHug · 2024-05-06T12:11:13Z

test/test_image.py

+    # and torchvision decoded outputs are equal.
+    # We're not testing against "welcome2" because PIL and GIFLIB disagee on what
+    # the background color should be (likely a difference in the way they handle
+    # transparency?)


For the curious, top row is what giflib / torchvision is decoding, bottom row is what PIL is decoding (PIL doesn't use giflib and has its own decoder). There seem to be a disagreement in what the background color should be. When I open the file on a file viewer, that background is marked as "transparent" anyways. I don't think it's worth worrying about that, at least for now.

What is PIL using internally to decode gif files? I wonder whether it would be more interesting to (re-)implement PIL's version of gif decoding instead of vendoring giflib here?

Part of the PIL GIF decoder is here: https://github.com/python-pillow/Pillow/blob/58a47978af9f34851ce926303d05fa677010ce2a/src/libImaging/GifDecode.c

But I think there are other parts elsewhere because I suspect this part only decodes the file into a "colormap" image, i.e. the values aren't RGB, only indices pointing to values to the image's colormap. I'm not sure where the other part is, I didn't look.

I wonder whether it would be more interesting to (re-)implement PIL's version of gif decoding instead of vendoring giflib here?

I wouldn't want to re-implement a GIF decoder on our own, there is simply too much room for us to make mistakes. It is a lot safer and simpler to rely on GIFLIB which has been around for decades, and is still maintained to this day (https://sourceforge.net/p/giflib/code/ci/master/tree/NEWS).
We could vendor the PIL decoder instead of vendoring GIFLIB but I don't see how this would simplify anything.

NicolasHug · 2024-05-06T12:12:48Z

test/test_image.py

+    # transparency?)
+
+    path = tmpdir / f"{name}.gif"
+    url = f"https://sourceforge.net/p/giflib/code/ci/master/tree/pic/{name}.gif?format=raw"


We have avoided downloading test stuff from the internet in the past because it was tricky to make those work on the internal CI, but I'll just skip those tests internally and add something else.
This avoids having weak tests and polluting the repo with image test files.

We can also create random gif files using PIL for the internal CI?

We might eventually do that, but this won't be a problem for a while. We just need to focus on the GitHub CI for this PR, I'll handle the internal stuff later

NicolasHug · 2024-05-06T12:13:39Z

torchvision/csrc/io/image/cpu/giflib/README

No need to review any file in cpu/giflib, except maybe for this one.

torchvision/csrc/io/image/cpu/decode_gif.h

NicolasHug · 2024-05-06T12:28:09Z