CI: ubi8 considered harmful (the "0x3c" bug)

[edsantiago]

The redhat registry is unreliable. We see these flakes in podman frequently:

# podman [options] pull registry.access.redhat.com/ubi8-minimal
Trying to pull registry.access.redhat.com/ubi8-minimal:latest...
time="2022-11-28T22:25:14Z" level=warning
    msg="Signature \"https://access.redhat.com/webassets/docker/content/sigstore/ubi8-minimal@sha256=2db613439c1708842f3fd6104a6b7b831f1f9b1c223c225101649e6bc4ae2983/signature-7\" 
    has Content-Type \"text/html\", unexpected for a signature"
Error: copying system image from manifest list: Source image rejected:
    parsing signature https://access.redhat.com/webassets/docker/content/sigstore/ubi8-minimal@sha256=2db613439c1708842f3fd6104a6b7b831f1f9b1c223c225101649e6bc4ae2983/signature-7:
    unrecognized signature format, starting with binary 0x3c

We already fixed buildah to not use it. And we keep adding workarounds elsewhere for registry bugs. It's time to just give up.

Some recent failure logs:

int podman fedora-36 root container: unknown failure

• fedora-36 : int podman fedora-36 root container
  • PR Allow manual override of install location in Windows installer #16659
    • 11-28 17:25

int podman fedora-37 rootless host: unknown failure

• fedora-37 : int podman fedora-37 rootless host
  • PR Allow manual override of install location in Windows installer #16659
    • 11-28 17:21
  • PR DO NOT MERGE: buildah vendor treadmill #13808
    • 11-18 08:31

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[Luap99]

still seeing these:
https://api.cirrus-ci.com/v1/artifact/task/5846543626928128/html/int-podman-fedora-37-root-container.log.html
https://api.cirrus-ci.com/v1/artifact/task/6409493580349440/html/int-podman-fedora-37-rootless-host.log.html

[edsantiago]

Well, quay.io has been pretty bad the last two weeks, but I have hopes that it will recover after the holidays. Filed #17017.

[MehdiSellamiAdministrator]

The "unrecognized signature format, starting with binary 0x3c" error message in Podman usually occurs when you try to pull an image that is not a valid Docker or OCI (Open Container Initiative) image. The error message indicates that Podman is interpreting the image as an HTML file, which is not a valid container image format.

To fix this issue, you should check the image source and ensure that it is a valid Docker or OCI image. Here are some steps you can take to troubleshoot the issue:

Check the image source: Make sure that you are pulling the image from a valid container image registry, such as Docker Hub or Quay.io.

Corrupted or incomplete image or container file: If the file you are trying to load is corrupted or incomplete, it may not be recognized by Podman. Try downloading the file again or verifying that it is not corrupted.

Verify the image format: Check that the image format is a valid Docker or OCI image format. You can verify the image format by downloading the image using curl and checking its file type. For example, the following command downloads the "nginx" image from Docker Hub and checks its file type.

The output should indicate that the file type is a Docker image or OCI image, depending on the format of the image.

File encoding issue: If the file you are trying to load has a different encoding than expected, this could also cause the error. Try converting the file to the correct encoding before running the podman load command.

Use the correct command: If you are trying to pull an image from a non-standard registry or using a different image format, you may need to use a different command to pull the image. For example, you can use the skopeo command to pull images from non-standard registries or in different image formats.

This copies the image from the private registry to the local file system in the dir directory.

By following these steps, you can troubleshoot the "unrecognized signature format" error message in Podman and ensure that you are pulling valid container images.

Configuring Linux container tools to check image signatures:

The Linux container tools (Podman, Skopeo, and Buildah) have built-in support for detached signatures, as well as the CRI-O container engine from Kubernetes and the Red Hat OpenShift Container Platform.

https://developers.redhat.com/blog/2019/10/29/verifying-signatures-of-red-hat-container-images

Check the image signature: If the image file is signed, check the signature to ensure that it is valid and matches the image file. Podman uses GPG (GNU Privacy Guard) signatures to verify the authenticity of signed images. You can use the podman trust inspect command to check the signature status of an image.

Use a different image source: If none of the above solutions work, try using a different image source or registry to obtain the image. The issue may be related to a specific registry or image file, and switching to a different source may help to resolve the problem.

[edsantiago]

@MehdiSellamiAdministrator thanks for taking the time to follow up with helpful advice. Unfortunately, in this case, none of that is applicable: the error we were seeing was a flake, meaning, exactly the same command will fail in nondeterministic, seemingly random ways. My recollection was that the failures were time-dependent, such that multiple independent tests would fail at close times; this strongly suggests a problem with the registry itself. That is beyond our control.