RFE: pull --retry

[edsantiago]

[like #14048 but for pull]

This is causing multiple CI flakes per day:

# podman [options] pull registry.access.redhat.com/ubi8-init
Trying to pull registry.access.redhat.com/ubi8-init:latest...
Error: copying system image from manifest list: Source image rejected: Get "https://access.redhat.com/webassets/docker/content/sigstore/ubi8-init@sha256=f6afbab2349ef86bd4ac0d59ba4b9b5df3e176b4bdeaff643c3e6386a7414c24/signature-2": net/http: TLS handshake timeout

We have no control over broken registries or network hiccups.

Could we just implement podman pull --retry 3 with perhaps an exponential backoff?

[vrothberg]

Podman is already doing three retries by default (see https://github.com/containers/common/blob/main/libimage/copier.go#L273) but it seem the specific error isn't caught in the retry semantic.

[vrothberg]

@mtrmac AFAIKS, there's no public error type we could check for, so we may need to string compare. WDYT?

[rhatdan]

This would certainly make sense to retry on this case.

[mtrmac]

@mtrmac AFAIKS, there's no public error type we could check for, so we may need to string compare.

What’s the exact type/value? Reading the code, I’d expect this to be an ordinary errors.Wrapf-ed HTTP error, testable via errors.As/errors.Is (In particular, it should not be a signature.PolicyRequirementError.)

Going further, net/http.tlsHandshakeTimeoutError (if that’s what the underlying error is) is not public, but it should be possible to test that via net.Error.Timeout(). Adding that to the retry logic would make sense, I think there’s at least one other report for missing retry logic that suggests testing that condition.

Now, the hard part about all of these reports is reproducing them enough to test the heuristics…

[edsantiago]

It's not just TLS timeout, there's also internal error:

Error: copying system image from manifest list: Source image rejected:
      Get "https://access.redhat.com/webassets/docker/content/sigstore/ubi8-minimal@sha256=9a81cce19ae2a962269d4a7fecd38aec60b852118ad798a265c3f6c4be0df610/signature-3":
       remote error: tls: internal error

(seen here)

[vrothberg]

NOTE: this is not fixed until c/common@main is vendored into Podman

[praiskup]

Can we configure the number of retries, and delay somehow? Seems like Mock/Copr users have problems with pull failures, and we could afford even say 30s delays (instead of the default 1s).

[vrothberg]

@praiskup, no, that's is currently not configurable for Podman. Please open a new issue if you desire such a feature.