docker_push incompatible with quay.io (manifest schema version not supported?)

[benley]

I'm trying to push an image to quay.io with docker_push, and get the following error:

$ bazel run //python2.7:push_python27_cpp 
INFO: Analysed target //python2.7:push_python27_cpp (0 packages loaded).
INFO: Found 1 target...
Target //python2.7:push_python27_cpp up-to-date:
  bazel-bin/python2.7/push_python27_cpp
INFO: Elapsed time: 0.167s, Critical Path: 0.01s
INFO: Build completed successfully, 1 total action

INFO: Running command line: bazel-bin/python2.7/push_python27_cpp
+ ../pusher/file/pusher.par --name=quay.io/postmates/distroless_python27_cpp:dev --config=python2.7/python27_cpp.config --digest=base/with_tmp-layer.tar.gz.sha256 --digest=base/base-layer.tar.gz.sha256 --digest=python2.7/python27-layer.tar.gz.sha256 --digest=python2.7/python27_cpp-layer.tar.gz.sha256 --layer=base/with_tmp-layer.tar.gz --layer=base/base-layer.tar.gz --layer=python2.7/python27-layer.tar.gz --layer=python2.7/python27_cpp-layer.tar.gz
ERROR:root:Error during upload of: quay.io/postmates/distroless_python27_cpp:dev
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "../pusher/file/pusher.par/__main__.py", line 129, in <module>
  File "../pusher/file/pusher.par/__main__.py", line 124, in main
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_session_.py", line 300, in upload
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_session_.py", line 228, in _put_manifest
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_http_.py", line 340, in Request
containerregistry.client.v2_2.docker_http_.V2DiagnosticException: response: {'status': '415', 'content-length': '131', 'server': 'nginx/1.13.3', 'connection': 'keep-alive', 'date': 'Wed, 02 Aug 2017 19:14:01 GMT', 'content-type': 'application/json'}
manifest invalid: {u'message': u'manifest schema version not supported'}

[mattmoor]

I believe this is because Quay still doesn't support v2.2.

Theoretically we could fall back on v2.1, but the library we're using doesn't have a good/portable way of performing the signing that would require.

It really surprises me that they don't support v2.2, since OCI is basically a fork of it...

[mattmoor]

@jonjohnsonjr Perhaps we can attempt OCI if v2.2 doesn't work. WDYT?

[jonjohnsonjr]

Seems reasonable, but we'd probably want this to be optional to avoid breaking any clients that consume these images but don't support OCI.

I'm not sure of the best way to surface that option. Adding a --use_oci flag to everything seems inelegant...

[mattmoor]

@jonjohnsonjr I was thinking that if v2.2 upload fails we'd fallback on oci.

[mattmoor]

I'm going to close this in favor of: #110

I don't think we're going to support v2 schema 1 push because the signing support is annoying to properly support in the library in a portable way. I think that we'd likely get this by supporting OCI (I'm assuming Quay supports that because they've been such big proponents), or by Quay finally supporting schema 2.

[hsyed]

@mattmoor, is there some interim workaround for this issue ?

[mattmoor]

I believe our underlying library has OCI support now, if quay.io supports OCI I can start taking a look at adding an oci_push rule under contrib/ to unblock you.

[hsyed]

That would be fantastic. I'll await the workaround.

[mattmoor]

Well as-is, this gets a 500 from Quay, so either they don't support it, or it is incompatible with our library's implementation. @jonjohnsonjr do you want to TAL to see if you can determine which?

[jonjohnsonjr]

If they don't support schema 2 images I'd guess they don't support OCI, but I'll see if I can find out.

[mattmoor]

That would be my bet as well, but given @philips position on OCI, I'm fairly surprised that Quay doesn't yet support OCI. @philips do you guys have any timeline for v2.2 or OCI support?

AIUI, there isn't a good / portable way to deal with native deps (e.g. crypto) from PAR files, so falling back on v2 (which requires clients to sign manifests) is probably more trouble than it is worth.

[hsyed]

I suppose I'll have to write a genrule to setup the various pushes -- would make the rules_k8s less intuitive. @philips I just opened a quay account .... might have to choose another solution :(

[kalbasit]

Any news on this? I still can't push to Quay.

[jonjohnsonjr]

Does quay actually not support Schema 2 manifests? I haven't actually confirmed it myself. Schema 1 manifests are (IMO) ~broken by design, and Schema 2 has been around for over 2 years now, so I'm really surprised by this.

It wouldn't be impossible to make this work, but I don't think we want to carry around the baggage...

The problem is that the pushers in https://github.com/google/containerregistry/tree/master/tools try to push in the schema 2 format.

Luckily for you, that library has a compatibility layer for converting from schema 2 to schema 1: https://github.com/google/containerregistry/blob/3135604d9e825c145f03367194b3e3fd8ee87ca3/client/v2_2/v2_compat_.py#L174

So... you could fix this with just a few lines of code, but I'd personally be opposed to merging it back into here. Juggling those formats is a headache and I'd like to see schema 1 die peacefully with everything being OCI going forward.

Maybe something like a deprecated_schema1_docker_push rule could be added, but I'd want it to be clear that it's not something we encourage :P

[mattmoor]

@jonjohnsonjr see my first comment on this thread :)

The library we use doesn't expose a way to sign v2.1 manifests, and the effort to add it (as @jonjohnsonjr indicates) is a poor use of time.

IMO v2.1 is effectively EOL. We aren't adding it at all in our Go port of the library, so it knowingly won't support Quay at all.

It sucks, and I hate to say it, but the simplest solution would be to migrate to a modern registry.

[kalbasit]

It sucks, and I hate to say it, but the simplest solution would be to migrate to a modern registry.

Do you have a suggestion? I know Bazel works with Docker hub, however, they do not offer Robot Accounts (like quay does) which I make use of extensively in Kubernetes.

[mattmoor]

GCR supports robots (disclaimer: I started the GCR team).

I'd be surprised if the registries from the other cloud providers didn't also support robot accounts.

I'm not sure what your selection criteria is?

[jonjohnsonjr]

Obviously biased suggestion: gcr.io :)

Less biased suggestion: if you're using a cloud provider, I'd use their registry due to speed-of-light-being-finite reasons, e.g. gcr, ecr, acr.

[kalbasit]

We push images for every CI build, on every branch. This allows our dev team to push out the images from their branches to dev environment to do Dev QA before merging and releasing. I've looked at AWS ECR before and they have a limit of 1000 tags per image. Also we require robo accounts but that's pretty much what we rely on.

[mattmoor]

I've looked at AWS ECR before and they have a limit of 1000 tags per image

/facepalm

This shouldn't be a problem for GCR, and you can use it from AWS (we had customers doing that before ECR launched).

I wish this didn't sound like a big shill for GCR... :-/

[kalbasit]

I'll give GCR a try then, Thanks Matt.

[jonjohnsonjr]

FYI: moby/buildkit#409 (comment)