docker load fails on Windows #2732

jwhb · 2022-04-07T11:13:13Z

Environmental Info:
RKE2 Version: v1.23.5+rke2r1

Node(s) CPU architecture, OS, and Version:
Windows Server 2019 Datacenter Version 1809 (Build 17763.2686), amd64

Cluster Configuration:
No cluster yet, goal is cluster with Windows nodes.

Describe the bug:

Followed RKE 2 Windows Air-Gap Install - Private Registry Method:

you may docker load the airgap image tarballs, then tag and push the loaded images.

When loading the image archive into Windows Docker EE (actually Mirantis CR) with docker load -i rke2-windows-1809-amd64-images.tar.gz,
Docker CLI reports unsupported os linux.

Docker CE on Ubuntu WSL rejects the very same windows images tar.gz file with the same error message. It seems that rke2-windows-1809-amd64-images.tar.gz is not compatible with docker load in Docker CE for Linux and Docker EE for Windows.

Steps To Reproduce:

Download rke2-windows-1809-amd64-images.tar.gz from latest release to Windows Server with Docker EE
Open elevated Windows terminal in directory with downloaded tar.gz
Run docker load -i rke2-windows-1809-amd64-images.tar.gz

Expected behavior:
Windows Docker EE loads the images from RKE2 release tar.gz.

Actual behavior:
Error message: unsupported os linux.

Additional context / logs:

C:\Temp\rancher_images>docker version
Client: Mirantis Container Runtime
 Version:           20.10.11
 API version:       1.41
 Go version:        go1.16.15m1
 Git commit:        761974f
 Built:             04/04/2022 20:18:51
 OS/Arch:           windows/amd64
 Context:           default
 Experimental:      true

Server: Mirantis Container Runtime
 Engine:
  Version:          20.10.11
  API version:      1.41 (minimum version 1.24)
  Go version:       go1.16.15m1
  Git commit:       1f612618f6
  Built:            04/04/2022 20:16:55
  OS/Arch:          windows/amd64
  Experimental:     false

C:\Temp\rancher_images>docker load -i rke2-windows-1809-amd64-images.tar.gz
unsupported os linux

On WSL Ubuntu with Docker:

➜ docker load -i airgap_files/rke2-windows-1809-amd64-images.tar.gz    
Loaded image: rancher/rke2-runtime:v1.23.5-rke2r1-windows-amd64
cannot load windows image on linux

The text was updated successfully, but these errors were encountered:

brandond · 2022-04-07T16:25:10Z

@rancher-max @phillipsj what tool do y'all usually use to load the tarballs into your registry for airgap testing?

rancher-max · 2022-04-07T16:35:00Z

I've only used the tarball when doing it in airgap, so I just use wget (on either a Mac or Linux machine that has access to my windows machine) to retrieve the file itself, copy it to the windows machine, and place it in the appropriate directory there (/var/lib/rancher/rke2/agent/images/).

I haven't had to use docker load for anything.

jwhb · 2022-04-07T16:45:31Z

My goal is to store the Windows images in a Docker v2 registry, so I can avoid preloading the images on the Windows nodes in /var/lib/rancher/rke2/agent/images/ (I assume this is C:\var\lib\...).

Essentially the "Private Registry Method" from Windows Air-Gap docs.

As I understand this document I can use a Windows system with Windows Docker EE installed, import the RKE2 windows images from the tar.gz using docker load, then use docker tag and docker push to make them available in my local registry.

The fallback would be the Windows Tarball Method, which requires me to copy the tarball onto each Windows node.

brandond · 2022-04-07T16:53:10Z

We use crane to build the windows image tarballs, you might consider using the same to copy them into your private registry. Docker does a poor job of managing images if they're not for the same platform it's running on.

rke2/scripts/package-windows-images

Lines 10 to 14 in abd8fd3

    
           # 1809/LTSC 
        
           crane --platform windows/amd64 pull \ 
        
             ${REGISTRY}/${REPO}/${PROG}-runtime:${DOCKERIZED_VERSION}-windows-amd64 \ 
        
             rancher/pause:${PAUSE_VERSION}-windows-1809-amd64 \ 
        
             rke2-windows-1809-amd64-images.tar

rancher-max · 2022-04-07T17:01:22Z

Ahh I see, yeah in that case, I still haven't messed with the docker load bit. I've hosted my private registry on a linux machine (so that I could use it for both the linux server and windows worker nodes).

The two images you need (from https://github.com/rancher/rke2/releases/download/v1.23.5%2Brke2r1/rke2-images.windows-amd64.txt) are:

docker.io/rancher/rke2-runtime:v1.23.5-rke2r1-windows-amd64
rancher/pause:3.6-windows-1809-amd64

I was able to get docker.io/rancher/rke2-runtime:v1.23.5-rke2r1-windows-amd64 into my registry with just:

docker pull docker.io/rancher/rke2-runtime:v1.23.5-rke2r1-windows-amd64
docker tag docker.io/rancher/rke2-runtime:v1.23.5-rke2r1-windows-amd64 my.registry.com/rancher/rke2-runtime:v1.23.5-rke2r1-windows-amd64
docker push my.registry.com/rancher/rke2-runtime:v1.23.5-rke2r1-windows-amd64

For the other image, I did have to use crane:

go install github.com/google/go-containerregistry/cmd/crane@latest
crane copy rancher/pause:3.6-windows-1809-amd64 my.registry.com/rancher/pause:3.6-windows-1809-amd64

Unfortunately I'm not 100% sure how to properly setup a windows-only registry, but hopefully this helps you 🤞

jwhb · 2022-04-07T21:19:09Z

We use crane to build the windows image tarballs, you might consider using the same to copy them into your private registry. Docker does a poor job of managing images if they're not for the same platform it's running on.

rke2/scripts/package-windows-images

Lines 10 to 14 in abd8fd3

# 1809/LTSC

crane --platform windows/amd64 pull \

${REGISTRY}/${REPO}/${PROG}-runtime:${DOCKERIZED_VERSION}-windows-amd64 \

rancher/pause:${PAUSE_VERSION}-windows-1809-amd64 \

rke2-windows-1809-amd64-images.tar

Thanks for the hint, it helps me understand the anatomy of these images. But I don't think that I can use crane to use the windows image tarball provided in the RKE2 GitHub release. My hope still is that I can just grab the tarball and do not have to create a tarball myself (with an online Windows Docker EE node).

Podman?

Before creating the issue I already tried podman load -i ... and it failed. Looking at it again I see that it failed because I was running in my non-root account and the error is: Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace (requested 197108:197121 for /bin/containerd-shim-runhcs-v1.exe): Check /etc/subuid and /etc/subgid: lchown /bin/containerd-shim-runhcs-v1.exe: invalid argument.

Podman!

I didn't see that coming (notice the IDs 197108:197121, >65535), but it works with modified ID ranges:

$ cat /etc/subuid /etc/subgid
jw:100000:200000
jw:100000:200000
$ podman system migrate
$ ➜ podman load -i airgap_files/rke2-windows-1809-amd64-images.tar.gz
Getting image source signatures
Copying blob 9bbd1b662743 skipped: already exists  
Copying config 184980b2fb done  
Writing manifest to image destination
Storing signatures
Getting image source signatures
Copying blob 4beae82b9843 skipped: already exists  
Copying blob c943530b8211 skipped: already exists  
Copying blob 90dbf06ef6a5 skipped: already exists  
Copying blob f186d149a4b1 skipped: already exists  
Copying config 9af01c3cdd done  
Writing manifest to image destination
Storing signatures
Loaded image(s): docker.io/rancher/rke2-runtime:v1.23.5-rke2r1-windows-amd64,localhost/rancher/pause:3.6-windows-1809-amd64
# no complaints about OS :^)

$ podman inspect docker.io/rancher/rke2-runtime:v1.23.5-rke2r1-windows-amd64 localhost/rancher/pause:3.6-windows-1809-amd64 -f "{{.Os}} - {{.RepoTags}}"
linux - [docker.io/rancher/rke2-runtime:v1.23.5-rke2r1-windows-amd64]
windows - [localhost/rancher/pause:3.6-windows-1809-amd64]

Notice how the runtime image is identified as linux and the pause image as windows.

Now the interesting part, pushing to my registry (docker.io/_/registry:2 on RHEL Docker CE):

$ podman push docker.io/rancher/rke2-runtime:v1.23.5-rke2r1-windows-amd64 10.171.128.236:5000/rancher/rke2-runtime:v1.23.5-rke2r1-windows-amd64 --tls-verify=false
Getting image source signatures
Copying blob 9bbd1b662743 done  
Copying config 184980b2fb done  
Writing manifest to image destination
Storing signatures
$ podman push localhost/rancher/pause:3.6-windows-1809-amd64 10.171.128.236:5000/rancher/pause:3.6-windows-1809-amd64 --tls-verify=false
# -snip-

Looks good, image was pushed to registry. Verifying that with skopeo:

$ skopeo inspect --tls-verify=false docker://10.171.128.236:5000/rancher/pause:3.6-windows-1809-amd64
{
    "Name": "10.171.128.236:5000/rancher/pause",
    "Digest": "sha256:7243f096e0054daa4fbceb2220727ad36ae838a5c664164bce7f7f8ef1eb7dd2",
    "RepoTags": [
        "3.6",
        "3.6-windows-1809-amd64"
    ],
    "Created": "2021-08-25T20:58:57.022319949Z",
    "DockerVersion": "",
    "Labels": null,
    "Architecture": "amd64",
    "Os": "windows",
    "Layers": [
        "sha256:0b58074eb07e66855011977e335042d20f94b0b4d01e758d1f8e0529e9ed0730",
        "sha256:e6e0fce93e19c5f6e42097810c82ac1c7760dd365bb62d937a15946997f9070b",
        "sha256:60dc28e41664baecf222b8e112e9804160a1a8a8563ec76f079413276d082a54",
        "sha256:af54810b52757096c904bbae9bca81ea4ae1a1be2ea10c1d94f5c8feec294c77"
    ],
    "Env": [
        "PATH=C:\\Windows\\system32;C:\\Windows;"
    ]
}

Good news. I'll test if I can deploy RKE2 on the Windows Server VMs with these images.

Looking back at my initial issue it's now clear why docker load failed on Windows and Linux Docker. Docker accepts only images that match the OS it's running on. The tarball has both a Windows and Linux image inside, at least this is what Podman and Skopeo tell me.

Convince yourself: skopeo inspect docker://docker.io/rancher/rke2-runtime:v1.23.5-rke2r1-windows-amd64 -f "{{.Os}}"

brandond · 2022-04-08T02:12:43Z

Yes, that's correct. And why I suggested using crane to copy the images from the tarball to your remote registry. I believe it should be capable of that.

jwhb · 2022-04-11T08:16:14Z

Yes, that's correct. And why I suggested using crane to copy the images from the tarball to your remote registry. I believe it should be capable of that.

@brandond: I did not find a subcommand of crane that accepts the tarball from RKE2 releases (rke2-windows-1809-amd64-images.tar.gz) as input and the docker registry as destination. What it seems to do is copy images from one registry to another registry (crane copy), copy images from a registry to a tarball (crane pull) or push an OCI image directory to a registry (crane push).

crane push failed because there were multiple images in the tarball:

$ /tmp/crane push airgap_files/rke2-windows-1809-amd64-images.tar.gz http://10.10.128.105:5000
Error: loading airgap_files/rke2-windows-1809-amd64-images.tar.gz as tarball: archive/tar: invalid tar header
$ gzip -k -d airgap_files/rke2-windows-1809-amd64-images.tar.gz
$ /tmp/crane push airgap_files/rke2-windows-1809-amd64-images.tar http://10.10.128.105:5000
Error: loading airgap_files/rke2-windows-1809-amd64-images.tar as tarball: tarball must contain only a single image to be used with tarball.Image
# same with --index

I can confirm that Windows Node setup works after copying the images to a private registry with skopeo.

Configuration used:

C:\etc\rancher\rke2\config.yaml:

server: https://10.10.128.110:9345
token: -snip-

C:\etc\rancher\rke2\registries.yaml:

configs:
  "10.10.128.105:5000":
    auth:
      username: rke
      password: "snipsnip"
mirrors:
  docker.io:
    endpoint:
      -  "http://10.10.128.105:5000"  # docker registry on linux docker

My Windows Server VMs have no way to connect to the internet, so they certainly only use the rke2 binary and the images from the local registry.

I think that the documentation should provide accurate instructions to use the tarball to fill a private registry. The podman method worked for me. Could this method be included in the documentation?

I suggest to keep this issue open to resolve this.

brandond · 2022-04-12T06:12:01Z

Hmm, that is interesting. I personally like skopeo a lot, in preference to crane or podman, but I'll defer to the Windows team on what tool they want to suggest for use.

rancher-max · 2022-04-12T15:29:52Z

I personally think it might be good here to be tool-agnostic. Mention that skopeo, podman, and crane have all been proven to work for this but users can use whichever tool they like that works to "copy images that have a set OS to your registry" and that docker load as of the time of the writing explicitly does not work.

jwhb · 2022-04-12T23:18:31Z

To summarize my observations about the potential tools to push RKE2 image tarballs (e.g. rke2-windows-1809-amd64-images.tar.gz) to a registry:

Docker: bad
- docker load does not work for these "mixed os" images.
- Generally requires a Linux and a Windows machine, if it would read the RKE2 tarballs. (bad)
Skopeo: bad
- does not seem to understand the tarball format. Neither oci-archive: nor docker-archive transports work. Would use skopeo sync if tarball format was supported.
Podman: good
- Works for these tarballs with podman load. May need to set UID range size to ~200,000.
- Does not care about os/arch definition in images for pull, load and push. (good!)
Crane: bad
- Supports multi-write (you use it for creating the tarballs), but does not support multi-read yet, see Support crane push tarBall contains more than one image? google/go-containerregistry#1109.
- Does not care about os/arch definition in images for pull and push. (good!)

There are some other tools that weren't mentioned here yet.

phillipsj · 2022-04-13T01:59:06Z

@jwhb thanks for that summary. We have an issue opened for the docs team to update with some guidance on the topic. I really like skopeo too and its interesting that skopeo sync doesn't support tarballs.

jwhb · 2022-04-13T10:53:07Z

I really like skopeo too and its interesting that skopeo sync doesn't support tarballs.

@phillipsj: I can elaborate on why it's not working out of the box with skopeo.

naive approach:

$ skopeo inspect docker-archive:airgap_files/rke2-windows-1809-amd64-images.tar
FATA[0000] Error retrieving manifest for image: Unexpected tar manifest.json: expected 1 item, got 2
# skopeo copy/sync will not work, because it doesn't read the archive like this

You can specify the index of an image in a docker-archive/tarball with :@, see containers/image#991:

$ skopeo inspect docker-archive:airgap_files/rke2-windows-1809-amd64-images.tar:@0 -f "{{.RepoTags}} {{.Architecture}} {{.Os}}"
[] amd64 linux
$ skopeo inspect docker-archive:airgap_files/rke2-windows-1809-amd64-images.tar:@1 -f "{{.RepoTags}} {{.Architecture}} {{.Os}}"
[] amd64 windows

So RepoTags are lost, but we can read individual images if we know their index. Now skopeo copy should also work for individual images in the tarball (tested with oci-archive as destination).

You might be able to combine that with this new feature: containers/skopeo#1581

Maybe I'm just missing a command flag somewhere. 🤔

brandond · 2022-04-13T16:51:39Z

It'd be nice if skopeo would just copy all the images in the tarball without having to pick them out individually.

sirredbeard added the kind/documentation Improvements or additions to documentation label Apr 11, 2022

This was referenced Apr 19, 2022

Documentation on how to load air-gap images into registry #2788

Closed

Fresh k3s v1.23.3+k3s1 airgap install, can't fetch mirrored-pause:3.6 k3s-io/k3s#5486

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker load fails on Windows #2732

docker load fails on Windows #2732

jwhb commented Apr 7, 2022 •

edited

Loading

brandond commented Apr 7, 2022

rancher-max commented Apr 7, 2022

jwhb commented Apr 7, 2022 •

edited

Loading

brandond commented Apr 7, 2022 •

edited

Loading

rancher-max commented Apr 7, 2022

jwhb commented Apr 7, 2022 •

edited

Loading

brandond commented Apr 8, 2022

jwhb commented Apr 11, 2022 •

edited

Loading

brandond commented Apr 12, 2022

rancher-max commented Apr 12, 2022 •

edited

Loading

jwhb commented Apr 12, 2022 •

edited

Loading

phillipsj commented Apr 13, 2022

jwhb commented Apr 13, 2022 •

edited

Loading

brandond commented Apr 13, 2022

docker load fails on Windows #2732

docker load fails on Windows #2732

Comments

jwhb commented Apr 7, 2022 • edited Loading

brandond commented Apr 7, 2022

rancher-max commented Apr 7, 2022

jwhb commented Apr 7, 2022 • edited Loading

brandond commented Apr 7, 2022 • edited Loading

rancher-max commented Apr 7, 2022

jwhb commented Apr 7, 2022 • edited Loading

Podman?

Podman!

brandond commented Apr 8, 2022

jwhb commented Apr 11, 2022 • edited Loading

brandond commented Apr 12, 2022

rancher-max commented Apr 12, 2022 • edited Loading

jwhb commented Apr 12, 2022 • edited Loading

phillipsj commented Apr 13, 2022

jwhb commented Apr 13, 2022 • edited Loading

brandond commented Apr 13, 2022

jwhb commented Apr 7, 2022 •

edited

Loading

jwhb commented Apr 7, 2022 •

edited

Loading

brandond commented Apr 7, 2022 •

edited

Loading

jwhb commented Apr 7, 2022 •

edited

Loading

jwhb commented Apr 11, 2022 •

edited

Loading

rancher-max commented Apr 12, 2022 •

edited

Loading

jwhb commented Apr 12, 2022 •

edited

Loading

jwhb commented Apr 13, 2022 •

edited

Loading