[BUG] Creating a pull-through registry seems to be broken

[cowwoc]

What did you do

• How was the cluster created?
  k3d registry create registry.docker.localhost --port 5000 --proxy-remote-url https://registry-1.docker.io

• What did you do afterwards?

  • docker pull registry.docker.localhost:5000/nginx:latest

What did you expect to happen

I expected the registry to pull nginx:latest from DockerHub, cache it locally, and return the image back to the client.

Screenshots or terminal output

docker pull registry.docker.localhost:5000/nginx:latest
Error response from daemon: failed to resolve reference "registry.docker.localhost:5000/nginx:latest": registry.docker.localhost:5000/nginx:latest: not found

Which OS & Architecture

• output of k3d runtime-info

arch: x86_64
cgroupdriver: cgroupfs
cgroupversion: "1"
endpoint: /var/run/docker.sock
filesystem: UNKNOWN
infoname: docker-desktop
name: docker
os: Docker Desktop
ostype: linux
version: 27.4.0

Which version of k3d

• output of k3d version

k3d version v5.8.1
k3s version v1.31.4-k3s1 (default)

Which version of docker

• output of docker version and docker info

docker version returns:

Client:
 Version:           27.4.0
 API version:       1.47
 Go version:        go1.22.10
 Git commit:        bde2b89
 Built:             Sat Dec  7 10:40:21 2024
 OS/Arch:           windows/amd64
 Context:           desktop-linux

Server: Docker Desktop 4.37.1 (178610)
 Engine:
  Version:          27.4.0
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.10
  Git commit:       92a8393
  Built:            Sat Dec  7 10:38:57 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.21
  GitCommit:        472731909fa34bd7bc9c087e4c27943f9835f111
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info returns:

Client:
 Version:    27.4.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.5.1
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-ai.exe
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.2-desktop.1
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.31.0-desktop.2
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.37
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-debug.exe
  desktop: Docker Desktop commands (Beta) (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-desktop.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.15.1
    Path:     C:\Users\Gili\.docker\cli-plugins\docker-scout.exe

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 27.4.0
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.153.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.31GiB
 Name: docker-desktop
 ID: 9cd2a19c-6a32-4004-9d22-c6f6db240e52
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Username: cowwoc
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  registry.docker.localhost:5000
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

[cowwoc]

cat /etc/docker/registry/config.yml from inside the registry container returns:

version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3

Notice there is no proxy directive instructing it to act as a pass-through proxy.

[cowwoc]

Nevermind. I figured it out.

The created registry works fine as a pass-through proxy. My misunderstanding was that I thought it was a full-fledged docker registry and I should be able to docker pull images out of it... You can't.

It turns out that the registry can only be used indirectly. Meaning, if you add:

"registry-mirrors": [
    "http://localhost:5000"
  ]

to ~/.docker/daemon.json and run docker pull nginx then your local docker engine will request the image from http://localhost:5000. If the local proxy has it, it'll return a copy; otherwise, it'll download a copy from dockerhub first.

So again, you can use the local instance as a proxy but you cannot interact with it directly.

If you are feeling generous, please add a little note to https://k3d.io/v5.8.1/usage/registries/#creating-a-registry-proxy-pull-through-registry letting people know this. I wasted 3 days (!!) trying to figure out what I was doing wrong. What a waste...

Anyway, aside from requesting a documentation update, feel free to close this issue. I consider it to be solved. Sorry for the false-alarm.

[cowwoc]

Hmm, this actually might be a bug in the docker registry: distribution/distribution#4477