Support insecure-registries for container runtime running inside of kind container

[font]

The host that is running kind to set up kind clusters may want to create container images to be pulled by the container runtime (docker/containerd daemons) running inside of the kind-<name>-control-plane containers e.g. kind-1-control-plane. To simplify this, it would be great to have a way to easily configure the container runtime running inside the kind containers with insecure-registries in order to pull images from the host's insecure registry. This would simplify the local registry setup on the host to not require TLS.

For now, I have used the following workaround:

1. Run local insecure registry on host:

docker run -d -p 5000:5000 --restart=always --name registry registry:2

2. Create daemon.json inside of the kind container via docker exec:

docker exec kind-1-control-plane bash -c "cat <<EOF > /etc/docker/daemon.json
{
    "insecure-registries": ["172.17.0.1:5000"]
}
EOF

3. Send SIGHUP to docker daemon in kind container in order to reload config:

docker exec kind-1-control-plane bash -c 'kill -s SIGHUP $(pgrep dockerd)'

This works for now and then any container image to be pulled needs to be specified like so:

docker pull 172.17.0.1:5000/<imagename>:<tag>

[BenTheElder]

SGTM, looks like both cri-o and containerd support this as well so if we want to use those inside the container in the future this can still be supported.

We can add a config option to specify a list of insecure registries and write it through to the daemon config before we start the daemon.

One thought though, specifically for the case of using a registry running on the host where kind is running, probably we can avoid the user needing to know what IP kind will see the host as, otherwise this config will be brittle / non-portable.

https://dev.to/bufferings/access-host-from-a-docker-container-4099 looks like an option for that. perhaps we can have config like:

insecureLocalRegistryPort: 5000

and then images can be at host.docker.internal:5000/foo-image ?

cc @munnerz

This also seems related to #28

[padiazg]

Please, take in account also that there is the possibility of using a private registry with self signed certificates, and to use this you need also put the corresponding CA certificate in place.
Ex: /etc/docker/certs.d/<registry-ip>/ca.crt

Also take in account that we can use a private registry as a proxy, and that must be configured in daemon.json too.
Ex:

{
    "insecure-registries": [
       "<registry-ip>"
     ],
     "registry-mirrors": [
       "https://<registry-mirror-ip>"
     ]
}

We are using two private registries, both use self signed certificates (mostly to avoid using :5000 in the image label), one for our own created images and the other as proxy due that we are in a restricted network.

[BenTheElder]

I think certs can be injected using #62
Edit: technically the config could be too, but note that we may switch to containerd on the nodes.

[BenTheElder]

We're injecting a dockerd systemd dropin for proxy settings now, I think we can look at something similar for insecure registries. Something like kind config containing a list of these registries -> write dropins on the nodes. See also #340

[dims]

fyi, federation folks are using this right now https://github.com/kubernetes-sigs/federation-v2/blob/master/scripts/create-clusters.sh

[h0tbird]

Alternatively you can also do something like this:

➜  ~ cat test.sh
#!/bin/bash

TEMP_DIR=$(mktemp -d /tmp/cluster-api.XXXX)

cat << EOF > ${TEMP_DIR}/kind-config.json
kind: Config
apiVersion: kind.sigs.k8s.io/v1alpha2
nodes:
- role: control-plane
  extraMounts:
    - containerPath: /etc/docker/daemon.json
      hostPath: ${TEMP_DIR}/docker-daemon.json
      readOnly: true
EOF

cat << EOF > ${TEMP_DIR}/docker-daemon.json
{
  "insecure-registries": ["http://172.17.0.1:5000"]
}
EOF

kind create cluster --config ${TEMP_DIR}/kind-config.json

[BenTheElder]

note that overwriting the entire daemon.json is not ideal as we move off the docker-shim: #425 (comment)

I think we will need a first class option in kind to configure insecure registries. this should be easier to add to v1alpha3 config now.

[fspaniol]

Hi,

I'm trying to add a registry as insecure but it seems that my control-plane does not have the docker binary.. :(

This is what I'm executing:

docker exec -it kind-control-plane bash
root@kind-control-plane:/# docker
bash: docker: command not found
root@kind-control-plane:/#

kind version
v0.4.0

Am I doing something wrong? :)

Cheers,
Fernando

[BenTheElder]

@fspaniol the control plane switched to containerd since this issue was first open 😅

[font]

Updated the title to be more generic. 😄

[fspaniol]

Ah haha, that explains it.

Btw, my use case was that I was trying to follow the tutorial from kubebuilder using kind and I was using a private registry to push my images and when a pod tried to fetch any image, it was getting the x509 issue. The solution I found was to deploy a registry within kind and now every works fine :)

The guide can be found here.

ps: thanks so much for kind, it makes kubernetes usage so much easier <3

[BenTheElder]

@font thanks :-)

@fspaniol Thanks for the feedback, I appreciate it and I'm sure others will find those links very useful.

---

If anyone's interested in this issue, ideally I'd like to find a way to patch .toml files similar to kustomizing kubernetes yaml, that way we can just add the insecure registries we need on top of whatever existing config we have composably. I suspect people are typically writing this config file by hand currently...

[TrentonAdams]

kind started using containerd and none of the solutions here work anymore, how do I go about adding an insecure registry now?

[BenTheElder]

@TrentonAdams the guide mentioned in #110 (comment) is one option for now.

[TrentonAdams]

@BenTheElder Thanks.

[crossdot]

I've got an external insecure registry and deploying it within kind is not an option for me. Is there a way to bring it to work?

[brightzheng100]

yup, just submitted as #1119

[BenTheElder]

Thanks for catching these and the PR

…

On Thu, Nov 21, 2019, 00:36 Bright Zheng ***@***.***> wrote: yup, just submitted as #1119 <#1119> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#110?email_source=notifications&email_token=AAHADK5G43GFITYY3D6DEF3QUZCBBA5CNFSM4GDZ5OUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEZM2OY#issuecomment-556977467>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHADK3RET3X6VJJKTZDWITQUZCBBANCNFSM4GDZ5OUA> .

[yuzp1996]

The host that is running kind to set up kind clusters may want to create container images to be pulled by the container runtime (docker/containerd daemons) running inside of the kind-<name>-control-plane containers e.g. kind-1-control-plane. To simplify this, it would be great to have a way to easily configure the container runtime running inside the kind containers with insecure-registries in order to pull images from the host's insecure registry. This would simplify the local registry setup on the host to not require TLS.

For now, I have used the following workaround:

1. Run local insecure registry on host:

docker run -d -p 5000:5000 --restart=always --name registry registry:2

1. Create daemon.json inside of the kind container via docker exec:

docker exec kind-1-control-plane bash -c "cat <<EOF > /etc/docker/daemon.json
{
    "insecure-registries": ["172.17.0.1:5000"]
}
EOF

1. Send SIGHUP to docker daemon in kind container in order to reload config:

docker exec kind-1-control-plane bash -c 'kill -s SIGHUP $(pgrep dockerd)'

This works for now and then any container image to be pulled needs to be specified like so:

docker pull 172.17.0.1:5000/<imagename>:<tag>

Hi I follow your step but find there is no pid like dockerd and so how can I restart docker to reload the daemon.json.
I find all the pid in the kind node container but can not find any pid that I can kill.
Can you give me some suggestions?

Thank you very much

[BenTheElder]

These steps are outdated.
Please see the containerdConfigPatches mechanism used here instead https://kind.sigs.k8s.io/docs/user/private-registries/

[FredericLeroy]

It concerns private registry, not insecure registry, isn't it ?

[BenTheElder]

The same mechanisms / patch type are used to configure all registries.

…

On Thu, Jun 25, 2020, 01:13 FredericLeroy ***@***.***> wrote: It concerns private registry, not insecure registry, isn't it ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#110 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHADK6UHTBEYVDTHAYYGOTRYMBLJANCNFSM4GDZ5OUA> .

[yuzp1996]

These steps are outdated.
Please see the containerdConfigPatches mechanism used here instead https://kind.sigs.k8s.io/docs/user/private-registries/

Thank you very much

I accpect your advice in my helm chart

    kind: Cluster
    apiVersion: kind.x-k8s.io/v1alpha4
    containerdConfigPatches:
    - |-
    {{- range .Values.insecureRegistry }}
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."{{ . }}"]
        endpoint = ["http://{{ . }}"]
    {{- end }}

and it work well there is no more error when pull image from insecure registry

Thanks again

[marouanabbessi]

Another way is to run your your registry as a pod inside the cluster , with a manifest similar to the following :

apiVersion: v1
kind: Namespace
metadata:
  name: registry
  labels:
    app: backend
---
kind: Service
apiVersion: v1
metadata:
  name: registry
  namespace: registry
  labels:
    run: registry
spec:
  selector:
    app: registry
  type: NodePort 
  ports:
    - protocol: TCP
      port: 30007
      nodePort: 30007
      targetPort: 5000
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: registry
  namespace: registry
spec:
 accessModes:
 - ReadWriteOnce
 volumeMode: Filesystem
 resources:
  requests:
    storage: 50Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: registry
 namespace:  registry
 labels:
  app: registry
spec:
 replicas: 1
 selector:
  matchLabels:
    app: registry
 template:
  metadata:
    labels:
      app: registry
  spec:
    containers:
    - name: registry
      image: registry:2
      ports:
      - containerPort: 5000
        protocol: TCP
      volumeMounts:
      - name: storage
        mountPath: /var/lib/registry
      env:
      - name: REGISTRY_HTTP_ADDR 
        value: :5000
      - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
        value: /var/lib/registry
    volumes:
    - name: storage
      persistentVolumeClaim:
        claimName: registry

Note the following points :

• REGISTRY_HTTP_ADDR is set to 5000 (you can change this to any other port)
• The registry is exposed using a Nodeport service so that Kind worker nodes can reach the private registry to download the images, using the url registry.registry.svc:30007 (30007 is an explicit port used by the NodePort service)

In addition to the above the following must be done :

1 - Add the following configuration to the Kind config file :

containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.registry.svc:30007"]
    endpoint = ["http://registry.registry.svc:30007"]

2- After installing the kind cluster add the hosts file entry "127.0.0.1 registry.registry.svc" to all worker nodes in order for kind to resolve the in-cluster private registry; the following is a one-liner that achieves this :

for container in $(docker ps | grep kind-worker | awk '{print $1}');do docker exec -u 0 $container bash -c "echo '127.0.0.1 registry.registry.svc' >> /etc/hosts " ;done
`
3- In order to be able to push images to the in-cluster private registry :

• port-forward the in-cluster private registry as follows :

# Make sure to kill any dangling processes listening on that 
kill -9 `lsof -i TCP:30007 | awk '{print $2}' |tail -n 1` &> /dev/null  port
# Port-forward your in-cluster private registry
nohup kubectl port-forward -n registry services/registry 30007:30007 & &> /dev/null

• Add "127.0.0.1 registry.registry.svc" entry to your local /etc/hosts file

Hope this helps :)

[hamidsafdari]

I hope this saves some time and hair for the next guy. I needed to deploy an app at work on Kubernetes and either of the solutions "insecure registries" and "deploying the registry inside the cluster" were not options because I couldn't do them in production. I also didn't want to port-forward to the registry every time I had to push an image. I could finally nail this after a few days, and I have recorded my steps in a script that you can use. The script will do the following:

1. Create a private registry behind HTTPS
2. Create a kind cluster that uses and trusts your private registry
3. Deploys a sample app on your cluster

https://github.com/hamidsafdari/kind-private-registry

I've also described all the steps in the README in case something doesn't work, and you want to debug. I've done this on Arch Linux and I had docker, kind, kubectl installed already. Also, I am fully aware that KinD is for testing and development and I will eventually use something like k3s for production.

[BenTheElder]

See also: https://kind.sigs.k8s.io/docs/user/local-registry/

Yes -- KIND should probably not be exposed to the internet in production, see the config docs. You can try for example kubeadm directly instead.

[hamidsafdari]

I did try that and I believe this thread is about the problems with that approach. The instructions in that link set up an insecure registry.

…

On Mon, 7 Nov 2022, 20:00 Benjamin Elder, ***@***.***> wrote: See also: https://kind.sigs.k8s.io/docs/user/local-registry/ Yes -- KIND should probably not be exposed to the internet in production, see the config docs. You can try for example kubeadm directly instead. — Reply to this email directly, view it on GitHub <#110 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHLI5VNENCXQ3YGPKVGQCTWHEN7RANCNFSM4GDZ5OUA> . You are receiving this because you commented.Message ID: ***@***.***>

[BenTheElder]

I did try that and I believe this thread is about the problems with that
approach. The instructions in that link set up an insecure registry.

Er ... The title of this issue is:

Support insecure-registries for container runtime running inside of kind container

With this in the issue description:

To simplify this, it would be great to have a way to easily configure the container runtime running inside the kind containers with insecure-registries in order to pull images from the host's insecure registry. This would simplify the local registry setup on the host to not require TLS.

This issue is not about setting up a secure registry, because as you noted KIND is not meant to be used in that sort of situation. This issue was about setting up the local insecure registry to be usable within the cluster.

The linked local registry guide is what closed this issue. Conversation about remote secure registries is off-topic for the original issue filed here.

This other guide:
https://kind.sigs.k8s.io/docs/user/private-registries/#use-a-certificate has some docs for using production / secured registries.

[hamidsafdari]

You're right. I'm sorry. I just got so desperate trying to make this work I didn't read the title of the issue. I've used all the links you have shared, using a certificate and KinD's local registry setup. I guess the only thing I could have done was figure out how to make k8s use the insecure registry. My assumption was that it wouldn't. Partly because I read somewhere that KinD has moved away from using docker and I didn't bother to see how to make the new container runtime pull from the insecure registry. I also thought k8s uses https as a security measure and there's no way to make it use http. Also, everything else I found online mentioned setting up a secure registry that I thought that's how it should be done. Like docker's guide on how to set up a secure registry, including the generation of a certificate:
https://docs.docker.com/registry/deploying/
and the Kubernetes instructions on how to pull images from private registries:
https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

I would probably use an insecure registry next time for faster set up.

I also found that k3d accepts an argument during cluster creation for setting up a local registry but k3d was very resource heavy for my laptop. Sticking with KinD for now.

[maomaoliu]

Hi @BenTheElder, I followed the doc Local Registry, which is using plugins."io.containerd.grpc.v1.cri", and when deploy image, it is still not working ..

Could you have a look and give some suggestion, please?

Error in deployment:

Failed to pull image "kind-registry:5000/service-a:latest": 
rpc error: code = Unknown desc = failed to pull and unpack image "kind-registry:5000/service-a:latest": 
failed to resolve reference "kind-registry:5000/service-a:latest": 
failed to do request: Head "https://kind-registry:5000/v2/service-a/manifests/latest": 
http: server gave HTTP response to HTTPS client

config yaml looks like

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
containerdConfigPatches:
  - |-
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:5001"]
      endpoint = ["http://kind-registry:5000"]
nodes:
  ...
  ...

[aojea]

failed to resolve reference "kind-registry:5000

you may be using podman without dns service enable? it fails to resolve the container by name 🤔

[maomaoliu]

failed to resolve reference "kind-registry:5000

you may be using podman without dns service enable? it fails to resolve the container by name 🤔

no issue on DNS, can lookup dns record in pods.
The reason is in the last line http: server gave HTTP response to HTTPS client

[essobedo]

@maomaoliu I met the same issue, and I had to modify the script to make it work.

Here are my changes:

...
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."${reg_name}:5000"]
    endpoint = ["http://${reg_name}:5000"]
...
  localRegistryHosting.v1: |
    host: "localhost:${reg_port}"
    hostFromClusterNetwork: "${reg_name}:5000"
...

As you can see, I changed the host of the registry, this way the images are pulled using the HTTP protocol instead of HTTPS.

I don't know if what I'm doing is a misuse but at least it helped me to fix my problem.

[BenTheElder]

No that's fine.
FYI unfortunately we're going to have to migrate to a different way of configuring these in the future due to containerd changes #2875

[essobedo]

@BenTheElder Shall I submit a PR to update the script before the future changes or it is not needed?

[BenTheElder]

I think switching the script to http is a reasonable change 👍

We can still do this in the future but the way we plumb the config through will be different, and eventually it will require breaking changes to that aspect I think.

[essobedo]

@BenTheElder here is the related PR #3161, let me know if it is good enough

[BenTheElder]

For the record: we're already pulling with HTTP inside of the cluster:

kind/site/static/examples/kind-with-registry.sh

Line 20 in 5cb68df

endpoint = ["http://${reg_name}:5000"]

[liverbirdkte]

Hi @BenTheElder, When I use kind-registry:5000 in my cluster application, it simply doesn't work, here is the error log:

Warning Failed 50s (x3 over 94s) kubelet Failed to pull image "kind-registry:5000/pilot:latest": rpc error: code = Unknown desc = failed to pull and unpack image "kind-registry:5000/pilot:latest": failed to resolve reference "kind-registry:5000/pilot:latest": failed to do request: Head "https://kind-registry:5000/v2/pilot/manifests/latest": http: server gave HTTP response to HTTPS client

I think this is because kubelet has no idea that this is an insecure registry, so I add a mapping in /etc/containerd/certs.d/kind-registry:5000/hosts.toml to tell containerd to use http for kind-registry:

[host. "http://kind-registry:5000"]

And my application pulls off the image successfuly.

Do you think this change is needed? If so I could submit a PR to fix the script.

[BenTheElder]

When kubelet is pulling an image and not some in-cluster application itself reaching an image you should use localhost:5001 and the registry config will remap it to http on the registry container to match the way you name and push it from the host

please see the current registry docs for a sample of pushing and deploying a container correctly

[BenTheElder]

Kubelet doesn't know about any of this, but you need to use localhost:5001 because that's what we told containerd to remap to match the host.

[BenTheElder]

https://kind.sigs.k8s.io/docs/user/local-registry/

[liverbirdkte]

Thanks for the info, Ben.

Let me provide more context, I followed the local registry doc to set up kind and local registry. I plan to install Istio in that cluster using the local registry, here is the yaml file for Istioctl:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
    profile: demo
    hub: kind-registry:5000
    tag: latest
    meshConfig:
      outboundTrafficPolicy:
        mode: REGISTRY_ONLY
      defaultConfig:
        proxyMetadata:
          BOOTSTRAP_XDS_AGENT: "true"

I tried with localhost:5001 but connection to this registry failed in starting istiod pod. Then I switched to kind-registry:5000 and got "http: server gave HTTP response to HTTPS client" error I mentioned before. This is why I have to add an entry in containerd registry to map kind-registry:5000 to http.

So I think for cases to use kind-registry, a http registry mirror is needed.