Provide a means to specify that docker images from any registry should be retrieved from a given registry or the embedded registry #9590
Assignees
Milestone
Comments
|
@brandond saw your comment in the fork PR but thought it was better to respond here. This should be a pretty simple feature to implement from a k3s perspective. I guess the most complicated part is agreeing on how it should be configured. As has been stated Containerd does support a default mirror option which will forward requests for all registries to the configured mirror. Having that as an option sounds like a pretty reasonable thing to have. For historical context there is a reason why Spegel does not use the default mirror configuration to mirror all registries. The challenge with implementing a registry mirror is that the OCI distribution spec does not require the the pulling client to specify the registry of the image it is requesting from the mirror. As long as we are digest references this will not be an issue, but it will create issues when resolving tags to digests as it requires the complete image name reference. Initially Spegel would include a specific header in the mirror configuration to include the original registry in every request. This is for example how Dragonfly solves this problem. Eventually I found that Containerd in its image pull implementation will include a All that needs to be changed in Spegel is to allow setting a empty registry list to disable registry filtering. With that small change everything should work and k3s should be able to just use the default mirror configuration. |
|
Since Spegel uses a regex filter to watch containerd events, it would probably be as easy as just converting a registry name of The ns query parameter seems to be an undocumented standard, nexus for example supports that when used as a pull through cache. |
|
Yeah I missed that part when initially writing my reply, hence my reply. The solution is pretty simple, I could throw together a PR now to allow all registries when the list is empty. |
Merged
|
I just quickly threw together k3s-io/spegel@b8f4f0a, which would require users to specifically opt in to mirroring images from all registries by specifying |
|
I think that either way would work, setting no registries in the list or a wild card. Could we however merge my PR to change to using upstream Spegel to avoid dealing with more diffs in the fork. |
|
I ran into a couple issues while further bumping on top of the current main branch in #9599, see spegel-org/spegel@38a05d0...k3s-io:spegel:e300eecb1d56552b307fafb2557c6faa329e1459 |
This was referenced Mar 6, 2024
|
##Environment Details Infrastructure
Node(s) CPU architecture, OS, and version: Linux 5.14.21-150500.53-default x86_64 GNU/Linux Cluster Configuration: Config.yaml: ReproductionResults: previous release behavior $ sudo journalctl -u k3s | grep -i spegel $ k scale deploy wonky --replicas=9 //I've elected to do this to generate peer to peer traffic as the pod scales out across the nodes instead of the initial single node in the cluster. Works pretty good :) Spegel is nice from a performance improvement on pod startup time with the image being on the hosts. //New behavior for both mirrors and latest tag on images stripped out some noise and yes 777.77 is a fake ip $ kgp -A -o wide $ sudo cat /etc/rancher/k3s/registries.yaml $ sudo cat /etc/systemd/system/k3s.service.env |
|
Thank you! 🙏🏼 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
We're in an air-gapped environment and want to be able to tell k3s to pull all docker images from either a given private registry, or the k3s embedded registry introduced in January 2024. The current feature requires us to edit the file
/etc/rancher/k3s/registries.yamlon each machine in the cluster, adding amirrors:section, then adding an entry for each distinct docker registry amongst all docker images required in the air-gapped environment. After editing the file, we need to restartk3sfor the changes to take effect, which is also undesirable.Describe the solution you'd like
As @brandond described here, we'd like to have either a command-line argument and configuration setting that enables us to statically instruct k3s to get all docker images from either a designated private registry or the embedded registry. Alternatively, this could be implemented in the context of a default fallback registry notion so as to work nicely with default endpoint fallback feature, that is, if k3s can't find a docker image amongst any configured registry mirrors, check a default private registry (if so configured), then the embedded registry (if enabled).
Describe alternatives you've considered
Our only alternative is to manually edit or generate/regenerate a
registries.yamlfile with a tool or script to contain the distinct set of docker registry names in themirrors:section and then restartk3s.Additional context
The full github discussion of this topic can be found here.
Thank you for considering this feature request. 🙏🏼
The text was updated successfully, but these errors were encountered: