Feature: Docker private registry support

[claussa]

What are you trying to do?

I want to use testcontainers with an image store on a private registry. Correct me if I am wrong, It is currently not possible on testcontainers-python but it is possible on testcontainers-java.

Why should it be done this way?

Example on how it is done on testcontainers-java: https://github.com/testcontainers/testcontainers-java/blob/994b385761dde7d832ab7b6c10bc62747fe4b340/core/src/main/java/org/testcontainers/utility/RegistryAuthLocator.java#L37

[alexanderankin]

thank you for the example - before reading that code, i never understood what this meant - i suppose in ci environments you have to get creative on how to pull your images, so that makes sense. if i understand correctly, this is just passing an auth parameter for pulling an image if it doesnt exist (and workaround is then just to pull the image ahead of time?)

[Tranquility2]

1. Will this need support for more than one auth server? or basic support is for a single config?
   the basic example is :

{
    "auths": {
        "registry.example.com:5000": {
            "auth": "bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ="
        }
    }
}

note the plural auths :)

2. I assume this does not include ~/.docker/config.json as its another feat all by itself right?

[Tranquility2]

Good news. I think I got it working for a single registry:

 echo "$DOCKER_AUTH_CONFIG"
{
        "auths": {
                "localhost:5000": {
                        "auth": "..."
                }
        },
}

>>> from testcontainers.redis import RedisContainer
>>> with RedisContainer() as redis_container:
...     redis_client = redis_container.get_client()
...
Pulling image testcontainers/ryuk:0.7.0
Container started: bb76132ce865
Waiting for container <Container: bb76132ce865> with image testcontainers/ryuk:0.7.0 to be ready ...
Pulling image localhost:5000/redis:latest
Container started: a48fd2609171
Waiting for container <Container: a48fd2609171> with image localhost:5000/redis:latest to be ready ...
Waiting for container <Container: a48fd2609171> with image localhost:5000/redis:latest to be ready ...
Waiting for container <Container: a48fd2609171> with image localhost:5000/redis:latest to be ready ...

[alexanderankin]

is there a PR to look at or is that just bulit-in functionality from docker-py?

[Tranquility2]

I have the code (I tried to keep it to something very minimal), not a built in as docker-py only supports simple login and you need to pass all the params (feel free to correct me if I missed something), so needed to unpack and decode the data.
In any case I need to prepare it for a PR, please tell me if my assumptions on #562 (comment) are correct so I'll know how to proceed.
Do we assume you only want to work with the private registry if the env var is available? if thats not the case the code needs to be a bit more complicated and on each Container run (docker pull) we will need to check the prefix of the image tag and login to the specific server.

[Tranquility2]

Feel free to assign me :)

[alvaromerinog]

Hello there! I am using amazon-ecr-credential-helper to login to a private AWS ECR registry but the line 103 of core/testcontainers/core/utils.py is raising an AttributeError exception because auth_config_dict is None. The value of my DOCKER_AUTH_CONFIG environment variable is:

{"credHelpers":{"<aws_account_id>.dkr.ecr.<region>.amazonaws.com": "ecr-login"}}

I suppose that this problem could be solve in the line 102 of that file with with a dict as default value of the key auths:

auth_config_dict: dict = json.loads(auth_config).get("auths", {})

This solution would break in the line 193 of core/testcontainers/core/docker_client.py because it could not get the first element of an empty list. The login method of DockerClient could be like this to avoid an IndexError:

    def login(self, docker_auth_config: str) -> None:
        """
        Login to a docker registry using the given auth config.
        """
        auth_config = parse_docker_auth_config(docker_auth_config)
        if auth_config:
            first_auth_config = auth_config[0] # Only using the first auth config  
            login_info = self.client.login(**first_auth_config._asdict())
            LOGGER.debug(f"logged in using {login_info}")

Are you agree with this solution? @Tranquility2 @alexanderankin

Thank you so much for your contribution to this repository. Have a great day!

[alexanderankin]

hi @alvaromerinog can you provide a complete code example that can verify that this functionality can work as expected, i can test this and figure out how to add tests for this. absent that, I dont believe that docker-py actually supports this - https://github.com/search?q=repo%3Adocker%2Fdocker-py%20login&type=code - at least from first glance. otherwise, please feel free to use the login helper yourself to get a token, set that as your environment variable, and use the DOCKER_AUTH_CONFIG functionality that exists today.

[alvaromerinog]

Hello @alexanderankin! I may have misunderstood your comment, but my intention is not to implement the ECR login helper using docker-py. In my pipeline I have a preconfigured DOCKER_AUTH_CONFIG variable generated previously by the ECR login helper. This variable has not the auths key in the dumped dictionary so the parse_auth_docker_config function fails when it tries to iterate the items of None. I have made the PR #646 to understand and solve the problem I have found. Any suggestion is welcome! Thank you so much!

[Tranquility2]

I think we can assume this is a different use case (based on https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#use-credential-helpers)
credential-helpers also uses DOCKER_AUTH_CONFIG but it has a different scheme:

{
  "credHelpers": {
    "<aws_account_id>.dkr.ecr.<region>.amazonaws.com": "ecr-login"
  }
}

vs

{
    "auths": {
        "registry.example.com:5000": {
            "auth": "bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ="
        }
    }
}

btw apparently you can also do:

{
  "credsStore": "ecr-login"
}

It looks like other mechanisms also use this file to control auth

@alvaromerinog are you just trying to get testcontainers-python to ignore the extra data? if that is the case, #646 seems in the right direction but maybe we should switch a logic so that in that case (when we actually don't have the auth data) we will never try to login (around the DockerClient init).
What do you think?
In any case will be happy to help if needed :)

[Tranquility2]

Another alternative is to follow testcontainers-java
https://github.com/testcontainers/testcontainers-java/blob/994b385761dde7d832ab7b6c10bc62747fe4b340/core/src/main/java/org/testcontainers/utility/RegistryAuthLocator.java#L141
and implement all the use cases:

• helper
• credsHelper
• base64 encoded auth (This is the one we currently have)

This is not trivial as we will need something like runCredentialProvider (That basically uses runCredentialProgram)

[Tranquility2]

@alexanderankin I also have an idea on how we can be more visible regarding this and on the same time avoid the problem @alvaromerinog is facing.
I'll try and create a PR asap.

[alexanderankin]

if testcontainers-java implements invoking credHelpers as a separate process then I am okay with an implementation of that here as well.

[Tranquility2]

I need to do some investigation regarding the actual process we will need to invoke.
Regarding separate process, I always feel a bit uncomfortable just calling other programs in python (from a security preceptive) but I guess we can make that work.

On the bright side, I created #647, I think it will address this issue + adds more clarity on the status + help us prepare for the next steps, please have a look.

[bonelli]

Hello everybody.

I don't know if this feat request is intended also to add the possibility to have a default prefix for Docker Hub images, in order to use a private Docker Registry configured as proxy cache.

In testcontainers-java this default docker hub proxy is configured with an env var: TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX

This is super nice for not flooding Docker Hub with pull requests on CI/CD environments, I already use it in Java and I'd like it in python too

[jonesbusy]

Same issue here. The testcontainers-python is missing support for TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX

Because of this we cannot pull from Artifactory and get rate limit from dockerhub

java and node implementation that we also use support the env var