need to support custom CA certificates and proxy settings

[ChetRHosey]

We use a transparent proxy which does MITM inspection of some HTTPS traffic. This means that some HTTPS connections are re-signed by an organizational CA rather than the original CA.

This interferes with ansible-builder, as it's not clear how we would add the custom CA root(s) to the builder image, and it fails to validate the connection to Galaxy:

#8 0.851 ERROR! Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/': <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1108)>
#8 ERROR: executor failed running [/bin/sh -c ansible-galaxy collection install $ANSIBLE_GALAXY_CLI_COLLECTION_OPTS -r requirements.yml --collections-path /usr/share/ansible/collections]: exit code: 1

Full output follows.

Some of our sites require explicit HTTP/HTTPS proxy settings, so ideally it would be possible to set those as well.

$ ansible-builder --version
0.4.0
$ ansible-builder build
Running command:
  docker build -f context/Dockerfile -t ansible-execution-env:latest context
#1 [internal] load build definition from Dockerfile
#1 sha256:da308b69c518bf933c2fe118954bb242105f66dbab8e01c524ce9c0149d68625
#1 transferring dockerfile: 517B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:5a456dc77426d604b098c2d1d21b47a02ba4e7b7ec117389ab91b07a62a31324
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for quay.io/ansible/ansible-runner:devel
#3 sha256:a2ee052f444f48b483ef7462f9f15dc496066ca125add16c92e38bb837b6dd2b
#3 DONE 0.6s

#4 [1/5] FROM quay.io/ansible/ansible-runner:devel@sha256:9873fcce94e9ac5eea49a6a8be051271d1eaf37269bcb7cde0e43a045ef7bbbb
#4 sha256:753a06865a330ab0144695c33d3db1821431ab90998e5e86162a89f396accf74
#4 DONE 0.0s

#9 [internal] load build context
#9 sha256:6b5546cd88b98a90418e016afd6fcc2e1b9e7db9cd9abf195d0aee0db43c718e
#9 transferring context: 70B done
#9 DONE 0.0s

#5 [2/5] ADD _build /build
#5 sha256:3dddbf34e726feebd74c3ded7df14ac1573b46ce0bfecfda18679dced81fd8fa
#5 CACHED

#6 [3/5] WORKDIR /build
#6 sha256:a3ada9751c5a39677262d706c33282b755d9bd7f7c0cbd1fedc8a7072c309dd0
#6 CACHED

#7 [4/5] RUN ansible-galaxy role install -r requirements.yml --roles-path /usr/share/ansible/roles
#7 sha256:b8fe3e7b811c725ad02bb7a2a66423213ceaa32949adbd32ed9ffced188e517d
#7 CACHED

#8 [5/5] RUN ansible-galaxy collection install  -r requirements.yml --collections-path /usr/share/ansible/collections
#8 sha256:b6e4d69fc25adac6e4cf24057628907904f2615bb026c148f86dfa18da6fb69c
#8 0.619 Starting galaxy collection install process
#8 0.619 Process install dependency map
#8 0.851 ERROR! Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/': <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1108)>
#8 ERROR: executor failed running [/bin/sh -c ansible-galaxy collection install $ANSIBLE_GALAXY_CLI_COLLECTION_OPTS -r requirements.yml --collections-path /usr/share/ansible/collections]: exit code: 1
------
 > [5/5] RUN ansible-galaxy collection install  -r requirements.yml --collections-path /usr/share/ansible/collections:
------
executor failed running [/bin/sh -c ansible-galaxy collection install $ANSIBLE_GALAXY_CLI_COLLECTION_OPTS -r requirements.yml --collections-path /usr/share/ansible/collections]: exit code: 1

An error occured (rc=1), see output line(s) above for details.

I added lines to the generated Dockerfile to add the organizational certs and run update-ca-trust, and then ran the Docker build manually. The image build then succeeded. Ideally there would be a way to pass the organizational CA and proxy settings into ansible-builder for automatic inclusion in the build process.

$ cat context/Dockerfile
ARG ANSIBLE_RUNNER_IMAGE=quay.io/ansible/ansible-runner:devel
ARG PYTHON_BUILDER_IMAGE=quay.io/ansible/python-builder:latest

FROM $ANSIBLE_RUNNER_IMAGE as galaxy

ARG ANSIBLE_GALAXY_CLI_COLLECTION_OPTS=
ADD _build /build

COPY ca-certificates/ /etc/pki/ca-trust/source/anchors
RUN update-ca-trust

WORKDIR /build
RUN ansible-galaxy role install -r requirements.yml --roles-path /usr/share/ansible/roles
RUN ansible-galaxy collection install $ANSIBLE_GALAXY_CLI_COLLECTION_OPTS -r requirements.yml --collections-path /usr/share/ansible/collections

[saxx0n]

We have the same problem, but are not using a transparent proxy. ansible-builder does not use environmental variables for proxy settings, and without them, when attempting to build we get:

Starting galaxy collection install process
Process install dependency map
ERROR! Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/': <urlopen error [Errno 99] Cannot assign requested address>
The command '/bin/sh -c ansible-galaxy collection install $ANSIBLE_GALAXY_CLI_COLLECTION_OPTS -r requirements.yml --collections-path /usr/share/ansible/collections' returned a non-zero code: 1

After viewing the proxy logs, it is clear that no attempt is being made to use the proxy. When the exact same command/config is run from a box with direct internet access it functions correctly.

[AlanCoding]

What environment variables would we need to forward into the container? The pattern we need is pretty off-the-shelf, we just apply it to the galaxy stage in our case.

podman build --build-arg SOME_NAME_AS_BUILD_ARG

then in the Containerfile

FROM $EE_BASE_IMAGE as galaxy
ARG ANSIBLE_GALAXY_CLI_COLLECTION_OPTS=
ARG SOME_NAME_AS_BUILD_ARG=
ENV SOME_NAME_AS_ENV_VAR=$SOME_NAME_AS_BUILD_ARG

That doesn't address the cases where something needs to be mounted.

For this issue, I would like some help pinning down exactly what would need to be passed.

[ghyde]

You can already pass HTTP Proxy environment variables with podman build --http-proxy ... (reference). So, I don't think there would be a need to add ENV entries in the Containerfile. Instead, we could add an --http-proxy option to ansible-builder, and it would simply append it to the podman build ... command when it executes.

[ChetRHosey]

The issue isn’t just the proxy variables themselves, but also that some corporate proxies (even some transparent ones, which intercept traffic without being explicitly specified) will inspect and reencrypt traffic using an organization-specific certificate.

This means that, in order to fully work in proxies environments, it’s necessary to allow custom CA roots to be added to the container’s trust store. Pip and ansible-galaxy would both need to be able to trust an organizational CA in order to work. It seems unlikely that podman’s flag would also inject the outer system’s trust settings as different container images would need them to be injected in different places and formats.

[stanislav-zaprudskiy]

Customizable prepend and append configs only apply to the final image in Dockerfile being generated by ansible-builder. What would be great to have are the configuration parameters to define prepend and append (probably named differently, or just existing in a separate config section which is not additional_build_steps) for other images in the multi-stage build too (e.g. for FROM $EE_BASE_IMAGE as galaxy and FROM $EE_BUILDER_IMAGE as builder images) - as most of the magic when downloading requirements.txt, etc from custom sources happens there. This could be worked around by using custom overridden images in EE_BASE_IMAGE and EE_BUILDER_IMAGE args, but that requires to build those images at first.

[ikke-t]

I just got pointed to this issue. For the next ones running into this, I document here I went through the issue while working in such transparent firewall env at customer. Sorry I don't have access to files, but I try to describe the steps.

If you work on tower host, use awx account so all images are visilbe and referencable as localhost:imagename:tag in tower configs. Naturally later on you want the images to be rebuilt automatically by OpenShift buildconfig triggering on any RH image update, and served from a corporate repository (Quay).

1. Insert Company CA into builder image

Problem is ansible-builder fails to download the EE images due https certs being wrong. For this I rebuilt the builder image and added CA certs in there. This is normal buildah/podman build using Containerfile. Now from top of my head the containerfile is about the following like:

FROM ansible-automation-platform-20-early-access/ansible-builder-rhel8

COPY cp *.pem /usr/share/pki/ca-trust-source/anchors/
RUN update-ca-trust

build the container like:

buildah bud -t ansible-builder-rhel8:certs Containerfile

2. Use the new builder image

Use the options you normally would use with ansible-builder, but add the reference to above image:

ansible-builder EE_BUILDER_IMAGE=ansible-builder-rhel8:certs 

3. Add the certs into EE image

While you customize your EE images, it might help you to add the certs in there too. This is done by modifying the ansible-builder definition file additional steps, like e.g:

additional_build_steps: 
  prepend: |
    COPY cp *.pem /usr/share/pki/ca-trust-source/anchors/
    RUN update-ca-trust

I don't have access to such system right now, but I recall it tries to copy those *.pem from relative path ./context/*.pem. It will create that file when if fails at first, so you get the name. Normally you'd do first mkdir context; cp mycerts/*.pem ./context. If that was the name, I'm not sure.

4. Add the EE into Tower

At this point you can create a new reference to EE image into tower. Go to Execution environments, and either change an existing default EE or create a new one pointing to e.g. localhost/my-ee:latest.

Now for you to know, I failed next due yum not trusting the given CA, as it only trusts RH CA. Then I realized I need to point yum to internal satellite or rpm repo for that to work

Good luck, hopefully this saves someone's time in such an evil MITM environment :) .

[ChetRHosey]

As a modification to step 2 above, we reference the customized images in execution-environment.yaml. This keeps the ansible-builder execution simple since we don't have to remember to pass the image names.

build_arg_defaults:
  ANSIBLE_RUNNER_IMAGE: ansible-runner:with-root-ca
  PYTHON_BUILDER_IMAGE: python-builder:with-root-ca

[MallocArray]

I just ran into the same issue. I have a corporate proxy that needs to have environment variables set to access
ENV https_proxy http://192.168.200.1

I can add this to the additional_build_steps but the ansible-builder process fails at the builder step when trying to install python requirements.txt and it has no knowledge of the proxy. The additional_build_steps additions don't happen until the next step in the build

Manually adding the ENV line after the "FROM $EE_BUILDER_IMAGE as builder" line works as long as I manually call the container process (docker in my case) and if I re-run ansible-builder my changes are removed

Having ansible-builder either have options to accept proxy settings, or support additional options in execution-environment.yml to allow for adding settings such as this to the builder step is needed to keep us from having to run ansible-builder, hand change the Dockerfile, and then run docker/podman

[friendlypenguin]

same problem here when using ansible-builder from RedHat Automation Platform 2.1 repositories.
used @ikke-t solution

[rseabra]

We worked around this by defining environment variables before calling podman build:
export all_proxy=... http_proxy=... https_proxy=.... no_proxy=....

As for certificates, we use sed to edit the ContainerFile:

1. copy the certificates into context/
2. run sed as: sed -i -e "/as galaxy/a COPY *.crt /etc/pki/ca-trust/source/anchors\nRUN update-ca-trust" ContainerFile

[samweisgamdschie]

@rseabra can you give us more details about how you brought the *_proxy variables into the build container? Do you call podman build manually? Or do you meant ansible-builder build?

[rseabra]

Hi, @samweisgamdschie , I'm sorry, I only noticed the mention now.

I gave up on ansible-builder build and indeed use podman login, podman build, podman push, podman logout.

[Shrews]

A new version of builder is due to be released within the next few weeks that will have new features that should allow you to accomplish what you need here (copy certs, run update-ca-trust command). This code has been merged to the devel branch so you are welcome to experiment with that and help us to identify bugs.

The tl;dr of it is that the new version 3 of the execution environment file (docs here) has new sections to allow you to add any files to the build context, and new insertion points that allow you to add custom instructions before and after any of the build phases.

[rseabra]

Great news, it's not very easy for me to test the devel branch as we're running on a strict Red Hat Satellite based setup, but I'll keep an eye out for the new release!

[Akasurde]

With Ansible Builder v3, you can use any base image, copy additional files, and set ENV variables. Please give it a try. Here is the link to the option documenation. Thanks.

[Akasurde]

Closing due to no activity.