Support of local environmental variable on Ansible Controller that populated from ansible variables for kubectl connection plugin

[yurnov]

SUMMARY

kubernetes.core.kubectl connection plugin can be used to execute the content of plays inside pods in the Kubernetes cluster, it's easy to use an efficient tool. But sometimes clusters require local environmental variables for authorization, i.e. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for authorization, even we have a kubeconfig.

So, it's will be nice to have support of local environment variable for kubectl connection plug-in. Importantly, I talking about the local variable that required for kubectl binary (technically aws binary in case of AWS, but other cases may require another enviromental variable for another cluster type), not for remote enviromental variable that is natively supported by Ansible.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

kubernetes.core.kubectl connection plugin

ADDITIONAL INFORMATION

In some cases kubectl require not only kubeconfig file, but some additional credentionals (i.e. platform/cloud specific) for anothentication/authorization, and enviromental variable is conviniant place to store them, i.e. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in case of AWS EKS with kubeconfig that invoke aws binary. AWS is only one example, the similar can be conviniant for another cloud providers as well.

It's not a issue in case if Ansible is used to manage something just in one cluster, but in case of multiple clusters mentioned in inventory, kubectl connection plug can't be used.

So, this issue it for request support of local environment variable, to have working something like:

- hosts: cluster1:cluster2
  gather_facts: no
  any_errors_fatal: yes
  vars:
    ansible_connection: "kubernetes.core.kubectl"
    ansible_kubectl_namespace: "test"
    ansible_kubectl_config: "{{ kubeconfig }}"
    ansible_kubectl_pod: "shell"
    ansible_kubectl_container: "shell"
    ansible_kubectl_local_env:
       AWS_ACCESS_KEY_ID: "{{ aws_access_key_id }}"
       AWS_SECRET_ACCESS_KEY: "{{ aws_secret_access_key }}"
  environment:
    TEST_ENV1: value1
    TEST_ENV2: value2

  tasks:
  - name: test
    ansible.builtin.shell: env
    register: result
  - debug:
      var: result.stdout_lines

Expected result for this snippet is getting enviromental variable on running container in two clusters.
Actual result (as on version 3.0.1 of kubernetes.core collection) is failing connection at least to one cluster.

It's important specifically for kubernetes.core.kubectl to have ability to use same playbook in multi-cluster (and multi public cluster provider) scenario.

Update:

to be fully clear, this issue is request of implementation support of:

    ansible_kubectl_local_env:
       AWS_ACCESS_KEY_ID: "{{ aws_access_key_id }}"
       AWS_SECRET_ACCESS_KEY: "{{ aws_secret_access_key }}"

from a example above, and it's a local environmental variable on the ansible controller, not on on target host

[yurnov]

I tried to test this way with test playbook, but it doesn't work:

root@ubuntu-shell:/# cat test2.yaml
- hosts: localhost
  gather_facts: no
  any_errors_fatal: yes
  vars:
    ansible_connection: "kubectl"
    ansible_kubectl_namespace: "test"
    ansible_kubectl_config: "/.kube/config"
    ansible_kubectl_pod: "ubuntu"
    ansible_kubectl_container: "ubuntu"
    ansible_kubectl_local_shell_args: "EKS_TOKEN=test"
  environment:
    TEST_ENV1: value1
    TEST_ENV2: value2

  tasks:
  - name: test
    ansible.builtin.shell: env
    register: result
  - debug:
      var: result.stdout_lines
root@ubuntu-shell:/# ansible-playbook -vvv test2.yaml
ansible-playbook [core 2.14.5]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.10/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (/usr/bin/python3)
  jinja version = 3.1.3
  libyaml = True
No config file found; using defaults
host_list declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
script declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
auto declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
yaml declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
ini declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
toml declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: test2.yaml ***************************************************************************************************************************
1 plays in test2.yaml

PLAY [localhost] *******************************************************************************************************************************

TASK [test] ************************************************************************************************************************************
task path: /test2.yaml:16
redirecting (type: connection) ansible.builtin.kubectl to kubernetes.core.kubectl
<127.0.0.1> ESTABLISH kubectl CONNECTION
<127.0.0.1> EXEC ['/usr/bin/bash', '-c', 'EKS_TOKEN=test', '', '&&', '', '/usr/local/bin/kubectl', '-n', 'test', '--kubeconfig', '/.kube/config', 'exec', '-i', 'ubuntu', '-c', 'ubuntu', '--', '/bin/sh', '-c', "/bin/sh -c 'echo ~ && sleep 0'"]
<127.0.0.1> EXEC ['/usr/bin/bash', '-c', 'EKS_TOKEN=test', '', '&&', '', '/usr/local/bin/kubectl', '-n', 'test', '--kubeconfig', '/.kube/config', 'exec', '-i', 'ubuntu', '-c', 'ubuntu', '--', '/bin/sh', '-c', '/bin/sh -c \'echo "`pwd`" && sleep 0\'']
<127.0.0.1> EXEC ['/usr/bin/bash', '-c', 'EKS_TOKEN=test', '', '&&', '', '/usr/local/bin/kubectl', '-n', 'test', '--kubeconfig', '/.kube/config', 'exec', '-i', 'ubuntu', '-c', 'ubuntu', '--', '/bin/sh', '-c', '/bin/sh -c \'( umask 77 && mkdir -p "` echo ~/.ansible/tmp `"&& mkdir "` echo ~/.ansible/tmp/ansible-tmp-1713438683.4774194-5211-164264421022768 `" && echo ansible-tmp-1713438683.4774194-5211-164264421022768="` echo ~/.ansible/tmp/ansible-tmp-1713438683.4774194-5211-164264421022768 `" ) && sleep 0\'']
fatal: [localhost]: FAILED! => {
    "msg": "failed to resolve remote temporary directory from ansible-tmp-1713442333.866006-5231-217269445930807: `( umask 77 && mkdir -p \"` echo ~/.ansible/tmp `\"&& mkdir \"` echo ~/.ansible/tmp/ansible-tmp-1713442333.866006-5231-217269445930807 `\" && echo ansible-tmp-1713442333.866006-5231-217269445930807=\"` echo ~/.ansible/tmp/ansible-tmp-1713442333.866006-5231-217269445930807 `\" )` returned empty string"
}

NO MORE HOSTS LEFT *****************************************************************************************************************************
PLAY RECAP *************************************************************************************************************************************localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

[yurnov]

Hi @abikouo and @Akasurde, could you, as active contributor to this repo, please check this issue?

[gravesm]

Why wouldn't you just do:

- hosts: localhost
  gather_facts: false
  vars:
    ansible_connection: kubectl
    ansible_kubectl_namespace: default
    ansible_kubectl_pod: mypod
    ansible_kubectl_container: ubuntu
  tasks:
    - ansible.builtin.shell: echo $TEST_ENV
      register: result
      environment:
        TEST_ENV: value

    - debug:
        var: result

[avelytchko]

ansible_connection: kubectl

The plugin requires a successful connection before passing any tasks and environment variables, but AWS environment variables are necessary for the connection, resulting in a failure. We encountered an unreachable state, causing any task to fail immediately. EKS kubeconfig relies on AWS CLI for RBAC, so AWS environment variables must be exposed before ansible_connection: kubectl execution. Unfortunately, we're unable to expose these variables on the host itself, so we're looking to accomplish this solely through Ansible.

[yurnov]

Hi @gravesm,

it works in general cases, and TEST_ENV will be populated into target container after connection, but my ticket is for local (not target) environmental variable that requried for esbalishing connection and need to be set on local (ansible controller) before connection will be established in general case. That environmental variable required for authentication for connection isfelf.

[gravesm]

Sorry, maybe I'm not understanding your problem. The connection plugin is just spawning a kubectl child process and will inherit whatever environment your initial ansible process has. If you just need, for example, AWS_ACCESS_KEY_ID in the environment that the kubectl process is run in, just make sure it's set in whatever environment you are running ansible-playbook.

[yurnov]

Hi @gravesm, you are correct, this connection plugin just uses kubectl binary, and the local env variable is inherited, and playbook like that:

- hosts: cluster1
  gather_facts: no
  any_errors_fatal: yes
  vars:
    ansible_connection: "kubernetes.core.kubectl"
    ansible_kubectl_namespace: "test"
    ansible_kubectl_config: "{{ kubeconfig }}"
    ansible_kubectl_pod: "shell"
    ansible_kubectl_container: "shell"
  environment:
    TEST_ENV1: value1
    TEST_ENV2: value2

  tasks:
  - name: test
    ansible.builtin.shell: env
    register: result
  - debug:
      var: result.stdout_lines

works fine when I have local environment variable required for kubectl, i.e. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY that mentioned earlier, or any other that is required by kubectl in specific case. But it works perfectly when I have only one host (kubernetes cluster in case on kubernetes.core.kubectl), but if I have a few hosts with different credentials, it will not work.

Please have an example:

root@ubuntu-shell:/# cat inventory.ini
[k8s]
foo.example.com ansible_connection=kubernetes.core.kubectl  kubeconfig=/root/.kube/foo.example.com.config  aws_access_key_id=foo.id aws_secret_access_key=foo.key
bar.example.com ansible_connection=kubernetes.core.kubectl  kubeconfig=/root/.kube/bar.example.com.config  aws_access_key_id=bar.id aws_secret_access_key=bar.key
baz.example.com ansible_connection=kubernetes.core.kubectl  kubeconfig=/root/.kube/baz.example.com.config
root@ubuntu-shell:/# cat playbook.yaml
---
- hosts: k8s
  gather_facts: no
  any_errors_fatal: yes
  vars:
    ansible_connection: "kubernetes.core.kubectl"
    ansible_kubectl_namespace: "test"
    ansible_kubectl_config: "{{ kubeconfig }}"
    ansible_kubectl_pod: "shell"
    ansible_kubectl_container: "shell"
    ansible_kubectl_local_env:
       AWS_ACCESS_KEY_ID: "{{ aws_access_key_id | default('') }}"
       AWS_SECRET_ACCESS_KEY: "{{ aws_secret_access_key | default('') }}"
  environment:
    TEST_ENV1: value1
    TEST_ENV2: value2

  tasks:
  - name: test
    ansible.builtin.shell: env
    register: result
  - debug:
      var: result.stdout_lines

That will not work

update: updated the comment with some typo, and added a foo.example.com that does not require any additional variable for authentication as the example of a complex real-world scenario with few clusters with different authentication.

[yurnov]

I tried to implement support of local env variable on my side, please find #699

[yurnov]

Hi @gravesm,

would you like to review PR #702 where the support of the local environmental variable requested in that issue was added? That was tested locally with some debug output, to do so _connect method modified to:

    def _connect(self, port=None):
        """Connect to the container. Nothing to do"""
        super(Connection, self)._connect()
        if not self._connected:
            display.vvv(
              "ESTABLISH {0} CONNECTION".format(self.transport),
              host=self._play_context.remote_addr,
            )
            local_env = self._local_env()
            for key, value in local_env.items():
              display.vvv("ENV: {0}={1}".format(key, value), host=self._play_context.remote_addr)
            self._connected = True

(so, part

local_env = self._local_env()
            for key, value in local_env.items():
              display.vvv("ENV: {0}={1}".format(key, value), host=self._play_context.remote_addr)

was added)

Also, it was tested with AWS EKS and real environmental variables with AWS credentials that is my real case of why I created this issue.

BR/Yuriy

[yurnov]

Hi @gravesm,

any chance to have PR reviewed soon?

BR/Yuriy

[yurnov]

PR #702 is merged to main and backported to stable-3 as #705, so, it will be included in version 3.1.0 or any other minor further release.

Thanks, @gravesm and @abikouo for their review and support

[yurnov]

Added in release 3.1.0 by the PR #702

Changelog is here

Closing ticket as completed