[connection]: Unable to access kind cluster inside a custom Github Action

[uditgaurav]

What is happening?

• I'm Using a KinD cluster in Github Action the flow of workflow yaml is:
  • Creating KinD cluster (Created successfully)
  • Getting Nodes via kubectl get nodes

NAME                 STATUS   ROLES    AGE   VERSION
kind-control-plane   Ready    master   68s   v1.17.0

• Now I'm trying to use the cluster inside my personal Github action bypassing kubeconfig. It is showing connection error:

exit status 1: The connection to the server 127.0.0.1:32768 was refused - did you specify the right host or port?

• Did I need to do anything else?

Workflow-yaml

name: Push
on:
  push:
    branches: [ master ]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2

    - name: Creating KinD cluster
      run: |
        GO111MODULE=on 
        curl -Lo ./kind https://github.com/kubernetes sigs/kind/releases/download/v0.7.0/kind-$(uname)-amd64
        chmod +x ./kind
        sudo mv ./kind /usr/local/bin/kind
        kind version
        kind create cluster --loglevel debug
        sleep 60

    - name: Getting Nodes
      run: |
        kubectl get nodes

    - name: Set config
      run: echo ::set-env name=KUBE_CONFIG_DATA::$(base64 -w 0 ~/.kube/config)
  
      #This is the custom Github Action inside which the kubectl command is not running.
    - name: Running Litmus pod delete chaos experiment
      uses: uditgaurav/[email protected]

[BenTheElder]

glancing at this, the second to last step looks suspect.
/triage support

[uditgaurav]

@BenTheElder Thanks for a quick response
In that step, it is actually saving the kubeconfig data of kind cluster in a variable KUBE_CONFIG_DATA
I'm able to get the kubeconfig value if I do an echo to the variable in the next step.

The last step is actually running this script

[BenTheElder]

I see, the set-env is some GitHub actions magic.
I'm not familiar enough with actions.

stomping kubeconfig like that seems less than desirable, but I don't see a bug in it, skimming your script. I would still suspect a big with this, especially if kubectl get nodes worked in the previous step.

there are community supported GitHub actions for kind BTW.

[uditgaurav]

Ah... Github action community would be a better place for this. Okay one thing for using this cluster in a separate environment, the kubeconfig will be enough or should I change anything else? like IP of apiserver or listeningaddress?

[BenTheElder]

I don't know, that depends on implementation details of the GitHub actions environment.

[aledbf]

@uditgaurav please check https://github.com/engineerd/setup-kind

[tamalsaha]

Github Actions Ubuntu images come pre-installed with kind. You can find it out here or send a pr if you want to update it. https://github.com/actions/virtual-environments/search?q=kind&unscoped_q=kind

If you want a specific version of kind, use https://github.com/engineerd/setup-kind wihtout any issues. This works flawlessly.

[uditgaurav]

Thanks, @aledbf, and @tamalsaha that's cool to know that the ubuntu image itself contains kind. But I'm facing issues in my custom action uditgaurav/[email protected], So when I'm using this action it takes me to a different environment i.e my docker environment used for this action and now I'm not using ubuntu-latest.
So from this action/container how to access the cluster. I tried passing kubeconfig but it doesn't work as mentioned above.
I used https://github.com/engineerd/setup-kind for the cluster setup but the issue remains the same.

[BenTheElder]

Taking a second look: does this last action run in a container?

Localhost / the loopback interface is unique to each container. 127.0.0.1 is local to each container...

You can use kind export kubeconfig --internal on 0.7, on 0.8+ you'd either need this step to run on the kind network somehow (not sure if this is possible with GitHub actions), or else maybe #1558

[uditgaurav]

Thanks @BenTheElder
My kind version is 0.7
kind export kubeconfig --internal gives ERROR: unknown flag: --internal maybe I'm missing some parameter?

So, I passed this to action and now it seems to be running fine for me.

- name: Export kubeconfig
  run: |
    kind get kubeconfig --name "kind" --internal | sed "s/kind-control-plane/$(docker inspect "kind-control-plane" --format "{{ .NetworkSettings.Networks.kind.IPAddress }}")/g" > config

[BenTheElder]

I forgot we only have "kind get kubeconfig --internal >$HOME/.kube/config" This may be an issue on 0.8+ ... I'm not sure if github exposes what docker network they use for actions or any way to configure it. it looks like no and no.

…

On Wed, May 27, 2020 at 1:53 PM UDIT GAURAV ***@***.***> wrote: My kind version is 0.7 kind export kubeconfig --internal gives ERROR: unknown flag: --internal maybe I'm missing some parameter? So, I passed this to action and now it seems to be running fine for me. - name: Export kubeconfig run: | kind get kubeconfig --name "kind" --internal | sed "s/kind-control-plane/$(docker inspect "kind-control-plane" --format "{{ .NetworkSettings.Networks.kind.IPAddress }}")/g" > config — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1619 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHADK3BS4Q5BGCH4YRHVODRTV4SRANCNFSM4NISAQSQ> .

[uditgaurav]

Ok, So now after using kind get kubeconfig --internal >$HOME/.kube/config also it works
I think I got the current solution. We are good to close this issue now.
Thanks, @BenTheElder, and all for the quick and awesome support.

[BenTheElder]

awesome thanks!

I'd like to figure out 0.8+, but we can revisit that later :-)

[uditgaurav]

Hi @BenTheElder, It looks like now with kind v0.7.0 and kind get kubeconfig --internal >$HOME/.kube/config i'm unable to run the k8s inside a container (GitHub actions) so I updated to the latest version as of now that is v0.11.1 but still getting the same issue. I have also tried with

kind get kubeconfig --name "kind" --internal | sed "s/kind-control-plane/$(docker inspect "kind-control-plane" --format "{{ .NetworkSettings.Networks.kind.IPAddress }}")/g" > config

But it also gives when ran with actions (inside a container):

Unable to connect to the server: dial tcp 172.18.0.5:6443: i/o timeout

$kubectl cluster-info --context kind-kind

Kubernetes control plane is running at https://172.18.0.5:6443
CoreDNS is running at https://172.18.0.5:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Do we have any solution for version 0.8+?

[BenTheElder]

we run kind in github actions and don't do anything special at all.

[BenTheElder]

https://github.com/kubernetes-sigs/kind/tree/main/.github

[uditgaurav]

Thanks for a quick response.
It is working fine when used within github workflows. But the issue comes if we use the ~./kube/config of the kind in a container and execute some k8s commands (that is inside actions in a workflow).

[BenTheElder]

you need to make sure that container is on or attached to the kind network when using the --internal address, or else on the host network when using the loopback address.

we have some mention of the network in the release notes for v0.8

[BenTheElder]

like, basically this is just docker networking, kind can't do anything here. if you run containers on a different network and don't connect them docker intentionally isolates them. you can connect them either by changing the container network when creating it or using docker network connect ...

the loopback address is only reachable from host networking because linux.