Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed Connectivity Test For Service #143

Closed
ddodoo opened this issue Oct 8, 2024 · 15 comments
Closed

Failed Connectivity Test For Service #143

ddodoo opened this issue Oct 8, 2024 · 15 comments

Comments

@ddodoo
Copy link

ddodoo commented Oct 8, 2024 •
edited
Loading 

cloud-provider-kind
I1008 12:26:29.928955 3216398 app.go:46] FLAG: --enable-lb-port-mapping="false"
I1008 12:26:29.929037 3216398 app.go:46] FLAG: --enable-log-dumping="false"
I1008 12:26:29.929069 3216398 app.go:46] FLAG: --logs-dir=""
I1008 12:26:29.929098 3216398 app.go:46] FLAG: --v="2"
I1008 12:26:30.491692 3216398 controller.go:174] probe HTTP address https://kind-control-plane:6443
I1008 12:26:30.494510 3216398 controller.go:177] Failed to connect to HTTP address https://kind-control-plane:6443: Get "https://kind-control-plane:6443": dial tcp: lookup kind-control-plane on 127.0.0.53:53: server misbehaving
I1008 12:26:30.494612 3216398 controller.go:174] probe HTTP address https://kind-control-plane:6443
I1008 12:26:30.496497 3216398 controller.go:177] Failed to connect to HTTP address https://kind-control-plane:6443: Get "https://kind-control-plane:6443": dial tcp: lookup kind-control-plane on 127.0.0.53:53: server misbehaving
I1008 12:26:31.496766 3216398 controller.go:174] probe HTTP address https://kind-control-plane:6443
I1008 12:26:31.499533 3216398 controller.go:177] Failed to connect to HTTP address https://kind-control-plane:6443: Get "https://kind-control-plane:6443": dial tcp: lookup kind-control-plane on 127.0.0.53:53: server misbehaving
I1008 12:26:33.500454 3216398 controller.go:174] probe HTTP address https://kind-control-plane:6443
I1008 12:26:33.502926 3216398 controller.go:177] Failed to connect to HTTP address https://kind-control-plane:6443: Get "https://kind-control-plane:6443": dial tcp: lookup kind-control-plane on 127.0.0.53:53: server misbehaving
I1008 12:26:36.504487 3216398 controller.go:174] probe HTTP address https://kind-control-plane:6443
I1008 12:26:36.507202 3216398 controller.go:177] Failed to connect to HTTP address https://kind-control-plane:6443: Get "https://kind-control-plane:6443": dial tcp: lookup kind-control-plane on 127.0.0.53:53: server misbehaving
E1008 12:26:40.509507 3216398 controller.go:151] Failed to connect to apiserver kind: <nil>
I1008 12:26:40.949657 3216398 controller.go:174] probe HTTP address https://127.0.0.1:40095
I1008 12:26:41.043855 3216398 controller.go:84] Creating new cloud provider for cluster kind
I1008 12:26:41.079310 3216398 controller.go:91] Starting cloud controller for cluster kind
I1008 12:26:41.079795 3216398 controller.go:235] Starting service controller
I1008 12:26:41.080572 3216398 shared_informer.go:313] Waiting for caches to sync for service
I1008 12:26:41.079952 3216398 node_controller.go:176] Sending events to api server.
I1008 12:26:41.080380 3216398 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I1008 12:26:41.081413 3216398 node_controller.go:185] Waiting for informer caches to sync
I1008 12:26:41.081643 3216398 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I1008 12:26:41.120366 3216398 reflector.go:368] Caches populated for *v1.Service from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243
I1008 12:26:41.129164 3216398 reflector.go:368] Caches populated for *v1.Node from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243
I1008 12:26:41.180758 3216398 shared_informer.go:320] Caches are synced for service
I1008 12:26:41.180900 3216398 controller.go:737] Syncing backends for all LB services.
I1008 12:26:41.180926 3216398 controller.go:741] Successfully updated 0 out of 0 load balancers to direct traffic to the updated set of nodes
I1008 12:26:41.182245 3216398 instances.go:47] Check instance metadata for kind-control-plane
I1008 12:26:41.261830 3216398 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"172.18.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1008 12:26:41.472106 3216398 node_controller.go:271] Update 1 nodes status took 289.961582ms.

I followed the demo example to setup kind locally and use the cloud-provider-kind extension to provision a service of type loadbalancer with external IP

curl 172.18.0.3:80/hostname
curl: (28) Failed to connect to 172.18.0.3 port 80 after 133406 ms: Couldn't connect to server

I can't figure out why the service is unreachable
@aojea
Copy link
Contributor

aojea commented Oct 8, 2024

You need to provide more information, operating system and versions of the components you are running at minimum, also if you use docker or podman or ...

@ddodoo
Copy link
Author

ddodoo commented Oct 8, 2024

@aojea

ubuntu

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.1 LTS"
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"

kind

kind version
kind v0.24.0 go1.22.6 linux/amd64

docker

docker --version
Docker version 27.3.1, build ce12230

I followed the ff. instructions to install cloud-provider-kind

You can install cloud-provider-kind using go install:

go install sigs.k8s.io/cloud-provider-kind@latest

@ddodoo
Copy link
Author

ddodoo commented Oct 8, 2024

Is this beviour normal from the cloud-provider-kind logs ?

I1008 12:26:30.491692 3216398 controller.go:174] probe HTTP address https://kind-control-plane:6443
I1008 12:26:30.494510 3216398 controller.go:177] Failed to connect to HTTP address https://kind-control-plane:6443: Get "https://kind-control-plane:6443": dial tcp: lookup kind-control-plane on 127.0.0.53:53: server misbehaving
I1008 12:26:30.494612 3216398 controller.go:174] probe HTTP address https://kind-control-plane:6443
I1008 12:26:30.496497 3216398 controller.go:177] Failed to connect to HTTP address https://kind-control-plane:6443: Get "https://kind-control-plane:6443": dial tcp: lookup kind-control-plane on 127.0.0.53:53: server misbehaving
I1008 12:26:31.496766 3216398 controller.go:174] probe HTTP address https://kind-control-plane:6443
I1008 12:26:31.499533 3216398 controller.go:177] Failed to connect to HTTP address https://kind-control-plane:6443: Get "https://kind-control-plane:6443": dial tcp: lookup kind-control-plane on 127.0.0.53:53: server misbehaving
I1008 12:26:33.500454 3216398 controller.go:174] probe HTTP address https://kind-control-plane:6443
I1008 12:26:33.502926 3216398 controller.go:177] Failed to connect to HTTP address https://kind-control-plane:6443: Get "https://kind-control-plane:6443": dial tcp: lookup kind-control-plane on 127.0.0.53:53: server misbehaving
I1008 12:26:36.504487 3216398 controller.go:174] probe HTTP address https://kind-control-plane:6443
I1008 12:26:36.507202 3216398 controller.go:177] Failed to connect to HTTP address https://kind-control-plane:6443: Get "https://kind-control-plane:6443": dial tcp: lookup kind-control-plane on 127.0.0.53:53: server misbehaving
E1008 12:26:40.509507 3216398 controller.go:151] Failed to connect to apiserver kind: <nil>

@aojea
Copy link
Contributor

aojea commented Oct 8, 2024

that is not the problem , it tries first to connect first to the internal endpoint and falls back to the portmapped later.

If you do docker ps you can see the containers of the cluster and the loadbalancer.

What example are you using? can you validate that the pods are running and that you can connect to the service on the nodeport ports on the 172.18.0.2 address?

@ddodoo
Copy link
Author

ddodoo commented Oct 8, 2024 •
edited
Loading 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: policy-local
  labels:
    app: MyLocalApp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: MyLocalApp
  template:
    metadata:
      labels:
        app: MyLocalApp
    spec:
      containers:
      - name: agnhost
        image: registry.k8s.io/e2e-test-images/agnhost:2.40
        args:
          - netexec
          - --http-port=8080
          - --udp-port=8080
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: lb-service-local
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  selector:
    app: MyLocalApp
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

@aojea

The example provided in the documentation

I am following the instructions of the documentation

The pod is running

image
image

curl command fails
image

I cannot connect to the service, the curl command fails

@ddodoo
Copy link
Author

ddodoo commented Oct 8, 2024 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: policy-local
  labels:
    app: MyLocalApp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: MyLocalApp
  template:
    metadata:
      labels:
        app: MyLocalApp
    spec:
      containers:
      - name: agnhost
        image: registry.k8s.io/e2e-test-images/agnhost:2.40
        args:
          - netexec
          - --http-port=8080
          - --udp-port=8080
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: lb-service-local
spec:
  type: NodePort
  externalTrafficPolicy: Local
  selector:
    app: MyLocalApp
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
      nodePort: 30080

I updated the config from LoadBalancer to NodePort

image

image

Service is still unreachable
image

@aojea

@aojea
Copy link
Contributor

aojea commented Oct 8, 2024

then you most probably have something in your host filtering that traffic or not routing it correctly

@ddodoo
Copy link
Author

ddodoo commented Oct 8, 2024 •
edited
Loading

@aojea

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30000
    hostPort: 80

I can reach to reach service by appending extra port mappings to kind cluster config

https://kind.sigs.k8s.io/docs/user/configuration/#nodeport-with-port-mappings

@aojea
Copy link
Contributor

aojea commented Oct 8, 2024

docker exec into the container created for the loadbalancer kindccm-QEWQE... and try to curl from there

@ddodoo
Copy link
Author

ddodoo commented Oct 8, 2024

I am able to curl from outside the cluster and access from the browser as well

Thank you for your support @aojea

@ddodoo ddodoo closed this as completed Oct 8, 2024
@aojea
Copy link
Contributor

aojea commented Oct 9, 2024

I am able to curl from outside the cluster and access from the browser as well

Thank you for your support @aojea

what was the problem?

@ddodoo
Copy link
Author

ddodoo commented Oct 9, 2024 

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30000
    hostPort: 80

I had to update the kind cluster config with extra port mappings

https://kind.sigs.k8s.io/docs/user/configuration/#nodeport-with-port-mappings

@aojea

@aojea
Copy link
Contributor

aojea commented Oct 9, 2024

It must work without that

@ddodoo
Copy link
Author

ddodoo commented Oct 9, 2024 •
edited
Loading

docker exec into the container created for the loadbalancer kindccm-QEWQE... and try to curl from there

This is the container running beside the kind container, envoyproxy/envoy:v1.30.1

I can reach to service via service cluster IP and container port on the cluster

@aojea
Copy link
Contributor

aojea commented Oct 9, 2024

so, the question is why you can not reach the forwarded port from that container in the host, if you do docker ps you'll see those ports

2d6f9816bce7   envoyproxy/envoy:v1.30.1                                        "/docker-entrypoint.…"   10 days ago    Up 8 days     0.0.0.0:49153->10000/tcp, :::49153->10000/tcp   kindccm-PFB765GEO5MNW3Z3N7AZB7XVPFXJDHY7LSKJDIJK

@a-da a-da mentioned this issue Dec 4, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@aojea @ddodoo