Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed Connectivity Test For Service V2 ubuntu+podman #165

Closed
a-da opened this issue Dec 4, 2024 · 8 comments
Closed

Failed Connectivity Test For Service V2 ubuntu+podman #165

a-da opened this issue Dec 4, 2024 · 8 comments

Comments

@a-da
Copy link

a-da commented Dec 4, 2024

The ticket is very similar to #143 but unfortunately was not clear how to solve it.

Please instruct me how to troubleshoot this and finally fix it.
I will open a followup PR to add troubleshoot.md to be referenced in readme for others as soon I fix this setup.

Background

OS

$ cat /etc/*release*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.1 LTS"
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

IP tables

sudo iptables -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Setup

kind create cluster
enabling experimental podman provider
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.31.2) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
ubuntu@oraclu:~/src_kind$ kubectl cluster-info --context kind-kind
Kubernetes control plane is running at https://127.0.0.1:33139
CoreDNS is running at https://127.0.0.1:33139/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
ubuntu@oraclu:~/src_kind$ $ kubectl label node kind-control-plane node.kubernetes.io/exclude-from-external-load-balancers-node/kind-control-plane unlabeled
$: command not found
ubuntu@oraclu:~/src_kind$ kubectl label node kind-control-plane node.kubernetes.io/exclude-from-external-load-balancers-node/kind-control-plane unlabeled
error: at least one label update is required
ubuntu@oraclu:~/src_kind$ cloud-provider-kind
I1204 07:51:49.984497   85199 app.go:46] FLAG: --enable-lb-port-mapping="false"
I1204 07:51:49.984529   85199 app.go:46] FLAG: --enable-log-dumping="false"
I1204 07:51:49.984534   85199 app.go:46] FLAG: --logs-dir=""
I1204 07:51:49.984538   85199 app.go:46] FLAG: --v="2"
enabling experimental podman provider
I1204 07:51:50.299089   85199 controller.go:174] probe HTTP address https://kind-control-plane:6443
I1204 07:51:50.304846   85199 controller.go:177] Failed to connect to HTTP address https://kind-control-plane:6443: Get "https://kind-control-plane:6443": dial tcp: lookup kind-control-plane on 127.0.0.53:53: server misbehaving
I1204 07:51:50.304884   85199 controller.go:174] probe HTTP address https://kind-control-plane:6443
I1204 07:51:50.305716   85199 controller.go:177] Failed to connect to HTTP address https://kind-control-plane:6443: Get "https://kind-control-plane:6443": dial tcp: lookup kind-control-plane on 127.0.0.53:53: server misbehaving
I1204 07:51:51.305826   85199 controller.go:174] probe HTTP address https://kind-control-plane:6443
I1204 07:51:51.317042   85199 controller.go:177] Failed to connect to HTTP address https://kind-control-plane:6443: Get "https://kind-control-plane:6443": dial tcp: lookup kind-control-plane on 127.0.0.53:53: server misbehaving
I1204 07:51:53.318168   85199 controller.go:174] probe HTTP address https://kind-control-plane:6443
I1204 07:51:53.319567   85199 controller.go:177] Failed to connect to HTTP address https://kind-control-plane:6443: Get "https://kind-control-plane:6443": dial tcp: lookup kind-control-plane on 127.0.0.53:53: server misbehaving
I1204 07:51:56.319734   85199 controller.go:174] probe HTTP address https://kind-control-plane:6443
I1204 07:51:56.321241   85199 controller.go:177] Failed to connect to HTTP address https://kind-control-plane:6443: Get "https://kind-control-plane:6443": dial tcp: lookup kind-control-plane on 127.0.0.53:53: server misbehaving
E1204 07:52:00.322366   85199 controller.go:151] Failed to connect to apiserver kind: <nil>
I1204 07:52:00.764654   85199 controller.go:174] probe HTTP address https://127.0.0.1:33139
I1204 07:52:00.783186   85199 controller.go:84] Creating new cloud provider for cluster kind
I1204 07:52:00.800638   85199 controller.go:91] Starting cloud controller for cluster kind
I1204 07:52:00.800722   85199 node_controller.go:176] Sending events to api server.
I1204 07:52:00.800760   85199 controller.go:235] Starting service controller
I1204 07:52:00.801114   85199 shared_informer.go:313] Waiting for caches to sync for service
I1204 07:52:00.800884   85199 node_controller.go:185] Waiting for informer caches to sync
I1204 07:52:00.801246   85199 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I1204 07:52:00.801267   85199 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I1204 07:52:00.805987   85199 reflector.go:368] Caches populated for *v1.Service from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243
I1204 07:52:00.806247   85199 reflector.go:368] Caches populated for *v1.Node from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243
I1204 07:52:00.901514   85199 shared_informer.go:320] Caches are synced for service
I1204 07:52:00.901732   85199 controller.go:737] Syncing backends for all LB services.
I1204 07:52:00.901825   85199 controller.go:741] Successfully updated 0 out of 0 load balancers to direct traffic to the updated set of nodes
I1204 07:52:00.902676   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 07:52:01.005523   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 07:52:01.016964   85199 node_controller.go:271] Update 1 nodes status took 115.396278ms.
I1204 07:52:31.018630   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 07:52:31.123464   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 07:52:31.138594   85199 node_controller.go:271] Update 1 nodes status took 120.236963ms.
I1204 07:52:39.736380   85199 controller.go:402] Ensuring load balancer for service default/lb-service-local
I1204 07:52:39.736726   85199 controller.go:958] Adding finalizer to service default/lb-service-local
I1204 07:52:39.738514   85199 event.go:389] "Event occurred" object="default/lb-service-local" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I1204 07:52:39.758854   85199 loadbalancer.go:28] Ensure LoadBalancer cluster: kind service: lb-service-local
I1204 07:52:39.812225   85199 server.go:95] container kindccm-JQR2TXKKQBRJNYZ4LUJ4TY4C2YAW4FANP7ZCY5PN for loadbalancer is not running
I1204 07:52:39.889594   85199 server.go:104] creating container for loadbalancer
I1204 07:52:39.889678   85199 server.go:249] creating loadbalancer with parameters: [--detach --tty --label io.x-k8s.cloud-provider-kind.cluster=kind --label io.x-k8s.cloud-provider-kind.loadbalancer.name=kind/default/lb-service-local --net kind --init=false --hostname kindccm-JQR2TXKKQBRJNYZ4LUJ4TY4C2YAW4FANP7ZCY5PN --privileged --restart=on-failure --sysctl=net.ipv4.ip_forward=1 --sysctl=net.ipv4.conf.all.rp_filter=0 --publish=10000/TCP --publish-all docker.io/envoyproxy/envoy:v1.30.1 bash -c echo -en 'node:
  cluster: cloud-provider-kind
  id: cloud-provider-kind-id

dynamic_resources:
  cds_config:
    resource_api_version: V3
    path: /home/envoy/cds.yaml
  lds_config:
    resource_api_version: V3
    path: /home/envoy/lds.yaml

admin:
  access_log_path: /dev/stdout
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 10000
' > /home/envoy/envoy.yaml && touch /home/envoy/cds.yaml && touch /home/envoy/lds.yaml && while true; do envoy -c /home/envoy/envoy.yaml && break; sleep 1; done]
I1204 07:52:40.114718   85199 server.go:112] updating loadbalancer
I1204 07:52:40.114768   85199 proxy.go:239] address type Hostname, only InternalIP supported
I1204 07:52:40.114780   85199 proxy.go:274] envoy config info: &{HealthCheckPort:31680 ServicePorts:map[IPv4_80_TCP:{Listener:{Address:0.0.0.0 Port:80 Protocol:TCP} Cluster:[{Address:10.89.0.2 Port:31615 Protocol:TCP}]}] SessionAffinity:None SourceRanges:[]}
I1204 07:52:40.118177   85199 proxy.go:292] updating loadbalancer with config
resources:
- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
  name: listener_IPv4_80_TCP
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 80
      protocol: TCP
  filter_chains:
  - filters:
    - name: envoy.filters.network.tcp_proxy
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
        access_log:
        - name: envoy.file_access_log
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
        stat_prefix: tcp_proxy
        cluster: cluster_IPv4_80_TCP
I1204 07:52:40.266512   85199 proxy.go:303] updating loadbalancer with config
resources:
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
  name: cluster_IPv4_80_TCP
  connect_timeout: 5s
  type: STATIC
  lb_policy: RANDOM
  health_checks:
  - timeout: 5s
    interval: 3s
    unhealthy_threshold: 2
    healthy_threshold: 1
    no_traffic_interval: 5s
    always_log_health_check_failures: true
    always_log_health_check_success: true
    event_log_path: /dev/stdout
    http_health_check:
      path: /healthz
  load_assignment:
    cluster_name: cluster_IPv4_80_TCP
    endpoints:
      - lb_endpoints:
        - endpoint:
            health_check_config:
              port_value: 31680
            address:
              socket_address:
                address: 10.89.0.2
                port_value: 31615
                protocol: TCP
I1204 07:52:40.615923   85199 server.go:128] get loadbalancer status
I1204 07:52:40.661309   85199 controller.go:999] Patching status for service default/lb-service-local
I1204 07:52:40.661529   85199 event.go:389] "Event occurred" object="default/lb-service-local" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuredLoadBalancer" message="Ensured load balancer"
I1204 07:53:01.139155   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 07:53:01.230769   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 07:53:01.239388   85199 node_controller.go:271] Update 1 nodes status took 100.400178ms.
I1204 07:53:31.240434   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 07:53:31.337481   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 07:53:31.345573   85199 node_controller.go:271] Update 1 nodes status took 105.266423ms.
I1204 07:54:01.346481   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 07:54:01.434528   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 07:54:01.444452   85199 node_controller.go:271] Update 1 nodes status took 98.069195ms.
I1204 07:54:31.445666   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 07:54:31.534368   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 07:54:31.544087   85199 node_controller.go:271] Update 1 nodes status took 98.508119ms.
I1204 07:55:01.544680   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 07:55:01.638792   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 07:55:01.647603   85199 node_controller.go:271] Update 1 nodes status took 102.980921ms.
I1204 07:55:31.648745   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 07:55:31.746834   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 07:55:31.755938   85199 node_controller.go:271] Update 1 nodes status took 107.260161ms.
I1204 07:56:01.756565   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 07:56:01.847699   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 07:56:01.856547   85199 node_controller.go:271] Update 1 nodes status took 100.453578ms.
I1204 07:56:31.856870   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 07:56:31.950361   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 07:56:31.959072   85199 node_controller.go:271] Update 1 nodes status took 102.357355ms.
I1204 07:57:01.959880   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 07:57:02.049385   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 07:57:02.058082   85199 node_controller.go:271] Update 1 nodes status took 98.331997ms.
I1204 07:57:32.058229   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 07:57:32.155081   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 07:57:32.163409   85199 node_controller.go:271] Update 1 nodes status took 105.271943ms.
I1204 07:58:02.172612   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 07:58:02.296829   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 07:58:02.306328   85199 node_controller.go:271] Update 1 nodes status took 136.129749ms.
I1204 07:58:32.307136   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 07:58:32.398883   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 07:58:32.408137   85199 node_controller.go:271] Update 1 nodes status took 101.112023ms.
I1204 07:59:02.408368   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 07:59:02.510899   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 07:59:02.520219   85199 node_controller.go:271] Update 1 nodes status took 111.931524ms.
I1204 07:59:32.521022   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 07:59:32.617634   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 07:59:32.627420   85199 node_controller.go:271] Update 1 nodes status took 106.512993ms.
I1204 08:00:02.628483   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 08:00:02.725211   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 08:00:02.734723   85199 node_controller.go:271] Update 1 nodes status took 106.334912ms.
I1204 08:00:32.735938   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 08:00:32.838650   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 08:00:32.847797   85199 node_controller.go:271] Update 1 nodes status took 111.928284ms.
I1204 08:01:02.848359   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 08:01:02.942754   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 08:01:02.951620   85199 node_controller.go:271] Update 1 nodes status took 103.319043ms.
I1204 08:01:32.952182   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 08:01:33.042196   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}
I1204 08:01:33.051550   85199 node_controller.go:271] Update 1 nodes status took 99.575168ms.
I1204 08:02:03.052903   85199 instances.go:47] Check instance metadata for kind-control-plane
I1204 08:02:03.151343   85199 instances.go:75] instance metadata for kind-control-plane: &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}

Set load balancer

ubuntu@oraclu:~/src_kind$ kubectl apply -f loadbalancer_etp_local.yaml
deployment.apps/policy-local created
service/lb-service-local created

Query load balancer

ubuntu@oraclu:~/src_kind$ kubectl get service/lb-service-local
NAME               TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
lb-service-local   LoadBalancer   10.96.244.51   10.89.0.3     80:31615/TCP   10s
ubuntu@oraclu:~/src_kind$ curl 10.89.0.3:80/hostname
curl: (28) Failed to connect to 10.89.0.3 port 80 after 135633 ms: Couldn't connect to server

Additional configurations

ubuntu@oraclu:~/src_kind$ kubectl get nodes -A --show-labels
NAME                 STATUS   ROLES           AGE    VERSION   LABELS
kind-control-plane   Ready    control-plane   5m3s   v1.31.2   beta.kubernetes.io/arch=arm64,beta.kubernetes.io/os=linux,kubernetes.io/arch=arm64,kubernetes.io/hostname=kind-control-plane,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=

$ podman ps
CONTAINER ID  IMAGE                                                                                           COMMAND               CREATED         STATUS         PORTS                      NAMES
6413cba1a05b  docker.io/kindest/node@sha256:18fbefc20a7113353c7b75b5c869d7145a6abd6269154825872dc59c1329912e                        15 minutes ago  Up 15 minutes  127.0.0.1:33139->6443/tcp  kind-control-plane
972b47d00588  docker.io/envoyproxy/envoy:v1.30.1                                                              bash -c echo -en ...  13 minutes ago  Up 13 minutes  0.0.0.0:45639->10000/tcp   kindccm-JQR2TXKKQBRJNYZ4LUJ4TY4C2YAW4FANP7ZCY5PN
@aojea
Copy link
Contributor

aojea commented Dec 4, 2024 

                address: 10.89.0.2
                port_value: 31615

that should be the nodeport, if you do curl 10.89.0.2:31615 does it work?

@a-da
Copy link
Author

a-da commented Dec 4, 2024

Nope

I have to restart server so IP may change

ubuntu@oraclu:~$ kubectl get service/lb-service-local
NAME               TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
lb-service-local   LoadBalancer   10.96.244.51   10.89.0.3     80:31615/TCP   8h

Using EXTERNAL-IP

ubuntu@oraclu:~$ curl --connect-timeout 10 10.89.0.3:80
curl: (28) Failed to connect to 10.89.0.3 port 80 after 10002 ms: Timeout was reached

ubuntu@oraclu:~$ curl --connect-timeout 10 10.89.0.3:31615
curl: (28) Failed to connect to 10.89.0.3 port 31615 after 10002 ms: Timeout was reached

with the IP from cloud-provider-kind log

# I1204 16:25:39.055444  122498 node_controller.go:271] Update 1 nodes status took 103.055289ms.
# I1204 16:26:09.056270  122498 instances.go:47] Check instance metadata for kind-control-plane
# I1204 16:26:09.149978  122498 instances.go:75] instance metadata for kind-control-plane: # &cloudprovider.InstanceMetadata{ProviderID:"kind://kind/kind/kind-control-plane", InstanceType:"kind-node", NodeAddresses:[]v1.NodeAddress{v1.NodeAddress{Type:"Hostname", Address:"kind-control-plane"}, v1.NodeAddress{Type:"InternalIP", Address:"10.89.0.2"}, v1.NodeAddress{Type:"InternalIP", Address:"fc00:f853:ccd:e793::2"}}, Zone:"", Region:"", AdditionalLabels:map[string]string(nil)}


$ curl --connect-timeout 10 10.89.0.2:80
curl: (28) Failed to connect to 10.89.0.2 port 80 after 10002 ms: Timeout was reached
ubuntu@oraclu:~$ curl --connect-timeout 10 10.89.0.2:31615
curl: (28) Failed to connect to 10.89.0.2 port 31615 after 10002 ms: Timeout was reached

@aojea
Copy link
Contributor

aojea commented Dec 4, 2024

so you can not connect from your host to the containers? are they not running in the same vm?

@a-da
Copy link
Author

a-da commented Dec 4, 2024

nope,
ubuntu setup, in ubuntu is podman with kind, no VM.

@aojea
Copy link
Contributor

aojea commented Dec 4, 2024

can you ping or connect to the containers directly?

@a-da
Copy link
Author

a-da commented Dec 7, 2024

can you ping or connect to the containers directly?

ubuntu@oraclu:~$ podman ps
CONTAINER ID  IMAGE                                                                                           COMMAND               CREATED        STATUS         PORTS                      NAMES
6413cba1a05b  docker.io/kindest/node@sha256:18fbefc20a7113353c7b75b5c869d7145a6abd6269154825872dc59c1329912e                        3 days ago     Up 10 minutes  127.0.0.1:33139->6443/tcp  kind-control-plane
7510cef9716f  docker.io/envoyproxy/envoy:v1.30.1                                                              bash -c echo -en ...  9 minutes ago  Up 9 minutes   0.0.0.0:41359->10000/tcp   kindccm-JQR2TXKKQBRJNYZ4LUJ4TY4C2YAW4FANP7ZCY5PN

@aojea could you please provide the commands for ping and connect ?
ping command will not work on port, right ?
connect not sure what do you mean here

@a-da
Copy link
Author

a-da commented Dec 7, 2024

nevermind @aojea, it worked

Solution purge podman and install docker.

For podman it may be required a few more pre configuration.

$ kubectl get service/lb-service-local
NAME               TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
lb-service-local   LoadBalancer   10.96.142.102   172.18.0.3    80:32697/TCP   9s
ubuntu@oraclu:~/src_kind$ curl 172.18.0.3:80/hostname
policy-local-77dc66c467-tjxqm

@aojea do you test this with podman, maybe I can help with setup of CI+podman in this project ;) let me know if this is welcome.

@a-da a-da changed the title Failed Connectivity Test For Service V2 Failed Connectivity Test For Service V2 ubuntu+podman Dec 7, 2024
@aojea
Copy link
Contributor

aojea commented Dec 8, 2024

Having CI sounds like a good idea, but podman needs to be supported first in kind kubernetes-sigs/kind#1778 and is far from it, so we can only offert best effort support to podman

@a-da a-da closed this as completed Dec 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@aojea @a-da