Can't ping pod that is in different node. Http requests take a lot of time or they're lost

[korallo159]

What happened?

I installed a Kubernetes cluster with the default Calico CNI. When I'm in a pod on node1, I can ping another pod on the same node without any issues. However, if the pod is on node2, the ping fails, and HTTP requests are very slow and sometimes lost.

What did you expect to happen?

I can successfully ping a pod located on a different node. Additionally, HTTP requests are consistently processed without any delay or loss.

How can we reproduce it (as minimally and precisely as possible)?

Use ubuntu 22.04

ansible-playbook -i ../cluster/hosts.yaml --become --become-user=root -e "ansible_ssh_common_args='-p 6969'" -u ubuntu cluster.yml

OS

Ubuntu 22.04.4 LTS (Jammy Jellyfish)"

Version of Ansible

ansible [core 2.15.8]
config file = /kubespray/ansible.cfg
configured module search path = ['/kubespray/library']
ansible python module location = /usr/local/lib/python3.10/dist-packages/ansible
ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible
python version = 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (/usr/bin/python3)
jinja version = 3.1.2
libyaml = True

Version of Python

3.10.12

Version of Kubespray (commit)

64447e7

Network plugin used

calico

Full inventory with variables

node6 | SUCCESS => {
"hostvars[inventory_hostname]": {
"access_ip": "XXX.XXX.XXX.XXX",
"allow_unsupported_distribution_setup": false,
"ansible_check_mode": false,
"ansible_config_file": null,
"ansible_diff_mode": false,
"ansible_facts": {},
"ansible_forks": 5,
"ansible_host": "XX.XXX.XXX.XXX",
"ansible_inventory_sources": [
"/cluster/hosts.yaml"
],
"ansible_playbook_python": "/usr/bin/python3",
"ansible_verbosity": 0,
"ansible_version": {
"full": "2.15.8",
"major": 2,
"minor": 15,
"revision": 8,
"string": "2.15.8"
},
"argocd_enabled": false,
"auto_renew_certificates": false,
"bin_dir": "/usr/local/bin",
"calico_cni_name": "k8s-pod-network",
"calico_pool_blocksize": 26,
"cephfs_provisioner_enabled": false,
"cert_manager_enabled": false,
"cluster_name": "cluster.local",
"container_manager": "containerd",
"coredns_k8s_external_zone": "k8s_external.local",
"credentials_dir": "/cluster/credentials",
"default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir",
"deploy_netchecker": false,
"dns_domain": "cluster.local",
"dns_mode": "coredns",
"docker_bin_dir": "/usr/bin",
"docker_container_storage_setup": false,
"docker_daemon_graph": "/var/lib/docker",
"docker_dns_servers_strict": false,
"docker_iptables_enabled": "false",
"docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5",
"docker_rpm_keepcache": 1,
"enable_coredns_k8s_endpoint_pod_names": false,
"enable_coredns_k8s_external": false,
"enable_dual_stack_networks": false,
"enable_nat_default_gateway": true,
"enable_nodelocaldns": true,
"enable_nodelocaldns_secondary": false,
"etcd_data_dir": "/var/lib/etcd",
"etcd_deployment_type": "host",
"event_ttl_duration": "1h0m0s",
"group_names": [
"k8s_cluster",
"kube_node"
],
"groups": {
"all": [
"node1",
"node3",
"node6"
],
"calico_rr": [],
"etcd": [
"node1"
],
"k8s_cluster": [
"node1",
"node3",
"node6"
],
"kube_control_plane": [
"node1"
],
"kube_node": [
"node1",
"node3",
"node6"
],
"ungrouped": []
},
"helm_enabled": false,
"ingress_alb_enabled": false,
"ingress_nginx_enabled": false,
"ingress_publish_status_address": "",
"inventory_dir": "/cluster",
"inventory_file": "/cluster/hosts.yaml",
"inventory_hostname": "node6",
"inventory_hostname_short": "node6",
"ip": "XXX.XXX.XXX.XXX",
"k8s_image_pull_policy": "IfNotPresent",
"kata_containers_enabled": false,
"krew_enabled": false,
"krew_root_dir": "/usr/local/krew",
"kube_api_anonymous_auth": true,
"kube_apiserver_ip": "10.233.0.1",
"kube_apiserver_port": 6443,
"kube_cert_dir": "/etc/kubernetes/ssl",
"kube_cert_group": "kube-cert",
"kube_config_dir": "/etc/kubernetes",
"kube_encrypt_secret_data": false,
"kube_log_level": 2,
"kube_manifest_dir": "/etc/kubernetes/manifests",
"kube_network_node_prefix": 24,
"kube_network_node_prefix_ipv6": 120,
"kube_network_plugin": "calico",
"kube_network_plugin_multus": false,
"kube_ovn_default_gateway_check": true,
"kube_ovn_default_logical_gateway": false,
"kube_ovn_default_vlan_id": 100,
"kube_ovn_dpdk_enabled": false,
"kube_ovn_enable_external_vpc": true,
"kube_ovn_enable_lb": true,
"kube_ovn_enable_np": true,
"kube_ovn_enable_ssl": false,
"kube_ovn_encap_checksum": true,
"kube_ovn_external_address": "8.8.8.8",
"kube_ovn_external_address_ipv6": "2400:3200::1",
"kube_ovn_external_dns": "alauda.cn",
"kube_ovn_hw_offload": false,
"kube_ovn_ic_autoroute": true,
"kube_ovn_ic_dbhost": "127.0.0.1",
"kube_ovn_ic_enable": false,
"kube_ovn_ic_zone": "kubernetes",
"kube_ovn_network_type": "geneve",
"kube_ovn_node_switch_cidr": "100.64.0.0/16",
"kube_ovn_node_switch_cidr_ipv6": "fd00:100:64::/64",
"kube_ovn_pod_nic_type": "veth_pair",
"kube_ovn_traffic_mirror": false,
"kube_ovn_tunnel_type": "geneve",
"kube_ovn_vlan_name": "product",
"kube_owner": "kube",
"kube_pods_subnet": "10.233.64.0/18",
"kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112",
"kube_proxy_mode": "ipvs",
"kube_proxy_nodeport_addresses": [],
"kube_proxy_strict_arp": false,
"kube_script_dir": "/usr/local/bin/kubernetes-scripts",
"kube_service_addresses": "10.233.0.0/18",
"kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116",
"kube_token_dir": "/etc/kubernetes/tokens",
"kube_version": "v1.28.6",
"kube_vip_enabled": false,
"kube_webhook_token_auth": false,
"kube_webhook_token_auth_url_skip_tls_verify": false,
"kubeadm_certificate_key": "277ae8c57cbee67cfc9e76478a775d39cba0a9cdc22e7c7a2dbc5a5c57bbf454",
"kubeadm_patches": {
"dest_dir": "/etc/kubernetes/patches",
"enabled": false,
"source_dir": "/cluster/patches"
},
"kubernetes_audit": false,
"loadbalancer_apiserver_healthcheck_port": 8081,
"loadbalancer_apiserver_port": 6443,
"local_path_provisioner_enabled": false,
"local_release_dir": "/tmp/releases",
"local_volume_provisioner_enabled": false,
"macvlan_interface": "eth1",
"metallb_enabled": false,
"metallb_speaker_enabled": false,
"metrics_server_enabled": false,
"ndots": 2,
"no_proxy_exclude_workers": false,
"nodelocaldns_bind_metrics_host_ip": false,
"nodelocaldns_health_port": 9254,
"nodelocaldns_ip": "169.254.25.10",
"nodelocaldns_second_health_port": 9256,
"nodelocaldns_secondary_skew_seconds": 5,
"ntp_enabled": false,
"ntp_manage_config": false,
"ntp_servers": [
"0.pool.ntp.org iburst",
"1.pool.ntp.org iburst",
"2.pool.ntp.org iburst",
"3.pool.ntp.org iburst"
],
"omit": "__omit_place_holder__1a66123658146b77c8138314e89c42ca925ca873",
"persistent_volumes_enabled": false,
"playbook_dir": "/cluster",
"rbd_provisioner_enabled": false,
"registry_enabled": false,
"resolvconf_mode": "host_resolvconf",
"retry_stagger": 5,
"skydns_server": "10.233.0.3",
"skydns_server_secondary": "10.233.0.4",
"unsafe_show_logs": false,
"volume_cross_zone_attachment": false
}
}
node1 | SUCCESS => {
"hostvars[inventory_hostname]": {
"access_ip": "XXX.XXX.XXX.XXX",
"allow_unsupported_distribution_setup": false,
"ansible_check_mode": false,
"ansible_config_file": null,
"ansible_diff_mode": false,
"ansible_facts": {},
"ansible_forks": 5,
"ansible_host": "146.59.126.186",
"ansible_inventory_sources": [
"/cluster/hosts.yaml"
],
"ansible_playbook_python": "/usr/bin/python3",
"ansible_verbosity": 0,
"ansible_version": {
"full": "2.15.8",
"major": 2,
"minor": 15,
"revision": 8,
"string": "2.15.8"
},
"argocd_enabled": false,
"auto_renew_certificates": false,
"bin_dir": "/usr/local/bin",
"calico_cni_name": "k8s-pod-network",
"calico_pool_blocksize": 26,
"cephfs_provisioner_enabled": false,
"cert_manager_enabled": false,
"cluster_name": "cluster.local",
"container_manager": "containerd",
"coredns_k8s_external_zone": "k8s_external.local",
"credentials_dir": "/cluster/credentials",
"default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir",
"deploy_netchecker": false,
"dns_domain": "cluster.local",
"dns_mode": "coredns",
"docker_bin_dir": "/usr/bin",
"docker_container_storage_setup": false,
"docker_daemon_graph": "/var/lib/docker",
"docker_dns_servers_strict": false,
"docker_iptables_enabled": "false",
"docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5",
"docker_rpm_keepcache": 1,
"enable_coredns_k8s_endpoint_pod_names": false,
"enable_coredns_k8s_external": false,
"enable_dual_stack_networks": false,
"enable_nat_default_gateway": true,
"enable_nodelocaldns": true,
"enable_nodelocaldns_secondary": false,
"etcd_data_dir": "/var/lib/etcd",
"etcd_deployment_type": "host",
"event_ttl_duration": "1h0m0s",
"group_names": [
"etcd",
"k8s_cluster",
"kube_control_plane",
"kube_node"
],
"groups": {
"all": [
"node1",
"node3",
"node6"
],
"calico_rr": [],
"etcd": [
"node1"
],
"k8s_cluster": [
"node1",
"node3",
"node6"
],
"kube_control_plane": [
"node1"
],
"kube_node": [
"node1",
"node3",
"node6"
],
"ungrouped": []
},
"helm_enabled": false,
"ingress_alb_enabled": false,
"ingress_nginx_enabled": false,
"ingress_publish_status_address": "",
"inventory_dir": "/cluster",
"inventory_file": "/cluster/hosts.yaml",
"inventory_hostname": "node1",
"inventory_hostname_short": "node1",
"ip": "XXX.XXX.XXX.XXX",
"k8s_image_pull_policy": "IfNotPresent",
"kata_containers_enabled": false,
"krew_enabled": false,
"krew_root_dir": "/usr/local/krew",
"kube_api_anonymous_auth": true,
"kube_apiserver_ip": "10.233.0.1",
"kube_apiserver_port": 6443,
"kube_cert_dir": "/etc/kubernetes/ssl",
"kube_cert_group": "kube-cert",
"kube_config_dir": "/etc/kubernetes",
"kube_encrypt_secret_data": false,
"kube_log_level": 2,
"kube_manifest_dir": "/etc/kubernetes/manifests",
"kube_network_node_prefix": 24,
"kube_network_node_prefix_ipv6": 120,
"kube_network_plugin": "calico",
"kube_network_plugin_multus": false,
"kube_ovn_default_gateway_check": true,
"kube_ovn_default_logical_gateway": false,
"kube_ovn_default_vlan_id": 100,
"kube_ovn_dpdk_enabled": false,
"kube_ovn_enable_external_vpc": true,
"kube_ovn_enable_lb": true,
"kube_ovn_enable_np": true,
"kube_ovn_enable_ssl": false,
"kube_ovn_encap_checksum": true,
"kube_ovn_external_address": "8.8.8.8",
"kube_ovn_external_address_ipv6": "2400:3200::1",
"kube_ovn_external_dns": "alauda.cn",
"kube_ovn_hw_offload": false,
"kube_ovn_ic_autoroute": true,
"kube_ovn_ic_dbhost": "127.0.0.1",
"kube_ovn_ic_enable": false,
"kube_ovn_ic_zone": "kubernetes",
"kube_ovn_network_type": "geneve",
"kube_ovn_node_switch_cidr": "100.64.0.0/16",
"kube_ovn_node_switch_cidr_ipv6": "fd00:100:64::/64",
"kube_ovn_pod_nic_type": "veth_pair",
"kube_ovn_traffic_mirror": false,
"kube_ovn_tunnel_type": "geneve",
"kube_ovn_vlan_name": "product",
"kube_owner": "kube",
"kube_pods_subnet": "10.233.64.0/18",
"kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112",
"kube_proxy_mode": "ipvs",
"kube_proxy_nodeport_addresses": [],
"kube_proxy_strict_arp": false,
"kube_script_dir": "/usr/local/bin/kubernetes-scripts",
"kube_service_addresses": "10.233.0.0/18",
"kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116",
"kube_token_dir": "/etc/kubernetes/tokens",
"kube_version": "v1.28.6",
"kube_vip_enabled": false,
"kube_webhook_token_auth": false,
"kube_webhook_token_auth_url_skip_tls_verify": false,
"kubeadm_certificate_key": "277ae8c57cbee67cfc9e76478a775d39cba0a9cdc22e7c7a2dbc5a5c57bbf454",
"kubeadm_patches": {
"dest_dir": "/etc/kubernetes/patches",
"enabled": false,
"source_dir": "/cluster/patches"
},
"kubernetes_audit": false,
"loadbalancer_apiserver_healthcheck_port": 8081,
"loadbalancer_apiserver_port": 6443,
"local_path_provisioner_enabled": false,
"local_release_dir": "/tmp/releases",
"local_volume_provisioner_enabled": false,
"macvlan_interface": "eth1",
"metallb_enabled": false,
"metallb_speaker_enabled": false,
"metrics_server_enabled": false,
"ndots": 2,
"no_proxy_exclude_workers": false,
"nodelocaldns_bind_metrics_host_ip": false,
"nodelocaldns_health_port": 9254,
"nodelocaldns_ip": "169.254.25.10",
"nodelocaldns_second_health_port": 9256,
"nodelocaldns_secondary_skew_seconds": 5,
"ntp_enabled": false,
"ntp_manage_config": false,
"ntp_servers": [
"0.pool.ntp.org iburst",
"1.pool.ntp.org iburst",
"2.pool.ntp.org iburst",
"3.pool.ntp.org iburst"
],
"omit": "__omit_place_holder__1a66123658146b77c8138314e89c42ca925ca873",
"persistent_volumes_enabled": false,
"playbook_dir": "/cluster",
"rbd_provisioner_enabled": false,
"registry_enabled": false,
"resolvconf_mode": "host_resolvconf",
"retry_stagger": 5,
"skydns_server": "10.233.0.3",
"skydns_server_secondary": "10.233.0.4",
"unsafe_show_logs": false,
"volume_cross_zone_attachment": false
}
}
node3 | SUCCESS => {
"hostvars[inventory_hostname]": {
"access_ip": "XXX.XXX.XXX.XXX",
"allow_unsupported_distribution_setup": false,
"ansible_check_mode": false,
"ansible_config_file": null,
"ansible_diff_mode": false,
"ansible_facts": {},
"ansible_forks": 5,
"ansible_host": "146.59.47.228",
"ansible_inventory_sources": [
"/cluster/hosts.yaml"
],
"ansible_playbook_python": "/usr/bin/python3",
"ansible_verbosity": 0,
"ansible_version": {
"full": "2.15.8",
"major": 2,
"minor": 15,
"revision": 8,
"string": "2.15.8"
},
"argocd_enabled": false,
"auto_renew_certificates": false,
"bin_dir": "/usr/local/bin",
"calico_cni_name": "k8s-pod-network",
"calico_pool_blocksize": 26,
"cephfs_provisioner_enabled": false,
"cert_manager_enabled": false,
"cluster_name": "cluster.local",
"container_manager": "containerd",
"coredns_k8s_external_zone": "k8s_external.local",
"credentials_dir": "/cluster/credentials",
"default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir",
"deploy_netchecker": false,
"dns_domain": "cluster.local",
"dns_mode": "coredns",
"docker_bin_dir": "/usr/bin",
"docker_container_storage_setup": false,
"docker_daemon_graph": "/var/lib/docker",
"docker_dns_servers_strict": false,
"docker_iptables_enabled": "false",
"docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5",
"docker_rpm_keepcache": 1,
"enable_coredns_k8s_endpoint_pod_names": false,
"enable_coredns_k8s_external": false,
"enable_dual_stack_networks": false,
"enable_nat_default_gateway": true,
"enable_nodelocaldns": true,
"enable_nodelocaldns_secondary": false,
"etcd_data_dir": "/var/lib/etcd",
"etcd_deployment_type": "host",
"event_ttl_duration": "1h0m0s",
"group_names": [
"k8s_cluster",
"kube_node"
],
"groups": {
"all": [
"node1",
"node3",
"node6"
],
"calico_rr": [],
"etcd": [
"node1"
],
"k8s_cluster": [
"node1",
"node3",
"node6"
],
"kube_control_plane": [
"node1"
],
"kube_node": [
"node1",
"node3",
"node6"
],
"ungrouped": []
},
"helm_enabled": false,
"ingress_alb_enabled": false,
"ingress_nginx_enabled": false,
"ingress_publish_status_address": "",
"inventory_dir": "/cluster",
"inventory_file": "/cluster/hosts.yaml",
"inventory_hostname": "node3",
"inventory_hostname_short": "node3",
"ip": "XXX.XXX.XXX.XXX",
"k8s_image_pull_policy": "IfNotPresent",
"kata_containers_enabled": false,
"krew_enabled": false,
"krew_root_dir": "/usr/local/krew",
"kube_api_anonymous_auth": true,
"kube_apiserver_ip": "10.233.0.1",
"kube_apiserver_port": 6443,
"kube_cert_dir": "/etc/kubernetes/ssl",
"kube_cert_group": "kube-cert",
"kube_config_dir": "/etc/kubernetes",
"kube_encrypt_secret_data": false,
"kube_log_level": 2,
"kube_manifest_dir": "/etc/kubernetes/manifests",
"kube_network_node_prefix": 24,
"kube_network_node_prefix_ipv6": 120,
"kube_network_plugin": "calico",
"kube_network_plugin_multus": false,
"kube_ovn_default_gateway_check": true,
"kube_ovn_default_logical_gateway": false,
"kube_ovn_default_vlan_id": 100,
"kube_ovn_dpdk_enabled": false,
"kube_ovn_enable_external_vpc": true,
"kube_ovn_enable_lb": true,
"kube_ovn_enable_np": true,
"kube_ovn_enable_ssl": false,
"kube_ovn_encap_checksum": true,
"kube_ovn_external_address": "8.8.8.8",
"kube_ovn_external_address_ipv6": "2400:3200::1",
"kube_ovn_external_dns": "alauda.cn",
"kube_ovn_hw_offload": false,
"kube_ovn_ic_autoroute": true,
"kube_ovn_ic_dbhost": "127.0.0.1",
"kube_ovn_ic_enable": false,
"kube_ovn_ic_zone": "kubernetes",
"kube_ovn_network_type": "geneve",
"kube_ovn_node_switch_cidr": "100.64.0.0/16",
"kube_ovn_node_switch_cidr_ipv6": "fd00:100:64::/64",
"kube_ovn_pod_nic_type": "veth_pair",
"kube_ovn_traffic_mirror": false,
"kube_ovn_tunnel_type": "geneve",
"kube_ovn_vlan_name": "product",
"kube_owner": "kube",
"kube_pods_subnet": "10.233.64.0/18",
"kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112",
"kube_proxy_mode": "ipvs",
"kube_proxy_nodeport_addresses": [],
"kube_proxy_strict_arp": false,
"kube_script_dir": "/usr/local/bin/kubernetes-scripts",
"kube_service_addresses": "10.233.0.0/18",
"kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116",
"kube_token_dir": "/etc/kubernetes/tokens",
"kube_version": "v1.28.6",
"kube_vip_enabled": false,
"kube_webhook_token_auth": false,
"kube_webhook_token_auth_url_skip_tls_verify": false,
"kubeadm_certificate_key": "277ae8c57cbee67cfc9e76478a775d39cba0a9cdc22e7c7a2dbc5a5c57bbf454",
"kubeadm_patches": {
"dest_dir": "/etc/kubernetes/patches",
"enabled": false,
"source_dir": "/cluster/patches"
},
"kubernetes_audit": false,
"loadbalancer_apiserver_healthcheck_port": 8081,
"loadbalancer_apiserver_port": 6443,
"local_path_provisioner_enabled": false,
"local_release_dir": "/tmp/releases",
"local_volume_provisioner_enabled": false,
"macvlan_interface": "eth1",
"metallb_enabled": false,
"metallb_speaker_enabled": false,
"metrics_server_enabled": false,
"ndots": 2,
"no_proxy_exclude_workers": false,
"nodelocaldns_bind_metrics_host_ip": false,
"nodelocaldns_health_port": 9254,
"nodelocaldns_ip": "169.254.25.10",
"nodelocaldns_second_health_port": 9256,
"nodelocaldns_secondary_skew_seconds": 5,
"ntp_enabled": false,
"ntp_manage_config": false,
"ntp_servers": [
"0.pool.ntp.org iburst",
"1.pool.ntp.org iburst",
"2.pool.ntp.org iburst",
"3.pool.ntp.org iburst"
],
"omit": "__omit_place_holder__1a66123658146b77c8138314e89c42ca925ca873",
"persistent_volumes_enabled": false,
"playbook_dir": "/cluster",
"rbd_provisioner_enabled": false,
"registry_enabled": false,
"resolvconf_mode": "host_resolvconf",
"retry_stagger": 5,
"skydns_server": "10.233.0.3",
"skydns_server_secondary": "10.233.0.4",
"unsafe_show_logs": false,
"volume_cross_zone_attachment": false
}
}

Command used to invoke ansible

ansible-playbook -i ../cluster/hosts.yaml --become --become-user=root -e "ansible_ssh_common_args='-p 6969'" -u ubuntu cluster.yml

Output of ansible run

---

Anything else we need to know

No response

[korallo159]

Related with: #9601

[ant31]

Glad you fixed it and thank you for linking the related issue.