Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

flake: timeout waiting for nodeports on EKS(ENI) #355

Closed
michi-covalent opened this issue Jun 22, 2021 · 5 comments
Closed

flake: timeout waiting for nodeports on EKS(ENI) #355

michi-covalent opened this issue Jun 22, 2021 · 5 comments
Labels
area/CI Continuous Integration testing issue or flake ci/flake Issues tracking failing (integration or unit) tests. stale

Comments

@michi-covalent
Copy link
Contributor

flake instances

symptoms

connectivity check times out waiting for nodeports

⌛ [cilium-cilium-cli-962182561.us-east-2.eksctl.io] Waiting for NodePort 192.168.161.8:31469 (cilium-test/echo-other-node) to become ready...
⌛ [cilium-cilium-cli-962182561.us-east-2.eksctl.io] Waiting for NodePort 192.168.161.8:31991 (cilium-test/echo-same-node) to become ready...
⌛ [cilium-cilium-cli-962182561.us-east-2.eksctl.io] Waiting for NodePort 192.168.107.170:31991 (cilium-test/echo-same-node) to become ready...
⌛ [cilium-cilium-cli-962182561.us-east-2.eksctl.io] Waiting for NodePort 192.168.107.170:31469 (cilium-test/echo-other-node) to become ready...

Error: Connectivity test failed: timeout reached waiting for NodePort 192.168.107.170:31469 (cilium-test/echo-other-node)

other notes

i recently added a pre-test check to make sure all the nodeports are open: cbded72

this check seems to be failing pretty consistently on EKS (ENI). i don't see this check failing nearly as frequently in other environments.
@michi-covalent michi-covalent mentioned this issue Jun 22, 2021
Merged
@michi-covalent michi-covalent added the area/CI Continuous Integration testing issue or flake label Jun 23, 2021
@michi-covalent michi-covalent mentioned this issue Jun 23, 2021
Merged
@michi-covalent
Copy link
Contributor Author

michi-covalent commented Jun 23, 2021 •
edited
Loading

steps to repro:

  • follow https://docs.cilium.io/en/v1.10/gettingstarted/k8s-install-default/ to create an eks cluster and install cilium. use cilium-cli: v0.8.1.
  • run the connectivity check: 
    cilium hubble enable
cilium hubble port-forward&
cilium connectivity test
  • kubectl exec into the client pod and try connecting to the nodeport of echo-other-node service. the client pod can connect to the nodeport of the local node but not to the remote node.
  • hubble observe only shows SYNs: 
    % hubble observe -f --port 31865 --ip 192.168.12.49
Jun 23 03:56:33.667: cilium-test/client-6d586dc4bc-hzpcp:42925 -> 192.168.12.49:31865 to-stack FORWARDED (TCP Flags: SYN)
Jun 23 03:56:40.738: cilium-test/client-6d586dc4bc-hzpcp:42925 -> 192.168.12.49:31865 to-stack FORWARDED (TCP Flags: SYN)
Jun 23 03:56:48.930: cilium-test/client-6d586dc4bc-hzpcp:42925 -> 192.168.12.49:31865 to-stack FORWARDED (TCP Flags: SYN)

This was referenced Jun 23, 2021
Merged
Merged
Merged
michi-covalent pushed a commit that referenced this issue Jun 25, 2021
@jrajahalme @michi-covalent
test: Add DNS visibility annotations to client deployments
e0b8599 
Bring back the DNS visibility annotation (466e8f1) that got reverted in
PR #356. I reverted it because it depended on 07a161d which was causing
issue #355. I updated the commit so that it no longer depends on 07a161d.
Having DNS visibility might give us some additional info to debug #367.

Signed-off-by: Jarno Rajahalme <[email protected]>
Signed-off-by: Michi Mutsuzaki <[email protected]>
@michi-covalent michi-covalent mentioned this issue Jun 25, 2021
Closed
michi-covalent pushed a commit that referenced this issue Jun 25, 2021
@jrajahalme @michi-covalent
test: Add DNS visibility annotations to client deployments
8d7c8c1 
Bring back the DNS visibility annotation (466e8f1) that got reverted in
PR #356. I reverted the commit because it depended on 07a161d which was
causing issue #355. I modified the commit so that it no longer depends
on 07a161d. Having DNS visibility might give us some additional info to
debug #367.

Signed-off-by: Jarno Rajahalme <[email protected]>
Signed-off-by: Michi Mutsuzaki <[email protected]>
@nbusseneau nbusseneau added the ci/flake Issues tracking failing (integration or unit) tests. label Sep 8, 2021
@errordeveloper
Copy link
Contributor

errordeveloper commented Sep 10, 2021 •
edited
Loading

I'm seeing this on GKE also: https://github.com/cilium/cilium/runs/3557961582?check_suite_focus=true#step:14:29

@errordeveloper errordeveloper mentioned this issue Sep 10, 2021
Merged
@tklauser
Copy link
Member

Seen on #556: https://github.com/cilium/cilium-cli/pull/556/checks?check_run_id=3657524522

@tklauser tklauser mentioned this issue Sep 21, 2021
Merged
@github-actions GitHub Actions
Copy link

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

@github-actions github-actions bot added the stale label Sep 28, 2024
@github-actions GitHub Actions
Copy link

This issue has not seen any activity since it was marked stale.
Closing.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Oct 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/CI Continuous Integration testing issue or flake ci/flake Issues tracking failing (integration or unit) tests. stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@errordeveloper @tklauser @nbusseneau @michi-covalent