Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AMI 1.13 is failing with No networks found in /etc/cni/net.d #365

Closed
safiqueahemad opened this issue Oct 26, 2019 · 8 comments
Closed

AMI 1.13 is failing with No networks found in /etc/cni/net.d #365

safiqueahemad opened this issue Oct 26, 2019 · 8 comments

Comments

@safiqueahemad
Copy link

What happened:
Bringing up fresh cluster with control plan on 1.13. When spawn EKS worker node then they are failing with error below:
kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is n
04.339702 5701 cni.go:203] Unable to update cni config: No networks found in /etc/cni/net.d

What you expected to happen:
EKS node should join the cluster properly.
How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • AWS Region:
    us-east-1
  • Instance Type(s):
    t2.medium
  • EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion):
    platformVersion": "eks.5"
  • Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version):
    "version": "1.13"
  • AMI Version:
    1.13.7
  • Kernel (e.g. uname -a):
    Linux ip-10-190-21-242.ts-sandbox.cloud 4.14.146-119.123.amzn2.x86_64 Template is missing source_ami_id in the variables section #1 SMP Mon Sep 23 16:58:43 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  • Release information (run cat /etc/eks/release on a node):
    BASE_AMI_ID="ami-077381babe7660807"
    BUILD_TIME="Mon Oct 7 21:36:57 UTC 2019"
    BUILD_KERNEL="4.14.138-114.102.amzn2.x86_64"
    ARCH="x86_64"
@mogren
Copy link

mogren commented Oct 26, 2019

@safiqueahemad Hi, what CNI version is applied to the cluster?

kubectl describe daemonset aws-node -n kube-system | grep Image | cut -d "/" -f 2

To apply v1.5.3 to the cluster, try:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.5/config/v1.5/aws-k8s-cni.yaml

@safiqueahemad
Copy link
Author

safiqueahemad commented Oct 26, 2019

@mogren CNI v1.5.3 is already in place... see the output below:
amazon-k8s-cni:v1.5.3

@mogren
Copy link

mogren commented Oct 28, 2019

The next option to check is that the nodes have the permissions that the CNI needs to get IPs, that would be the managed policy AmazonEKS_CNI_Policy.

If that one is present, do the Subnets have enough free IPs? How many ENIs are available in the account?

@safiqueahemad
Copy link
Author

safiqueahemad commented Oct 28, 2019

@mogren ... Those (CNI policy, eni and subnet IPs) are I think looks correct...
I see some error logs but not sure if it could cause the issue (from eks node cloud-init-output.log):


Cannot find a valid baseurl for repo: amzn2-core/2/x86_64
Could not retrieve mirrorlist http://amazonlinux.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list error was
12: Timeout on http://amazonlinux.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list: (28, 'Connection timed out after 5000 milliseconds')
Oct 25 23:08:41 cloud-init[3963]: util.py[WARNING]: Package upgrade failed
Oct 25 23:08:41 cloud-init[3963]: cc_package_update_upgrade_install.py[WARNING]: 1 failed with exceptions, re-raising the last one
Oct 25 23:08:41 cloud-init[3963]: util.py[WARNING]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_package_update_upgrade_install.pyc'>) failed

I tried the url (http://amazonlinux.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list) manually and found reachable.

@gofman8
Copy link

gofman8 commented Nov 13, 2019

@mogren ... Those (CNI policy, eni and subnet IPs) are I think looks correct...
I see some error logs but not sure if it could cause the issue (from eks node cloud-init-output.log):

Cannot find a valid baseurl for repo: amzn2-core/2/x86_64

Could not retrieve mirrorlist http://amazonlinux.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list error was
12: Timeout on http://amazonlinux.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list: (28, 'Connection timed out after 5000 milliseconds')
Oct 25 23:08:41 cloud-init[3963]: util.py[WARNING]: Package upgrade failed
Oct 25 23:08:41 cloud-init[3963]: cc_package_update_upgrade_install.py[WARNING]: 1 failed with exceptions, re-raising the last one
Oct 25 23:08:41 cloud-init[3963]: util.py[WARNING]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_package_update_upgrade_install.pyc'>) failed
I tried the url (http://amazonlinux.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list) manually and found reachable.

Look at that
https://stackoverflow.com/questions/20822021/amazon-ec2-instance-cant-update-or-use-yum

@bobbui
Copy link

bobbui commented Jan 15, 2020

try this: #306 (comment)
i also have the same error, after upgrade CNI it's working

@mogren
Copy link

mogren commented Jan 20, 2020

Closing, see #306

@mogren mogren closed this as completed Jan 20, 2020
@Gwynbleidd017
Copy link

Same issue here, resolved by upgrading the CNI plugin.
Use : https://docs.aws.amazon.com/eks/latest/userguide/cni-upgrades.html and update region to fetch the right image.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants