Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubernetes cluster creation failure #5677

Closed
pmalode opened this issue Aug 21, 2018 · 6 comments
Closed

Kubernetes cluster creation failure #5677

pmalode opened this issue Aug 21, 2018 · 6 comments
Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Comments

@pmalode
Copy link

pmalode commented Aug 21, 2018

1. What kops version are you running? The command kops version, will display
this information.

1.10.0 (git-8b52ea6d1)

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

1.9.1

3. What cloud provider are you using?
AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

I am creating Kubernetes cluster using encrypted ami.
Steps followed:

  1. Create Image from existing Instance, which uses AWS CIS certified ami - CIS Ubuntu Linux 16.04 LTS Benchmark v1.0.0.4 - Level 1-2ba49cfb-babd-43d3-81f6-3657aeb0f101-ami-713e500e.4
  2. Copy and Encrypt the Image created using below command-
    aws ec2 copy-image --source-region=us-east-1 --source-image-id= --name=encrypted-ami --encrypted
  3. Created cluster by passing the encrypted image.
    kops create cluster cluterName \ --kubernetes-version 1.9.1 \
    --node-count 1 \
    --zones us-east-1c \
    --node-size t2.micro \
    --master-size t2.micro \
    --master-zones us-east-1c \
    --master-count 1 \
    --ssh-public-key path \
    --image encryptedImage \

5. What happened after the commands executed?

Node and Master instances are provisioned, but kops validate cluster command fails with below error

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address. The API DNS IP address is the placeholder address that kops creates: #. Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate. The protokube container and dns-controller deployment logs may contain more diagnostic information. Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.
Validation Failed

6. What did you expect to happen?

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

@transient1
Copy link

I had the same issue. Running kops 1.10 with mostly defaults to test
kops create cluster --zones us-east-1a --name cluster.my.domain --ssh-public-key key.pub --state=s3://my-state-bucket. I got the same error as you when trying to validate. I tried with different AMIs other than the default Debian Jessie 1.9 kops 1.10 uses, I tried specifying a network, tried letting it use the default kubenet, tried different AZs and machine types. The actual nodes would come up, but hopping on them and checking some things out I could see the following

  • there was no protokube container running. The only containers running were kube-proxy and its related pause container
  • nodeup was downloading the protokube image and loading it into docker, but never actually running it. The logs said skipping the provisioning of protokube on the nodes
  • the network plugin never went into a ready state. NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: Kubenet does not have netConfig. This is most likely due to lack of PodCIDR. Pretty sure this is because protokube didn't come up.

I was finally able to get a cluster working by using kops 1.9 instead and a CoreOS AMI.

@etehtsea
Copy link

etehtsea commented Oct 3, 2018

I had the same issue. I was able to fix it reverting 16985c3 (#5623).

P.S. I don't know what the cause of this bug is. Maybe this commit exposed another issue, but I've fixed it the most naive way.

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 1, 2019
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 31, 2019
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot
Copy link
Contributor

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Development

No branches or pull requests

5 participants