Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fargate node cannot run ebs-csi-controller #492

Closed
literalice opened this issue May 3, 2020 · 8 comments
Closed

Fargate node cannot run ebs-csi-controller #492

literalice opened this issue May 3, 2020 · 8 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Comments

@literalice
Copy link

/kind bug

What happened?

When the ebs-csi-controller pods are bound to fargate nodes, the pods stay in CrashLoopBackOff status.

What you expected to happen?

The ebs-csi-controller pods should successfully run.

How to reproduce it (as minimally and precisely as possible)?

  1. Creates a fargate profile for kube-system namespace.
  2. Deploys the csi driver like this:
kubectl apply -k "github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/?ref=master"

Anything else we need to know?:

I'm going to submit a pull request which fixes compute-type of the deployment like this:

# deploy/kubernetes/base/controller.yaml
# ...
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ebs-csi-controller
  template:
    metadata:
      annotations:
        eks.amazonaws.com/compute-type: ec2  # I'll add this annotation.
      labels:
        app: ebs-csi-controller

Error log is:

$ kubectl logs -p ebs-csi-controller-6cd475bdbb-5wkvs -n kube-system -c ebs-plugin
I0503 10:08:32.978758       1 driver.go:43] Driver: ebs.csi.aws.com Version: 0.4.0
panic: could not get metadata from AWS: EC2 instance metadata is not available

goroutine 1 [running]:
github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/driver.newControllerService(...)
	/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/driver/controller.go:63
github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/driver.NewDriver(0x7ffc9c21faed, 0x30, 0x3, 0x3, 0x0)
	/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/driver/driver.go:47 +0x237
main.main()
	/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/cmd/main.go:45 +0x15f

Fargate node cannot access the EC2 instance metadata endpoint and cannot run the pod normally.

Environment

  • Kubernetes version (use kubectl version): v1.15.11-eks-af3caf
  • Driver version: ebs.csi.aws.com Version: 0.4.0
@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label May 3, 2020
@wongma7
Copy link
Contributor

wongma7 commented May 4, 2020

Yes, the driver depends on the metadata service (discussed here) #474.

I agree with adding the annotation. Even if the metadata issue were overcome, there's no point running the driver on a fargate node where ebs volumes can't be attached.

literalice added a commit to literalice/aws-ebs-csi-driver that referenced this issue May 5, 2020
@literalice
Copy link
Author

Thank you for your comment. I created a PR #496 .

@sarbajitdutta
Copy link

I agree with adding the annotation. Even if the metadata issue were overcome, there's no point running the driver on a fargate node where ebs volumes can't be attached.

Is there any plans to attach ebs volumes to fargate nodes? We are running a heterogenous compute type in our eks cluster. Would really love this feature? @wongma7

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 9, 2020
@dodtsair
Copy link

Is this the issue being used to track EBS on fargate + EKS?

@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Sep 26, 2020
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot
Copy link
Contributor

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants