Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out of memory when running a lot of pods #211

Closed
stijndehaes opened this issue Jun 17, 2022 · 3 comments
Closed

Out of memory when running a lot of pods #211

stijndehaes opened this issue Jun 17, 2022 · 3 comments

Comments

@stijndehaes
Copy link

In environments where we are running a lot of pods we see the external resizer going out of memory.
In these environments we typically only run 10's of pvc's, but 1000's of pods.
I believe the pod informer might be the culprit.
Doesn't starting an informer launch a cache? This means all pods are currently cached on the resizer, which leads to excessive memory use. The code isn't actually using the cache, I think the code is only interested in the Add, Update, Delete part and not in the cache.

If this is correct I am willing to create a PR to fix this

@stijndehaes stijndehaes changed the title Out of memory when there are a lot of pods Out of memory when running a lot of pods Jun 17, 2022
@gnufied
Copy link
Contributor

gnufied commented Jun 17, 2022

The whole handleInUse thing is behind a flag and only applicable for use cases where a driver needs external-resizer to enforce offline only expansion strongly. If it is enabled in EKS with external-resizer, then that seems like a problem. cc @wongma7

@stijndehaes
Copy link
Author

@gnufied you are right, resizing can be done online on AWS i think. Good catch

@stijndehaes
Copy link
Author

I will close this ticket as this was fixed in the EBS CSI driver, by setting the argument correctly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants