Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rerun causes memory leak if enabled on KELO NUC #25

Open
m-decoster opened this issue Aug 27, 2024 · 2 comments
Open

Rerun causes memory leak if enabled on KELO NUC #25

m-decoster opened this issue Aug 27, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@m-decoster
Copy link
Collaborator

We left the python test_server.py script running during lunch today and when we came back, the KELO NUC became unresponsive (very slow and not accepting new SSH connections).

@m-decoster
Copy link
Collaborator Author

This appears to be a memory leak: memory keeps increasing while the server is running but idle.

@m-decoster
Copy link
Collaborator Author

m-decoster commented Sep 3, 2024

Profiling shows that the leak comes from rerun. It is not clear why exactly rerun is causing large memory usage, because the process is set to use a maximum of 1GB (and it mentions dropping old messages). Rerun logging has been set to disabled by default in d6ba06c until we find the cause.

To profile:

Add @profile above def step() in robile_platform.py, and then run

python -m memory_profiler test_server.py

@m-decoster m-decoster added the bug Something isn't working label Sep 30, 2024
@m-decoster m-decoster changed the title KELO NUC becomes unresponsive if server is left on for too long Rerun causes memory leak if enabled on KELO NUC Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant