Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix memory leak in Redis by using auto memory management #4054

Merged
merged 3 commits into from
Feb 15, 2019

Conversation

stephanie-wang
Copy link
Contributor

What do these changes do?

We normally use RedisModule_AutoMemory to turn on Redis' built-in automatic memory management, but we forgot to enable it for RAY.TABLE_APPEND. Running this command repeatedly will cause Redis to evict all keys and use ever increasing memory.

This PR enables automatic memory management for this command and attempts to restructure the Redis module to make explicit which Redis commands have the feature enabled.

Related issue number

I believe this should close #3884.

// Only use commands that have auto memory management.
// TODO(swang): Ideally, we would make the commands that don't have auto memory
// management inaccessible instead of just using separate namespaces.
using namespace auto_memory_redis_commands;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about we get rid of this, just have the redis_commands namespace that hides the non-auto ones, and put the others into the global namespace? That would be a bit simpler and would make the auto ones more "canonical".

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could even call the redis_command namespace something else, like internal etc. to make clear it is "hidden"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah good idea, thanks! Now I'm not sure why I started with two separate namespaces :)

@pcmoritz
Copy link
Contributor

This is great, good find :)

@ericl
Copy link
Contributor

ericl commented Feb 14, 2019

Nice! We should (in a separate PR) somehow get this issue to show up in the stress tests, perhaps just by running one of the test workloads in a low-memory node.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11967/
Test PASSed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11970/
Test FAILed.

@ericl ericl merged commit 3684e5b into ray-project:master Feb 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[rllib] Memory usage constantly growing while training IMPALA
4 participants