You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have a service that every N minutes loads a fastcache from file. Every time this happens, the program leaks memory. This was easily observable by simply loading a cache multiple times in the same program and then using htop/ActivityMonitor to see this steady increase of memory. In my particular use case, the cache is rather large (~18M keys, 662MB on disk).
The snippet to reproduce this is straight-forward:
After some profiling and debugging I've found that the use of getChunk() with memmap is the culprit. When we reload the cache every time, we never call putChunk, only getChunk, which causes the library to continuosly allocate memory via memmap which is never munmaped. To verify this, I added a panic to putChunk, which was never triggered.
I think perhaps one solution is to add LoadFromFile as a method also to the Cache object, so that the chunks from the old buckets can be reused when loading a new cache. I might be able to help with a PR if time permits, but I thought in the meantime you should be aware of this issue
The text was updated successfully, but these errors were encountered:
I have a service that every N minutes loads a fastcache from file. Every time this happens, the program leaks memory. This was easily observable by simply loading a cache multiple times in the same program and then using htop/ActivityMonitor to see this steady increase of memory. In my particular use case, the cache is rather large (~18M keys, 662MB on disk).
The snippet to reproduce this is straight-forward:
After some profiling and debugging I've found that the use of
getChunk()
withmemmap
is the culprit. When we reload the cache every time, we never callputChunk
, onlygetChunk
, which causes the library to continuosly allocate memory viamemmap
which is nevermunmap
ed. To verify this, I added apanic
toputChunk
, which was never triggered.I think perhaps one solution is to add
LoadFromFile
as a method also to theCache
object, so that the chunks from the old buckets can be reused when loading a new cache. I might be able to help with a PR if time permits, but I thought in the meantime you should be aware of this issueThe text was updated successfully, but these errors were encountered: