[BUG] Memory leak in __hash__()

[gusostow]

Welcome! You should write in your Bug Report:

I found a case where frozendict does not free memory. The old v1 implementation does free memory in this situation.

OS version (https://www.google.com/search?channel=fs&q=check+os+version&ie=utf-8&oe=utf-8):
your OS version

$ uname -r
3.10.0-1160.59.1.el7.x86_64

Python3 version (python3 -V -V):
your Python version

3.7.6

Steps to reproduce:

import resource
import frozendict

print("frozendict version:", frozendict.__version__)
print()

c = 0
while True:
    mapping = {}
    fz = frozendict.frozendict({i: i for i in range(1000)})
    none = mapping.setdefault(fz, None)

    if (c % 10000) == 0:
        max_rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
        print(f"iteration: {c}, max rss: {max_rss} kb")

    c += 1

Actual result (with the python stack trace if present):
the result you see

frozendict version: 2.3.0

iteration: 0, max rss: 32196 kb
iteration: 10000, max rss: 1793552 kb
iteration: 20000, max rss: 3554452 kb
iteration: 30000, max rss: 5315972 kb
iteration: 40000, max rss: 7076784 kb
iteration: 50000, max rss: 8838472 kb

Note that this will use up memory rather fast on your machine.

[Marco-Sulla]

Thank you for the report.

It's quite normal that v1 has not this problem, since it was in pure Python. Also the pure py package is not affected by this leak. You can see it using from frozendict.core import frozendict or installing the pure py package.

I suppose the problem is in __hash__(), since I get the same result with this code:

import resource
import frozendict

print("frozendict version:", frozendict.__version__)
print()

c = 0

while True:
    fz = frozendict.frozendict({i: i for i in range(1000)})
    hash(fz)
    
    if (c % 10000) == 0:
        max_rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
        print(f"iteration: {c}, max rss: {max_rss} kb")
    
    c += 1

[Marco-Sulla]

It should be fixed in frozendict 2.3.2: #56

Can you confirm?

[gusostow]

Yes, it seems to work. Thanks for the fast turn-around!

PS: Can you publish to conda-forge when you get the chance?

[Marco-Sulla]

Yes, it seems to work. Thanks for the fast turn-around!

Thank you for the report. I suppose I have to add to debug.py a memory leak tracer, as I wrote in the TODO list.

PS: Can you publish to conda-forge when you get the chance?

Emh. frozendict-feedstock no more automatically updates when a new version is published to Pypi, from when I played a little with configuration. I have to ask help to someone.

[Marco-Sulla]

PS: Can you publish to conda-forge when you get the chance?

Done. Unluckily it seems I can't have a noarch package alongside the C Extension on Conda without breaking the bot...

[gusostow]

Thanks, I appreciate it.