Memory leak?

[severinsimmler]

I'm using marshmallow with Falcon and have observed a quite strange memory leak. This minimal application does nothing but provide a GET endpoint serializing a dictionary and profiling the memory usage with memory_profiler:

from falcon import App, Request, Response
from marshmallow import fields, Schema
from memory_profiler import profile


class ArtistSchema(Schema):
    name = fields.Str()


class ArtistResource:
    @profile
    def on_get(self, request: Request, response: Response):
        response.text = ArtistSchema().dumps({"name": "David Bowie" * 10000})


application = App()
application.add_route('/artist', ArtistResource())

Sending now 100,000 requests like:

import requests


for _ in range(100000):
    requests.get("http://127.0.0.1:8000/artist")

will slowly increase the memory footprint from 53.0 MiB to 54.5 MiB (and would apparently continue to increase like this).

I'm constructing a new ArtistSchema object for every request. But if I do this only once an reuse it for every request:

ARTIST_SCHEMA = ArtistSchema()

class ArtistResource:
    @profile
    def on_get(self, request: Request, response: Response):
        response.text = ARTIST_SCHEMA.dumps({"name": "David Bowie" * 10000})

the memory usage does not increase. How is this possible? Or is this even a marshmallow issue?

My setup:

• Python 3.9.6
• marshmallow 3.14.1
• falcon 3.0.1
• gunicorn 20.1.0
• memory-profiler 0.60.0

[lafrech]

See #660 (and #732).

[deckar01]

class_registry does not grow, because the new schemas all have the same module path.

Here is a minimal repro:

#! mprof run
from marshmallow import fields, Schema

class ArtistSchema(Schema):
    name = fields.Str()

for _ in range(100000):
    ArtistSchema().dumps({"name": "David Bowie"})

If you remove the field, the leak goes away.

#! mprof run
from marshmallow import fields, Schema

class ArtistSchema(Schema):
    pass

for _ in range(100000):
    ArtistSchema().dumps({"name": "David Bowie"})

I suspect this is caused by circular references between the schema and the field. The schema has a list of fields. The field has a parent attribute. We would need to use a weakref proxy for the schema reference in the field to ensure that the tree unravels when the schema is ready to be released.

https://docs.python.org/3/library/weakref.html

[deckar01]

I confirmed that field_obj._bind_to_schema(field_name, proxy(self)) fixes the memory leak. Unfortunately wekrefs break a couple of tests that use lambdas and I believe that is fundamentally unfixable.

The garbage collector is supposed to be able to detect circular references, but we are probably doing something dynamic that it can't infer. Clearing the circular references in Schema.__del__ and Field.__del__ causes the garbage collector to behave correctly.

[deckar01]

I just ran the longer profile test on the dev branch and realized that the garbage collector does eventually cleanup these circular references on its own. I should have noticed that the memory usage leveled off around 3s in my very first test. I suspect the garbage collector has a strategy based on the percent of currently used memory. Everything appears to be working as intended.