Dangling references prevent GC to collect instances

[vakker]

Environment

• Grid2op version: 1.6.3
• System: ubuntu18.04

Bug description

Then the env instance is deleted (e.g. by del) then GC should collect it.
It does happen in most scenarios, but in some cases there are references to the env instance that doesn't allow for the collection of obsolete instances.

How to reproduce

Running the following shows that GC does it's job:

import gc
import sys

import grid2op


class TestClass:
    pass


def main():
    for i in range(50):
        env = grid2op.make('l2rpn_icaps_2021_small')
        d = TestClass()
        print(
            'Grid2Op instances:',
            len([
                o for o in gc.get_objects()
                if isinstance(o, grid2op.Environment.BaseEnv)
            ]))
        print('Test instances:',
              len([o for o in gc.get_objects() if isinstance(o, TestClass)]))
        # print('Ref', sys.getrefcount(d))
        # print('Refs', gc.get_referrers(d))
        del d
        del env
        gc.collect()
        print(
            'Grid2Op instances:',
            len([
                o for o in gc.get_objects()
                if isinstance(o, grid2op.Environment.BaseEnv)
            ]))
        print('Test instances:',
              len([o for o in gc.get_objects() if isinstance(o, TestClass)]))
        print('#########')


main()

However, if there's a wrapper around the env (as it's required for RLlib) there are calls to env.step and the wrapper (or other parts of the framework) might store a reference to the env through the returned observation, reward or info.

In my case using deepcopy (see below) resolves the issue:

obs_grid, reward, done, info = copy.deepcopy(self.env_grid.step(action_grid))

Otherwise, there are a pile of dangling instances that doesn't get collected.
Currently I'm using the following (that's called in the reset method of my wrapper when it's needed e.g. for curriculum learning):

    def _reset_env(self):
        self.env_grid.close()

        self.env_grid = None
        self._init_all()  # runs grid2op.make 
        self.episodes = 0

        gc.collect()

        # print(
        #     'Grid2Op instances:',
        #     len([
        #         o for o in gc.get_objects()
        #         if isinstance(o, grid2op.Environment.BaseEnv)
        #     ]))

The commented out bit prints the number of grid2op env instances in the given process.

[BDonnot]

I think one of the problem is that the "observation" contains a reference to the observation_space (in the environment) that has a backend.
This is used for "simulate" and i am not sure that this is "closed" properly. I would need to double check that to be sure it's the cause of this issue.

[BDonnot]

Hello,
With the current information, i am totally unable to reproduce the bug. I know we discussed it on discord a few weeks ago but i cannot manage to reproduce it though i'm convinced that there is a bug there.

As I cannot reproduce it, it is likely that this bug will be in next release of grid2op.

[vakker]

You mean you cannot reproduce the bug with an RLlib wrapper? If it helps, I can put together a simple repro example. It's also possible that I'm doing something wrong, so let's see.

[BDonnot]

Can you put some simple code here ? If possible without using RLLIB :-/ The smallest / simplest piece of code, the easiest it will be for me to identifiy the bug and then fix it :-)

[BDonnot]

Closing at it seems solved