investigate memory utilization post-native engine

[kwlzn]

currently, running ./pants --enable-v2-engine list :: in our monorepo leads to memory utilization of ~7GB in the main pants python2.7 process on my OSX machine. this is likely problematic for obvious reasons.

we'll want to dig into optimizing the memory footprint now that the native engine has landed.

[JieGhost]

By "main" pants process do you mean the main daemon process? or you are not using daemon?

[kwlzn]

@JieGhost the main pants' python2.7 process without the daemon.

[JieGhost]

@kwlzn Did python-backed v2 engine also consume that much memory when doing "list ::"?

[kwlzn]

@JieGhost looks like the peak memory utilization for a v1 list :: is ~921MB on my machine - so, considerably lower.

[JieGhost]

@kwlzn I mean v2 engine with python implementation. Not sure if we ever captured that before. But it should be much higher than v1 usage (921 MB), right?

[kwlzn]

@JieGhost ah, my bad. yeah, we did capture v2-python timings and iirc, it was roughly the same. we were thinking the native engine would help, but doesn't seem to have moved the needle on mem utilization at all. because of this, I suspect the high memory utilization is coming from the python side of the code vs the native bit.

[JieGhost]

I can repro the 7GB mem usage.

I checked the mem usage of ExternContext object, in particular, mem usage of _id_to_obj, _obj_to_id and _handles. I use sys.getsizeof method.

It seems _id_to_obj and _obj_to_id both uses around 50MB, and _handles uses around 134MB.
Since sys.getsizeof can only return size of the object itself, not including the size of the referenced objects in the original objects field, I added up the size of referenced objects as well.

size = 0
for k in self.scheduler._context._id_to_obj.keys():
  size += sys.getsizeof(self.scheduler._context._id_to_obj[k])

This returns around 68MB of size.

For _handles, it is around 461MB.

One thing I am not sure is whether those referenced objects have further referenced objects.
Since product graph is now in rust code, I need to find a way to figure out the mem usage there.

[stuhood]

This definitely needs to include the transitive references of the object.

[stuhood]

I think that this is probably approaching its finale once #4333, #4331, and #4334 land.

[kwlzn]

sgtm