Too much Gen0 memory usage on Region GC

[InCerryGit]

Description

I want to experience region GC on .NET6.0, I just tried this libclrgc.so in my test environment and .NET6.0 is able to use it. But it has a very high memory usage.

Use dotMemory analyze the dump, gen0 has 173 heaps.

Reproduction Steps

Create a .NET6 project, place the following *.so in the same directory as coreclr.so

https://github.com/Maoni0/clrgc/tree/main/issues/80073/demotion_fix/v2/linux

Configure environment variables:

DOTNET_GCName=libclrgc.so

Run the program。

Expected behavior

There shouldn't be so much memory usage。

Actual behavior

Memory usage is several times higher than SegmentGC

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

cc @Maoni0

[ghost]

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I want to experience region GC on .NET6.0, I just tried this libclrgc.so in my test environment and .NET6.0 is able to use it. But it has a very high memory usage.

Use dotMemory analyze the dump, gen0 has 173 heaps.

Reproduction Steps

Create a .NET6 project, place the following *.so in the same directory as coreclr.so

https://github.com/Maoni0/clrgc/tree/main/issues/80073/demotion_fix/v2/linux

Configure environment variables:

DOTNET_GCName=libclrgc.so

Run the program。

Expected behavior

There shouldn't be so much memory usage。

Actual behavior

Memory usage is several times higher than SegmentGC

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

cc @Maoni0

Author:	InCerryGit
Assignees:	-
Labels:	area-GC-coreclr, untriaged
Milestone:	-

[InCerryGit]

NET7 has the same issue use the libclrgc.dll linked below.
https://github.com/Maoni0/clrgc/tree/main/issues/80073/demotion_fix/v2/linux

.NET Version:

Host:
  Version:      7.0.2
  Architecture: x64
  Commit:       d037e070eb

.NET SDKs installed:
  No SDKs were found.

.NET runtimes installed:
  Microsoft.AspNetCore.App 7.0.2 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 7.0.2 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

System version：

bash#: cat /etc/alpine-release
3.17.1

[mangod9]

@InCerryGit, could you please clarify your last comment about .NET 7? Regions are enabled by default in 7, so if you are using libclrgc.so in 7, then you are falling back to segments.

[InCerryGit]

@InCerryGit, could you please clarify your last comment about .NET 7? Regions are enabled by default in 7, so if you are using libclrgc.so in 7, then you are falling back to segments.

Sorry, I forgot to explain this, I used the libclrgc.dll linked below instead of the one in the .NET7 SDK. @mangod9

https://github.com/Maoni0/clrgc/tree/main/issues/80073/demotion_fix/v2/linux

[Maoni0]

the only difference between that libclrgc.dll and the .net 7.0 version you are using is

1. demotion fix - Fixing a bug that causes us to mistakenly demote a gen2 region to gen0 #82413
2. decommit fix - Decommit memory before returning to the region_allocator #80640
3. special_sweep_p fix only for workstation GC - Fix special sweep issue for workstation #81248
4. a premature OOM fix - [.NET 7] Fix spurious OOMs after upgrading to 7 #79158 (this was just because that was my baseline which had that change not included in 7.0.2)

it's doubtful any of these would cause dramatic changes in heap usage.

I also have some instrumentation which takes 22,528 bytes per heap and 3KiB for all heaps. each heap_segment filed is increased by 8 bytes. so that can't be contributing to such dramatic increase.

I have no idea what "173 heaps" means. I don't use dotMemory so I'm not familiar with its terminology.

if you have a top level GC trace when running with 7.0 and with the libclrgc build, we can take a look.

[Maoni0]

also just making sure, when you say .net 7, you mean you are using .net 7 with no COMPlus_GCName env var specified, correct?

[InCerryGit]

also just making sure, when you say .net 7, you mean you are using .net 7 with no COMPlus_GCName env var specified, correct?

Yes, the .NET7 mentioned above is the one without COMPlus_GCName set. I uploaded the .NET6 dmp to onedrive. @Maoni0
https://1drv.ms/u/s!AoKi8-KT2tEKj5VUDDcX7s3OqhuIfA?e=etBZ4Q

[Maoni0]

do you have a top level GC trace for with and without that libcoreclr? how to collect one is described in mem-doc: https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#how-to-collect-top-level-gc-metrics

I also don't have the pwd to your share. but dumps aren't a good way to diagnose this kind of issues. collecting a top level GC trace for both before and after is the best way to get started.

[InCerryGit]

do you have a top level GC trace for with and without that libcoreclr? how to collect one is described in mem-doc: https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md#how-to-collect-top-level-gc-metrics

I also don't have the pwd to your share. but dumps aren't a good way to diagnose this kind of issues. collecting a top level GC trace for both before and after is the best way to get started.

I removed the pwd of dmp. is it possible for me to capture using the following command?

dotnet-trace collect -p 43 --profile gc-verbose

@Maoni0

[Maoni0]

please use the cmd from the link above -

dotnet trace collect -p <pid> -o <outputpath with .nettrace extension> --profile gc-collect --duration <in hh:mm:ss format>

don't use gc-verbose. it just adds overhead for no benefit. you don't have to do --duration if you are going to manually stop it.

[InCerryGit]

The trace file has been uploaded, the dump file is too big, I will upload it to the onedrive later。
@Maoni0
https://1drv.ms/u/s!AoKi8-KT2tEKj5VVkG3TOnnNAfi0mw?e=XP1aYV

NET7 without libclrgc.so nomaly

NET7 use libclrgc.so abnomaly
https://github.com/Maoni0/clrgc/tree/main/issues/80073/demotion_fix/v2/linux

[mangod9]

@InCerryGit wanted to understand why you are running with libclrgc here. With .NET 7, regions are enabled by default, and from your charts above it doesnt seem to show any memory regression?

[Maoni0]

@mangod9, @InCerryGit wanted to experiment regions on 6.0 so he wanted a clrgc with regions for that (unfortunately we have yet to ship clrgc.dlls with regions; would be great if we started to ship those. but I digress). but I'm glad he did try clrgc because it looks like we have a problem on linux - the reason why the heap size is a lot larger with clrgc is the gen0 min budget is huge - it's about 167mb with clrgc and 16mb with coreclr.

the trace with coreclr -

<GlobalHeapHistory FinalYoungestDesired="16,387,456" NumHeaps="8" CondemnedGeneration="0" Gen0ReductionCount="0" Reason="AllocSmall" GlobalMechanisms="Compaction, Promotion, Demotion, CardBundles"/>

the trace with clrgc

<GlobalHeapHistory FinalYoungestDesired="167,772,168" NumHeaps="8" CondemnedGeneration="0" Gen0ReductionCount="0" Reason="AllocSmall" GlobalMechanisms="Compaction, Promotion, Demotion, CardBundles"/>

is this something you could possibly take a look at? it may have something to do with the changes we made to PAL.

[InCerryGit]

@InCerryGit wanted to understand why you are running with libclrgc here. With .NET 7, regions are enabled by default, and from your charts above it doesnt seem to show any memory regression?

@mangod9 Like maoni said, at first I just wanted to experience regiongc on net6.0, so I used libclrgc built by maoni, the memory usage was several times higher. And this problem was also reproduced in net7.0.

about the memory regression, it is the same environment and load in the figure, normally the application should only take about 400mb of memory, but libclrgc takes 1.5gb.

[mangod9]

Hello @InCerryGit, does this issue repro for you on different hardware? I just tried today and I cannot repro it when running with libclrgc.so (with regions enabled) against .net 7 build. If it repros consistently, can you provide the full repro steps?

Sorry for the delay in response.

[InCerryGit]

Hello @InCerryGit, does this issue repro for you on different hardware? I just tried today and I cannot repro it when running with libclrgc.so (with regions enabled) against .net 7 build. If it repros consistently, can you provide the full repro steps?

Sorry for the delay in response.

@mangod9 I will try later today on different hardware and get back to you.

[ghost]

This issue has been marked needs-author-action and may be missing some important information.

[ghost]

This issue has been automatically marked no-recent-activity because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will remove no-recent-activity.

[ghost]

This issue will now be closed since it had been marked no-recent-activity but received no further activity in the past 14 days. It is still possible to reopen or comment on the issue, but please note that the issue will be locked if it remains inactive for another 30 days.