Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GC metric last_incremental_sweep #50190

Merged
merged 3 commits into from
Jul 6, 2023

Conversation

li1
Copy link
Contributor

@li1 li1 commented Jun 16, 2023

Records the time that the last incremental sweep ran.

This allows an application that explicitly runs GC when idle or at convenient points to be more intelligent about it.

Related: #50018

Copy link
Contributor

@kpamnany kpamnany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cc: @gbaraldi, @vchuravy

The set of useful/necessary GC metrics keeps growing and making this struct larger and messier. Maybe we should consider a way to reorganize and retrieve these metrics?

src/gc.c Outdated Show resolved Hide resolved
@gbaraldi
Copy link
Member

gbaraldi commented Jun 16, 2023

I think we shoud separate this struct into a couple things, it currently has a bunch of things that the user shouldn't really care about. #50144 adds a new struct that has the heuristics only. I think we should have 3 structs, a per thread gc_num that just counts allocations. A global one that tracks the timings and such, and the global heap tracking one I added in that PR.

With this we can remove a bunch of things, freed doesn't really mean much on the per thread thing for example

@oscardssmith
Copy link
Member

Do we need the per thread one? If we track pages rather than bytes (with your PR), we don't need to ever look at how much each thread allocated. (since the decision on when to GC happens when we ask for a new page which can take a lock to check the global heap one)

@gbaraldi
Copy link
Member

We need the per thread one to see how much a function call allocated for example for @time.

@oscardssmith
Copy link
Member

How does that work if the function switches threads while running (or spawns work onto another thread?)

@gbaraldi
Copy link
Member

I think it just gets it wrong? I haven't tested it

@oscardssmith
Copy link
Member

if so, would it be much worse to delete the per thread one and assume that people using @time to track allocations aren't also running allocating code on another thread?

@gbaraldi
Copy link
Member

The page based thing isn't precise for number of allocations, it just measures the heap it doesn't know anything about what is being done with it, so if you allocate 100GB but never go over 500mb of max heap that's what it will show.

@oscardssmith
Copy link
Member

makes sense.

@kpamnany
Copy link
Contributor

Happy to have this reorganized, but for now...

i686-w64-mingw32 failed with:

2023-06-20 04:10:26 EDT	Collecting and executing precompile statements
2023-06-20 04:10:26 EDT	└ Collect (Basic: ✓ 819, REPL 38/39: ◐ 1818) => Execute ◐ 1956LLVM ERROR: out of memory
2023-06-20 04:10:26 EDT	Allocation failed
2023-06-20 04:10:26 EDT	
2023-06-20 04:10:26 EDT	[6312] signal (22): SIGABRT
2023-06-20 04:10:26 EDT	in expression starting at none:0
2023-06-20 04:10:26 EDT	crt_sig_handler at C:/workdir/src\signals-win.c:100
2023-06-20 04:10:26 EDT	raise at C:\Windows\System32\msvcrt.dll (unknown line)
2023-06-20 04:10:26 EDT	abort at C:\Windows\System32\msvcrt.dll (unknown line)
2023-06-20 04:10:26 EDT	.text$_ZN4llvm22report_bad_alloc_errorEPKcb at C:\workdir\usr\bin\libLLVM-15jl.dll (unknown line)
2023-06-20 04:10:26 EDT	Allocations: 86146492 (Pool: 86098271; Big: 48221); GC: 249

Any idea what to do about this? Need a larger build machine?

@d-netto
Copy link
Member

d-netto commented Jun 20, 2023

#50205

kpamnany added a commit to RelationalAI/julia that referenced this pull request Jun 20, 2023
Records the time that the last incremental sweep ran.
kpamnany added a commit to RelationalAI/julia that referenced this pull request Jun 22, 2023
Records the time that the last incremental sweep ran.
@d-netto d-netto force-pushed the ms/last_incremental_sweep branch from bbf89d9 to 2179e0e Compare June 26, 2023 04:06
kpamnany added a commit to RelationalAI/julia that referenced this pull request Jun 29, 2023
Records the time that the last incremental sweep ran.
@kpamnany kpamnany merged commit 64ab537 into JuliaLang:master Jul 6, 2023
kpamnany pushed a commit to RelationalAI/julia that referenced this pull request Jul 6, 2023
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
kpamnany pushed a commit to RelationalAI/julia that referenced this pull request Jul 27, 2023
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
kpamnany pushed a commit to RelationalAI/julia that referenced this pull request Aug 9, 2023
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
kpamnany pushed a commit to RelationalAI/julia that referenced this pull request Aug 21, 2023
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
quinnj pushed a commit to RelationalAI/julia that referenced this pull request Sep 5, 2023
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
quinnj pushed a commit to RelationalAI/julia that referenced this pull request Sep 5, 2023
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Sep 14, 2023
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Sep 15, 2023
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Sep 18, 2023
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Sep 18, 2023
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Jan 17, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Jan 18, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Jan 23, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Jan 24, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Jan 25, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Jan 27, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Jan 30, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Jan 31, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Feb 1, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Feb 6, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Feb 7, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Feb 14, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Feb 21, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Feb 22, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
Drvi pushed a commit to RelationalAI/julia that referenced this pull request Feb 28, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Mar 1, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Mar 13, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
Drvi pushed a commit to RelationalAI/julia that referenced this pull request Apr 3, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
d-netto pushed a commit to RelationalAI/julia that referenced this pull request Apr 16, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Apr 23, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Apr 24, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Apr 30, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request Apr 30, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request May 2, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request May 9, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request May 19, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request May 26, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request May 28, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
DelveCI pushed a commit to RelationalAI/julia that referenced this pull request May 29, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
Drvi pushed a commit to RelationalAI/julia that referenced this pull request Jun 7, 2024
* Add GC metric `last_incremental_sweep`

* Update gc.c

* Update gc.c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants