Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Heap snapshot #42286

Closed
wants to merge 107 commits into from
Closed
Show file tree
Hide file tree
Changes from 106 commits
Commits
Show all changes
107 commits
Select commit Hold shift + click to select a range
44a74ea
add empty files
vilterp Sep 16, 2021
e207adf
Initial dummy implementation
NHDaly Sep 16, 2021
8e0c399
dummy print
NHDaly Sep 16, 2021
2f591be
make dummy callable
NHDaly Sep 16, 2021
ae84768
Add us to the makefile
NHDaly Sep 16, 2021
9084709
set up global heap snapshot and serializer function
vilterp Sep 16, 2021
7b9e929
it builds!
vilterp Sep 16, 2021
26f15c2
fix commas
vilterp Sep 16, 2021
e3713bb
add nodes; print addresses correctly
vilterp Sep 16, 2021
208a5ea
fill out metadata header
vilterp Sep 16, 2021
ec48cff
factor out string table
vilterp Sep 16, 2021
6e78dad
Remove stderr println in record_edge
vilterp Sep 16, 2021
5fcdd6d
Plug the snapsshotter into GC fo real do
NHDaly Sep 16, 2021
e398fe0
add todos
vilterp Sep 16, 2021
6db1ea5
direct output to a stream
vilterp Sep 16, 2021
00506ca
fix commas
NHDaly Sep 16, 2021
7671eff
rename to take_heap_snapshot
vilterp Sep 17, 2021
612c6b8
add Julia wrapper function GC.take_heap_snapshot
vilterp Sep 17, 2021
cb0c809
Add commented out code for getting size once we know the type
NHDaly Sep 17, 2021
3087e8f
Make every node an "object" type for now.
NHDaly Sep 17, 2021
c78e210
update hardcoded string tables
vilterp Sep 17, 2021
632b52d
trying to get type but also seggfault
NHDaly Sep 17, 2021
ace0d4e
don't write out empty arrays which confuse chrome
vilterp Sep 17, 2021
9471734
Re-enable the (Crashing) type
NHDaly Sep 17, 2021
5990e8a
guard against nullptr types for non-objects
NHDaly Sep 17, 2021
aa635a6
Start to support size and name, but still crashing, so i commented out
NHDaly Sep 17, 2021
dfb017a
Print node/edge counts & fix memory leak & build warnings
NHDaly Sep 17, 2021
e77c09e
Use node indexes instead of IDs
NHDaly Sep 17, 2021
d567018
fixup writing indices correctly
NHDaly Sep 17, 2021
81890b1
Added edge counts to the Node records
NHDaly Sep 17, 2021
e08f16b
DUH: OBJECTS NEED SIZES TO SHOW UP IN THE HEAP SNAPSHOT!
NHDaly Sep 17, 2021
be0491a
Get the size iff it's a datatype!! that's progress!
NHDaly Sep 17, 2021
348b327
bugfix for DataType size
NHDaly Sep 17, 2021
8d40da8
Tried adding the object type to the snapshots
NHDaly Sep 20, 2021
b6a4e86
get jl_value_t* from jl_taggedvalue_t*
vchuravy Sep 20, 2021
b923c90
Use jl_value_t* to get name and size
vchuravy Sep 20, 2021
7f4b068
Pretty-print type names!!! :D
NHDaly Sep 20, 2021
e823805
JSON-escape strings before printing them
NHDaly Sep 20, 2021
5b1277c
add assert (fails) that the edges are in order
NHDaly Sep 20, 2021
1100ddb
add hacked fieldname to gc_debug_edge
NHDaly Sep 20, 2021
02603d2
Change to the real format now that we understand it:
NHDaly Sep 20, 2021
128eefb
Fix node_type and edge_type string table printing
NHDaly Sep 20, 2021
ff20542
Fix node_name and edge_name serialization
NHDaly Sep 20, 2021
2044c6f
Add todo list
NHDaly Sep 20, 2021
b6a5b03
remove TODO on the bitwise or-ing, it isn't needed
NHDaly Sep 20, 2021
dea884c
Re-Split out the debug functions
NHDaly Sep 20, 2021
62da715
better organize TODO
NHDaly Sep 20, 2021
2a97716
Change all GC callsites to use newly named debug functions
NHDaly Sep 20, 2021
e02330c
Woohoo~ Restructured code to support last major changes
NHDaly Sep 20, 2021
83f7da6
Implement Field Names!!!!!!!!! woohooo
NHDaly Sep 20, 2021
c2dcb53
Update TODO list
NHDaly Sep 20, 2021
4f6a03c
Fix normal path GC code to only debug when enabled.
NHDaly Sep 21, 2021
392c590
Fix major bug in field name string table indexing:
NHDaly Sep 21, 2021
556c780
Fix some bugs with registering edges from Modules :)
NHDaly Sep 21, 2021
3b83fd4
Handle field names of NamedTuples without crashing
NHDaly Sep 21, 2021
6397269
rename _record_gc_node => _record_gc_edge
vilterp Sep 21, 2021
b01cd81
capture string and symbol values; tag them for viewer
vilterp Sep 21, 2021
fe63567
string and symbol sizes
vilterp Sep 21, 2021
3337ba5
add gc annotations
vilterp Sep 22, 2021
fc77bfa
add gc-heap-snapshot.h to gc.o dependency list
vchuravy Sep 22, 2021
ad6f1d5
[GCHEAP] add JL_NOTSAFEPOINT
vchuravy Sep 22, 2021
8b8cb43
Disable GC Snapshotting w/ inline check for performance
NHDaly Sep 22, 2021
3e11756
mark the gc_heap_snapshot_enabled check __unlikely
NHDaly Sep 22, 2021
fc29e39
fix annotations on new inline funcs
NHDaly Sep 22, 2021
e612f29
revert to just type name
vilterp Sep 22, 2021
d9116bb
start adding roots
vilterp Sep 22, 2021
a1cee9e
function to add edges from uber root to roots
vilterp Sep 22, 2021
8c3bb4a
add uber root and gc roots object
vilterp Sep 22, 2021
6f7fd5a
attempt to add subroots node and edges
vilterp Sep 22, 2021
21e0f7f
add internal_root
vchuravy Sep 22, 2021
0daf9ac
Fix segfault in _gc_heap_snapshot_record_root()
NHDaly Sep 23, 2021
8c84a64
fix edge count
vilterp Sep 23, 2021
dce89e9
make gc root recording conditional on heap snapshotting being on
vilterp Sep 23, 2021
6805334
mark gc_heap_snapshot_record_root as JL_NOTSAFEPOINT
vilterp Sep 23, 2021
bed9004
better property names for gc roots
vilterp Sep 24, 2021
5e801cb
Apply suggestions from code review
NHDaly Sep 27, 2021
c470ea9
more PR feedback
vilterp Sep 28, 2021
a1f0988
comments pointing at at Node source
vilterp Sep 28, 2021
4266a5c
use ios_t instead of JL_STREAM
vilterp Sep 28, 2021
47fe30d
print full types
vilterp Sep 28, 2021
ae7bf34
array sizes
vilterp Sep 30, 2021
e047ef3
try marking roots in mark_enqueued_tasks
vilterp Sep 30, 2021
188ed05
add internal edges when scanning the stack
vilterp Sep 30, 2021
a540560
Halfway through draft commit to construct full fieldpath from slot
NHDaly Oct 7, 2021
59b0eda
Start hooking up the field paths
NHDaly Oct 7, 2021
152ad15
connect up the code from before (but it's still not working)
NHDaly Oct 13, 2021
ddd07ad
Add debug logging for objects in `Main` module
NHDaly Oct 13, 2021
2da28df
Fix silly mistake in slot accounting :)
NHDaly Oct 13, 2021
09ddcd4
Clean up debug logs
NHDaly Oct 14, 2021
bd87eea
represent stack frames as nodes; only root running tasks
vilterp Oct 5, 2021
83afa81
tweak comment
vilterp Oct 7, 2021
ba4ea3f
Fixups, comments, todos
NHDaly Oct 14, 2021
9c04a81
fix merge conflict
NHDaly Oct 14, 2021
d43ca3a
Fix Tuple types in fieldpath_for_slot
NHDaly Oct 14, 2021
ee74794
Add TODO about overrunning type printing buffer
NHDaly Oct 14, 2021
0caead0
remove log
NHDaly Oct 14, 2021
a1f1501
Avoid recollect which might have been mucking our results
NHDaly Oct 22, 2021
95b4592
Hacky fix to make sure we get full marks in GC:
NHDaly Nov 18, 2021
b441c95
Make type string buffer 1MiB to (try to) avoid buffer overflow
NHDaly Nov 26, 2021
7fa3781
unbreak heap-snapshot.cpp
vilterp Dec 9, 2021
fb4fbd3
unbreak partr.c
vilterp Dec 9, 2021
6ab9e15
remove println
vilterp Dec 9, 2021
e8b32da
put things back I accidentally removed in the rebase
vilterp Dec 10, 2021
00f5451
add wrapper function
vilterp Feb 3, 2022
227cf9b
Merge branch 'master' into pv-heap-snapshot
vilterp Feb 3, 2022
07b3ba1
fix syntax error in makefile
vilterp Feb 3, 2022
e732d48
Fix copy/paste typos in gc_heap_snapshot_record_roots for jl_gc_queue…
NHDaly Jun 2, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions base/gcutils.jl
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,23 @@ Control whether garbage collection is enabled using a boolean argument (`true` f
"""
enable(on::Bool) = ccall(:jl_gc_enable, Int32, (Int32,), on) != 0

"""
GC.take_heap_snapshot(io::IOStream)
GC.take_heap_snapshot(filepath::String)

Write a snapshot of the heap, in the JSON format expected by the Chrome
Devtools Heap Snapshot viewer (.heapsnapshot extension), to the given
IO stream.
"""
function take_heap_snapshot(io)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
function take_heap_snapshot(io)
function take_heap_snapshot(io::IOStream)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tried that earlier, but it seems not to be defined yet:

$ make
...
LoadError(at "sysimg.jl" line 3: LoadError(at "Base.jl" line 97: LoadError(at "gcutils.jl" line 119: UndefVarError(var=:IOStream))))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like this made the builds fail, e.g. https://build.julialang.org/#/builders/63/builds/3778/steps/8/logs/stdio

either we need to get IOStream in scope somehow, or take the annotation back off

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

taking the annotation back off until we can figure out how to get it in scope in this file… I guess we have to figure out what file IOStream is defined in and make sure it gets included before this

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this one has been resolved, yeah?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still may be good to acquire the internal lock on io here first also

ccall(:jl_gc_take_heap_snapshot, Cvoid, (Ptr{Cvoid},), (io::IOStream).handle::Ptr{Cvoid})
end
function take_heap_snapshot(filepath::String)
open(filepath, "w") do io
take_heap_snapshot(io)
end
end

vilterp marked this conversation as resolved.
Show resolved Hide resolved
"""
GC.enable_finalizers(on::Bool)

Expand Down
7 changes: 4 additions & 3 deletions src/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,9 @@ RUNTIME_SRCS := \
jltypes gf typemap smallintset ast builtins module interpreter symbol \
dlload sys init task array dump staticdata toplevel jl_uv datatype \
simplevector runtime_intrinsics precompile \
threading partr stackwalk gc gc-debug gc-pages gc-stacks gc-alloc-profiler method \
jlapi signal-handling safepoint timing subtype \
threading partr stackwalk \
gc gc-debug gc-heap-snapshot gc-alloc-profiler gc-pages gc-stacks \
method jlapi signal-handling safepoint timing subtype \
crc32c APInt-C processor ircode opaque_closure codegen-stubs coverage
SRCS := jloptions runtime_ccall rtutils
ifeq ($(OS),WINNT)
Expand Down Expand Up @@ -289,7 +290,7 @@ $(BUILDDIR)/disasm.o $(BUILDDIR)/disasm.dbg.obj: $(SRCDIR)/debuginfo.h $(SRCDIR)
$(BUILDDIR)/dump.o $(BUILDDIR)/dump.dbg.obj: $(addprefix $(SRCDIR)/,common_symbols1.inc common_symbols2.inc builtin_proto.h serialize.h)
$(BUILDDIR)/gc-debug.o $(BUILDDIR)/gc-debug.dbg.obj: $(SRCDIR)/gc.h
$(BUILDDIR)/gc-pages.o $(BUILDDIR)/gc-pages.dbg.obj: $(SRCDIR)/gc.h
$(BUILDDIR)/gc.o $(BUILDDIR)/gc.dbg.obj: $(SRCDIR)/gc.h $(SRCDIR)/gc-alloc-profiler.h
$(BUILDDIR)/gc.o $(BUILDDIR)/gc.dbg.obj: $(SRCDIR)/gc.h $(SRCDIR)/gc-heap-snapshot.h $(SRCDIR)/gc-alloc-profiler.h
$(BUILDDIR)/init.o $(BUILDDIR)/init.dbg.obj: $(SRCDIR)/builtin_proto.h
$(BUILDDIR)/interpreter.o $(BUILDDIR)/interpreter.dbg.obj: $(SRCDIR)/builtin_proto.h
$(BUILDDIR)/jitlayers.o $(BUILDDIR)/jitlayers.dbg.obj: $(SRCDIR)/jitlayers.h $(SRCDIR)/codegen_shared.h
Expand Down
6 changes: 3 additions & 3 deletions src/gc-debug.c
Original file line number Diff line number Diff line change
Expand Up @@ -1205,10 +1205,10 @@ void gc_count_pool(void)
jl_safe_printf("************************\n");
}

int gc_slot_to_fieldidx(void *obj, void *slot)
int gc_slot_to_fieldidx(void *obj, void *slot) JL_NOTSAFEPOINT
{
jl_datatype_t *vt = (jl_datatype_t*)jl_typeof(obj);
int nf = (int)jl_datatype_nfields(vt);
int nf = (int)jl_datatype_nfields(vt); // what happens if you're inlined? lol
for (int i = 0; i < nf; i++) {
void *fieldaddr = (char*)obj + jl_field_offset(vt, i);
if (fieldaddr >= slot) {
Expand All @@ -1218,7 +1218,7 @@ int gc_slot_to_fieldidx(void *obj, void *slot)
return -1;
}

int gc_slot_to_arrayidx(void *obj, void *_slot)
int gc_slot_to_arrayidx(void *obj, void *_slot) JL_NOTSAFEPOINT
vtjnash marked this conversation as resolved.
Show resolved Hide resolved
{
char *slot = (char*)_slot;
jl_datatype_t *vt = (jl_datatype_t*)jl_typeof(obj);
Expand Down
Loading