v1.8-beta3 breaks StatProfilerHTML.jl

[tkluck]

Version v1.8.0-beta3 breaks the current release of StatProfilerHTML.jl with the following error message:

KeyError: key 0x0000000000000001 not found

(Full console input/output and versioninfo() pasted below.)

It was a bit tricky to bisect because there's a few other commits at which profiling and/or flamegraphs are broken for various reasons, but I'm now quite confident (by bisecting on grepping the error message) that this is caused by e6fa3ec, i.e. by #43816.

I'll continue to debug and see if I can work around it in my package, but ideally we find out how we can prevent a regression in the stdlib itself.

---

Full console input/output to reproduce:

[I] tkluck@tkluck-xps /t/empty_environment> julia --project=.
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.0-DEV.1442 (2022-02-03)
 _/ |\__'_|_|_|\__'_|  |  Commit e6fa3ec44e* (104 days old master)
|__/                   |

julia> versioninfo()
Julia Version 1.8.0-DEV.1442
Commit e6fa3ec44e* (2022-02-03 01:52 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.0 (ORCJIT, skylake)

(empty_environment) pkg> st
Status `/tmp/empty_environment/Project.toml`
  [a8a75453] StatProfilerHTML v1.4.1

(empty_environment) pkg> test StatProfilerHTML
     Testing StatProfilerHTML
      Status `/tmp/jl_jZpEqY/Project.toml`
  [864edb3b] DataStructures v0.18.12
  [08572546] FlameGraphs v0.2.10
  [0bc81568] HAML v0.3.5
  [a8a75453] StatProfilerHTML v1.4.1
  [ade2ca70] Dates `@stdlib/Dates`
  [9abbd945] Profile `@stdlib/Profile`
  [ea8e919c] SHA v0.7.0 `@stdlib/SHA`
  [8dfed614] Test `@stdlib/Test`
      Status `/tmp/jl_jZpEqY/Manifest.toml`
  [1520ce14] AbstractTrees v0.3.4
  [3da002f7] ColorTypes v0.11.2
  [5ae59095] Colors v0.12.8
  [34da2185] Compat v3.43.0
  [864edb3b] DataStructures v0.18.12
  [5789e2e9] FileIO v1.14.0
  [53c48c17] FixedPointNumbers v0.8.4
  [08572546] FlameGraphs v0.2.10
  [0bc81568] HAML v0.3.5
  [9b13fd28] IndirectArrays v1.0.0
  [1d6d02ad] LeftChildRightSiblingTrees v0.1.3
  [bac558e1] OrderedCollections v1.4.1
  [189a3867] Reexport v1.2.2
  [ae029012] Requires v1.3.0
  [a8a75453] StatProfilerHTML v1.4.1
  [0dad84c5] ArgTools v1.1.1 `@stdlib/ArgTools`
  [56f22d72] Artifacts `@stdlib/Artifacts`
  [2a0f44e3] Base64 `@stdlib/Base64`
  [ade2ca70] Dates `@stdlib/Dates`
  [8bb1440f] DelimitedFiles `@stdlib/DelimitedFiles`
  [8ba89e20] Distributed `@stdlib/Distributed`
  [f43a241f] Downloads v1.6.0 `@stdlib/Downloads`
  [7b1f6079] FileWatching `@stdlib/FileWatching`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [b27032c2] LibCURL v0.6.3 `@stdlib/LibCURL`
  [76f85450] LibGit2 `@stdlib/LibGit2`
  [8f399da3] Libdl `@stdlib/Libdl`
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [a63ad114] Mmap `@stdlib/Mmap`
  [ca575930] NetworkOptions v1.2.0 `@stdlib/NetworkOptions`
  [44cfe95a] Pkg v1.8.0 `@stdlib/Pkg`
  [de0858da] Printf `@stdlib/Printf`
  [9abbd945] Profile `@stdlib/Profile`
  [3fa0cd96] REPL `@stdlib/REPL`
  [9a3f8284] Random `@stdlib/Random`
  [ea8e919c] SHA v0.7.0 `@stdlib/SHA`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [1a1011a3] SharedArrays `@stdlib/SharedArrays`
  [6462fe0b] Sockets `@stdlib/Sockets`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [10745b16] Statistics `@stdlib/Statistics`
  [fa267f1f] TOML v1.0.0 `@stdlib/TOML`
  [a4e569a6] Tar v1.10.0 `@stdlib/Tar`
  [8dfed614] Test `@stdlib/Test`
  [cf7118a7] UUIDs `@stdlib/UUIDs`
  [4ec0a83e] Unicode `@stdlib/Unicode`
  [e66e0078] CompilerSupportLibraries_jll v0.5.0+0 `@stdlib/CompilerSupportLibraries_jll`
  [deac9b47] LibCURL_jll v7.73.0+4 `@stdlib/LibCURL_jll`
  [29816b5a] LibSSH2_jll v1.9.1+2 `@stdlib/LibSSH2_jll`
  [c8ffd9c3] MbedTLS_jll v2.24.0+2 `@stdlib/MbedTLS_jll`
  [14a3606d] MozillaCACerts_jll v2020.7.22 `@stdlib/MozillaCACerts_jll`
  [4536629a] OpenBLAS_jll v0.3.17+2 `@stdlib/OpenBLAS_jll`
  [83775a58] Zlib_jll v1.2.12+1 `@stdlib/Zlib_jll`
  [8e850b90] libblastrampoline_jll v4.0.0+0 `@stdlib/libblastrampoline_jll`
  [8e850ede] nghttp2_jll v1.41.0+1 `@stdlib/nghttp2_jll`
  [3f19e933] p7zip_jll v16.2.1+1 `@stdlib/p7zip_jll`
     Testing Running tests...
StatPofilerHTML: Error During Test at /home/tkluck/.julia/packages/StatProfilerHTML/FRimo/test/runtests.jl:12
  Got exception outside of a @test
  KeyError: key 0x0000000000000001 not found
  Stacktrace:
    [1] getindex
      @ ./dict.jl:498 [inlined]
    [2] StatProfilerHTML.Reports.Report(data::Vector{UInt64}, litrace::Dict{UInt64, Vector{Base.StackTraces.StackFrame}}, from_c::Bool)
      @ StatProfilerHTML.Reports ~/.julia/packages/StatProfilerHTML/FRimo/src/Reports.jl:106
    [3] statprofilehtml(data::Vector{UInt64}, litrace::Dict{UInt64, Vector{Base.StackTraces.StackFrame}}; from_c::Bool, path::String)
      @ StatProfilerHTML ~/.julia/packages/StatProfilerHTML/FRimo/src/StatProfilerHTML.jl:23
    [4] statprofilehtml (repeats 3 times)
      @ ~/.julia/packages/StatProfilerHTML/FRimo/src/StatProfilerHTML.jl:17 [inlined]
    [5] (::var"#2#4")()
      @ Main ~/.julia/packages/StatProfilerHTML/FRimo/test/runtests.jl:17
    [6] cd(f::var"#2#4", dir::String)
      @ Base.Filesystem ./file.jl:112
    [7] #1
      @ ~/.julia/packages/StatProfilerHTML/FRimo/test/runtests.jl:14 [inlined]
    [8] mktempdir(fn::var"#1#3", parent::String; prefix::String)
      @ Base.Filesystem ./file.jl:763
    [9] mktempdir(fn::Function, parent::String) (repeats 2 times)
      @ Base.Filesystem ./file.jl:761
   [10] macro expansion
      @ ~/.julia/packages/StatProfilerHTML/FRimo/test/runtests.jl:13 [inlined]
   [11] macro expansion
      @ ~/src/julia/usr/share/julia/stdlib/v1.8/Test/src/Test.jl:1356 [inlined]
   [12] top-level scope
      @ ~/.julia/packages/StatProfilerHTML/FRimo/test/runtests.jl:13
   [13] include(fname::String)
      @ Base.MainInclude ./client.jl:476
   [14] top-level scope
      @ none:6
   [15] eval
      @ ./boot.jl:368 [inlined]
   [16] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:276
   [17] _start()
      @ Base ./client.jl:522
Test Summary:   | Error  Total  Time
StatPofilerHTML |     1      1  6.0s
ERROR: LoadError: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.
in expression starting at /home/tkluck/.julia/packages/StatProfilerHTML/FRimo/test/runtests.jl:12
ERROR: Package StatProfilerHTML errored during testing

[tkluck]

I have a guess: before e6fa3ec, getdict!(dict, data) guarantees that every value in data is a valid key in dict; after that commit that's only the case for every value in strip_meta(data).

[KristofferC]

Ref kimikage/ProfileSVG.jl#64

[IanButterworth]

It's not valid to do an ip lookup for the metadata fields, so you have to handle them. #45403 isn't the right way to fix this. The downstream package needs to change the way it uses Profile internals.

Either strip them out entirely or handle iterating over them and don't expect them to be in the ip dict.

Check out ProfileView for an example of how that's handled.

[IanButterworth]

This should be the minimal patch tkluck/StatProfilerHTML.jl#29 and I added a TODO as handling and using the metadata would be more helpful to the user

[tkluck]

Many thanks to @IanButterworth for taking care of this issue. I'll close it now.