Add `report_invalidations` to tabulate invalidations summary #303

charleskawczynski · 2022-09-30T20:03:48Z

Hello 👋🏻! There has been a lot of activity around hunting down invalidations lately, for example:

As mentioned in this comment of the discourse thread, I've added a function called report_invalidations to ReportMetrics.jl which summarizes the output of SnoopCompileCore.@snoopr with quantitative and actionable information.

I'd like to call this function from the julia-invalidations github action. I thought it might be cleaner to add this functionality to SnoopCompile directly (rather than a separate package) since it's relatively light weight (barring PrettyTables) and uses SnoopCompile internals.

The ReportMetrics.jl version of the printed table looks like this:

[ Info: Number of invalidations for SomeWork: 7
[ Info: SomeWork: 6 total method invalidations
┌───────────────────────────────────┬─────────────┬───────────────┬─────────────────┐
│ <file name>:<line number>         │ Method Name │ Invalidations │ Invalidations % │
│                                   │             │    Number     │     (xᵢ/∑x)     │
├───────────────────────────────────┼─────────────┼───────────────┼─────────────────┤
│ SnoopCompile.jlirrationals.jl:182 │  BigFloat   │       1       │       50        │
│ SnoopCompile.jlirrationals.jl:182 │  BigFloat   │       1       │       50        │
│ SnoopCompile.jlirrationals.jl:192 │   Float32   │       0       │        0        │
│ SnoopCompile.jlirrationals.jl:191 │   Float64   │       0       │        0        │
│ SnoopCompile.jlirrationals.jl:192 │   Float32   │       0       │        0        │
│ SnoopCompile.jlirrationals.jl:191 │   Float64   │       0       │        0        │
└───────────────────────────────────┴─────────────┴───────────────┴─────────────────┘

codecov · 2022-09-30T20:08:55Z

Codecov Report

Base: 84.08% // Head: 83.70% // Decreases project coverage by -0.38% ⚠️

Coverage data is based on head (5b4a42f) compared to base (dcae62d).
Patch coverage: 87.50% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #303      +/-   ##
==========================================
- Coverage   84.08%   83.70%   -0.39%     
==========================================
  Files          17       18       +1     
  Lines        2162     2221      +59     
==========================================
+ Hits         1818     1859      +41     
- Misses        344      362      +18

Impacted Files	Coverage Δ
src/invalidations.jl	`74.06% <85.71%> (-3.74%)`	⬇️
src/report_invalidations.jl	`86.66% <86.66%> (ø)`
src/SnoopCompile.jl	`100.00% <100.00%> (ø)`
src/parcel_snoopi_deep.jl	`90.01% <0.00%> (+0.10%)`	⬆️
src/parcel_snoopi.jl	`95.00% <0.00%> (+0.85%)`	⬆️
SnoopPrecompile/src/SnoopPrecompile.jl	`94.11% <0.00%> (+6.61%)`	⬆️
SnoopCompileCore/src/snoopi_deep.jl	`100.00% <0.00%> (+8.00%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

charleskawczynski · 2022-10-26T23:02:29Z

bump 🙂

charleskawczynski · 2022-11-23T05:21:31Z

Bump! 🙂

timholy · 2022-12-01T11:00:27Z

I think we could have this, but I'd rather it be loaded under a @require PrettyTables = ... block. Often the best practice is to define the function

"""
Lotsa docs

Using `tabulated_invalidations` requires that you first load the PrettyTables package.
"""
function tabulated_invalidations end      # this defines a function with no methods, but the docstring is available.

as a "stub" in the main package, and then in __init__()

function __init__()
    # other stuff
    @require PettyTables = sha include("tabulated.jl")
end

and put the tabulated_invalidations methods in "tabulated.jl".

charleskawczynski · 2023-01-01T05:45:20Z

Ok, this now works I think as expected, here's a mwe (assuming some packages are loadable):

import SnoopCompileCore
invalidations = SnoopCompileCore.@snoopr begin
   using ForwardDiff
end;
import SnoopCompile
using PrettyTables # needed for report_invalidations to be defined
SnoopCompile.report_invalidations(;
   invalidations,
   process_filename = x -> last(split(x, ".julia/packages/"))
)

Which produces

┌ Warning: Could not attribute the following delayed invalidations:
└ @ SnoopCompile ~/Dropbox/Caltech/work/dev/clones/forks/SnoopCompile.jl/src/invalidations.jl:552
Any[MethodInstance for eltype(::Type{A} where A<:(NamedTuple{(:init,), _A} where _A<:Tuple{AbstractDict})), MethodInstance for eltype(::Type{T} where T<:Tuple{AbstractDict}), MethodInstance for eltype(::Type{A} where A<:(NamedTuple{(:maxlog, :form), _A} where _A<:Tuple{Int64, Type})), MethodInstance for eltype(::Type{T} where T<:Tuple{Int64, Type})] invalidated Core.MethodInstance[MethodInstance for StaticArrays.var"#s25#170"(::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any), MethodInstance for StaticArrays._precompile_()]
Any[MethodInstance for eltype(::Type{T} where T<:Tuple{AbstractDict}), MethodInstance for eltype(::Type{A} where A<:(NamedTuple{(:init,), _A} where _A<:Tuple{AbstractDict}))] invalidated MethodInstance for MacroTools.allbindings(::Any, ::Vector{Any})
[ Info: 233 methods invalidated for 6 functions
┌─────────────────────────────────────────────┬───────────────┬───────────────┬─────────────────┐
│ <file name>:<line number>                   │ Function Name │ Invalidations │ Invalidations % │
│                                             │               │               │     (xᵢ/∑x)     │
├─────────────────────────────────────────────┼───────────────┼───────────────┼─────────────────┤
│ ForwardDiff/QdStj/src/dual.jl:427           │ promote_rule  │      209      │       82        │
│ StaticArrays/x7lS0/src/abstractarray.jl:150 │    similar    │      18       │        7        │
│ StaticArrays/x7lS0/src/SizedArray.jl:88     │    convert    │      15       │        6        │
│ StaticArrays/x7lS0/src/mapreduce.jl:253     │   foldl##kw   │       8       │        3        │
│ ForwardDiff/QdStj/src/dual.jl:145           │    isequal    │       3       │        1        │
│ StaticArrays/x7lS0/src/SOneTo.jl:57         │  getproperty  │       3       │        1        │
└─────────────────────────────────────────────┴───────────────┴───────────────┴─────────────────┘

charleskawczynski · 2023-01-10T18:38:59Z

Based on JuliaDiff/ForwardDiff.jl#620 (comment) and JuliaDiff/ForwardDiff.jl#620 (comment), should we update how report_invalidations works? cc @ChrisRackauckas @timholy

timholy

This looks nearly merge-worthy. I don't think there are big changes needed in how you collect the invalidations, because:

You suggest collecting them with SnoopCompileCore rather than SnoopCompile ✔️
Nothing here uses a workload. Of course this summary will include many invalidations that may not affect the users' code. filtermod and precompile_blockers can restrict them to ones that affect particular modules or workloads. Are you wondering if you should support them?

src/invalidations.jl

test/snoopr.jl

timholy · 2023-01-11T13:52:03Z

src/invalidations.jl

+
+Print a tabular summary of invalidations given:
+
+ - `job_name::String` the name of the job


What's the role of the job_name? Does it have any functional significance? (Can I choose a name that's wrong or broken?)

You know, I think we can get rid of it. It's not really doing anything. I added it in case I'd run several invalidation jobs, so that the tables could be differentiated, but 1) that can be printed before the call to report_invalidations and 2) I think they'd need to be separate sessions anyway. I'll remove it

timholy · 2023-01-11T13:54:01Z

src/invalidations.jl

+ - `job_name::String` the name of the job
+
+An `@info` message also prints some information, for example
+if the table has been truncated (upon request).


This might be interpreted as saying "you have to request the @info message" whereas I think you mean "it was truncated by request," which in turn really means "it was truncated by specifying nrows."

But even this is confusing since nrows has a default value of 10, so you don't actually have to request it.

Very good feedback. I'm thinking we can skip this comment and just do a better job documenting n_rows. I'll update to something that I think is clearer/simpler

src/invalidations.jl

Co-authored-by: Tim Holy <[email protected]>

charleskawczynski · 2023-01-12T18:56:12Z

I think one last thing that we could do, that would really put this over the top in terms of ease of use, would be if we followed through with CliMA/ReportMetrics.jl#7 (comment), and made the file / line number hypertext links through PrettyTables (which I think is supported here).

I think this can be a followup feature, though. Thumbs up to open an issue 🙂

timholy · 2023-01-13T09:47:04Z

Thanks! This is a nice addition. Release of SnoopCompile 2.10.0 is underway.

This was referenced Oct 6, 2022

Create Invalidations.yml JuliaWeb/HTTP.jl#920

Merged

Create Invalidations.yml JuliaDiff/ChainRulesCore.jl#580

Merged

charleskawczynski added 2 commits December 31, 2022 13:40

Add tabulated_invalidations to tabulate inv summary

86b53f8

Apply suggestions

3082d33

charleskawczynski force-pushed the ck/tabulate_invalidations branch from f4eab32 to 3082d33 Compare December 31, 2022 21:40

charleskawczynski force-pushed the ck/tabulate_invalidations branch from c26a6e3 to 6bfd41d Compare January 1, 2023 05:47

Fixes, update msg, optional job name

a2f6e12

charleskawczynski force-pushed the ck/tabulate_invalidations branch from 6bfd41d to a2f6e12 Compare January 10, 2023 18:38

charleskawczynski changed the title ~~Add tabulated_invalidations to tabulate inv summary~~ Add report_invalidations to tabulate invalidations summary Jan 10, 2023

charleskawczynski changed the title ~~Add report_invalidations to tabulate invalidations summary~~ Add report_invalidations to tabulate invalidations summary Jan 10, 2023

timholy reviewed Jan 11, 2023

View reviewed changes

charleskawczynski and others added 2 commits January 12, 2023 09:13

Update src/invalidations.jl

a116dd8

Co-authored-by: Tim Holy <[email protected]>

Fix edge cases, improve docs, apply suggestions

5b4a42f

timholy merged commit 64a9a8e into timholy:master Jan 13, 2023

t-bltg mentioned this pull request Jan 13, 2023

enhance invalidations docs - increase cov #337

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `report_invalidations` to tabulate invalidations summary #303

Add `report_invalidations` to tabulate invalidations summary #303

charleskawczynski commented Sep 30, 2022 •

edited

Loading

codecov bot commented Sep 30, 2022 •

edited

Loading

charleskawczynski commented Oct 26, 2022

charleskawczynski commented Nov 23, 2022

timholy commented Dec 1, 2022

charleskawczynski commented Jan 1, 2023 •

edited

Loading

charleskawczynski commented Jan 10, 2023

timholy left a comment

timholy Jan 11, 2023

charleskawczynski Jan 12, 2023

timholy Jan 11, 2023

charleskawczynski Jan 12, 2023

charleskawczynski commented Jan 12, 2023

timholy commented Jan 13, 2023


		Print a tabular summary of invalidations given:

		- `job_name::String` the name of the job

Add report_invalidations to tabulate invalidations summary #303

Add report_invalidations to tabulate invalidations summary #303

Conversation

charleskawczynski commented Sep 30, 2022 • edited Loading

codecov bot commented Sep 30, 2022 • edited Loading

Codecov Report

charleskawczynski commented Oct 26, 2022

charleskawczynski commented Nov 23, 2022

timholy commented Dec 1, 2022

charleskawczynski commented Jan 1, 2023 • edited Loading

charleskawczynski commented Jan 10, 2023

timholy left a comment

Choose a reason for hiding this comment

timholy Jan 11, 2023

Choose a reason for hiding this comment

charleskawczynski Jan 12, 2023

Choose a reason for hiding this comment

timholy Jan 11, 2023

Choose a reason for hiding this comment

charleskawczynski Jan 12, 2023

Choose a reason for hiding this comment

charleskawczynski commented Jan 12, 2023

timholy commented Jan 13, 2023

Add `report_invalidations` to tabulate invalidations summary #303

Add `report_invalidations` to tabulate invalidations summary #303

charleskawczynski commented Sep 30, 2022 •

edited

Loading

codecov bot commented Sep 30, 2022 •

edited

Loading

charleskawczynski commented Jan 1, 2023 •

edited

Loading