Add options for varinfo() to show all and imported, sort by each column, search modules recursively #38042

IanButterworth · 2020-10-15T15:36:01Z

Expands the functionality of varinfo() with kwargs:

all : also list non-exported objects defined in the module, deprecated objects, and compiler-generated objects.
imported : also list objects explicitly imported from other modules.
sortby : the column to sort results by. Options are :name (default), :size, and :summary.

Edit:

recursive : recursively include objects in sub-modules, observing the same settings in each.
See Add options for varinfo() to show all and imported, sort by each column, search modules recursively #38042 (comment)

The first two arcs match the kwargs from names

julia> module Foo
       import Logging: @info
       const foo_const = rand(1000000)
       foo_var =         rand(1000000)
       foo_func() =      rand(1000000)
       end
Main.Foo

julia> varinfo(Foo)
  name       size summary
  –––– –––––––––– –––––––
  Foo  15.262 MiB Module 

julia> varinfo(Foo, all=true)
  name            size summary                        
  ––––––––– –––––––––– –––––––––––––––––––––––––––––––
  #eval      172 bytes DataType                       
  #foo_func  172 bytes DataType                       
  #include   172 bytes DataType                       
  Foo       15.262 MiB Module                         
  eval         0 bytes typeof(Main.Foo.eval)          
  foo_const  7.629 MiB 1000000-element Vector{Float64}
  foo_func     0 bytes typeof(Main.Foo.foo_func)      
  foo_var    7.629 MiB 1000000-element Vector{Float64}
  include      0 bytes typeof(Main.Foo.include)   

julia> varinfo(Foo, imported=true)
  name        size summary                     
  ––––– –––––––––– ––––––––––––––––––––––––––––
  @info    0 bytes Base.CoreLogging.var"#@info"
  Foo   15.262 MiB Module      

julia> varinfo(Foo, all=true, sortby=:size)
  name            size summary                        
  ––––––––– –––––––––– –––––––––––––––––––––––––––––––
  Foo       15.262 MiB Module                         
  foo_const  7.629 MiB 1000000-element Vector{Float64}
  foo_var    7.629 MiB 1000000-element Vector{Float64}
  #eval      172 bytes DataType                       
  #foo_func  172 bytes DataType                       
  #include   172 bytes DataType                       
  eval         0 bytes typeof(Main.Foo.eval)          
  foo_func     0 bytes typeof(Main.Foo.foo_func)      
  include      0 bytes typeof(Main.Foo.include)    

julia> varinfo(Foo, all=true, sortby=:summary)
  name            size summary                        
  ––––––––– –––––––––– –––––––––––––––––––––––––––––––
  foo_const  7.629 MiB 1000000-element Vector{Float64}
  foo_var    7.629 MiB 1000000-element Vector{Float64}
  #eval      172 bytes DataType                       
  #foo_func  172 bytes DataType                       
  #include   172 bytes DataType                       
  Foo       15.262 MiB Module                         
  eval         0 bytes typeof(Main.Foo.eval)          
  foo_func     0 bytes typeof(Main.Foo.foo_func)      
  include      0 bytes typeof(Main.Foo.include)

Octogonapus · 2020-10-15T15:38:55Z

This was a great help to me 👍

IanButterworth · 2020-10-16T23:41:04Z

Not sure who's best to review.

@JeffBezanson? (just on git last history, though it was just a reorg)

IanButterworth · 2020-10-18T02:56:28Z

It'd be a shame if this didn't make it into the 1.6 feature freeze (which I believe is imminent)

I don't know who covers this kind of thing, so some wild tags.. @tkf @KristofferC

vchuravy

Looks fine to me.

stdlib/InteractiveUtils/src/InteractiveUtils.jl

fredrikekre · 2020-10-19T22:00:52Z

sort_size is a bit strange maybe. How about ; sortby::Symbol=:name (default now I guess?) and you can use sortby=:size?

IanButterworth · 2020-10-19T22:10:11Z

That sounds good. Might as well allow :summary too

rfourquet · 2020-10-20T08:12:04Z

sort_size is a bit strange maybe.

I had the same feel and also thought about something like sortby, but it looked like whenever this keyword would be used, it would be sortby=:size so I was not sure. Also, maybe use sortedby to be in line with Profile.print? (although I prefer sortby for brevity...)

IanButterworth · 2020-10-20T12:01:13Z

I feel like sortby is right.

Also, it can be useful to sort by the summary, to collect object types together.

IanButterworth · 2020-10-22T06:56:06Z

I've added functionality to recursively include objects from sub-modules too

julia> varinfo(JSON, all = true, recursive = true, sortby=:size)
  name                                        size summary                                                            
  –––––––––––––––––––––––––––––––––––– ––––––––––– –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
  JSON                                 295.224 KiB Module                                                             
  Parser                               200.799 KiB Module                                                             
  Writer                               184.403 KiB Module                                                             
  Serializations                       128.265 KiB Module                                                             
  Writer.##meta#51                      24.998 KiB IdDict{Any, Any} with 18 entries                                   
  Parser.##meta#51                      22.226 KiB IdDict{Any, Any} with 17 entries                                   
  Common                                17.806 KiB Module                                                             
  Common.ESCAPED_ARRAY                  12.433 KiB 256-element Vector{Vector{UInt8}}                                  
  Serializations.##meta#51               5.661 KiB IdDict{Any, Any} with 4 entries                                    
  Common.##meta#51                       1.308 KiB IdDict{Any, Any} with 1 entry                                      
  Parser.#4#5                            240 bytes UnionAll                                                           
  Common.ESCAPES                         232 bytes Dict{UInt8, UInt8} with 8 entries                                  
  Common.REVERSE_ESCAPES                 232 bytes Dict{UInt8, UInt8} with 8 entries                                  
  Parser.MemoryParserState               196 bytes DataType                                                           
  JSONText                               188 bytes DataType                                                           
  Writer.JSONText                        188 bytes DataType                                                           
  #eval                                  172 bytes DataType                                                           
  #include                               172 bytes DataType                                                           
  Common.#eval                           172 bytes DataType                                                           
  Common.#include                        172 bytes DataType                                                           
  Parser.##parse#1                       172 bytes DataType                                                           
  Parser.##parse#2                       172 bytes DataType                                                           
  Parser.##parsefile#3                   172 bytes DataType                                                           
  Parser.#_count_before                  172 bytes DataType                                                           
  Parser.#_error                         172 bytes DataType                                                           
  Parser.#_error_expected_char           172 bytes DataType                                                           
  Parser.#_get_parsercontext             172 bytes DataType                                                           
  Parser.#advance!                       172 bytes DataType                                                           
  Parser.#byteat                         172 bytes DataType                                                           
  Parser.#chomp_space!                   172 bytes DataType                                                           
  Parser.#current                        172 bytes DataType                                                           
  Parser.#eval                           172 bytes DataType                                                           
  Parser.#float_from_bytes               172 bytes DataType                                                           
  Parser.#hasleadingzero                 172 bytes DataType                                                           
  Parser.#hasmore                        172 bytes DataType                                                           
  Parser.#include                        172 bytes DataType                                                           
  Parser.#incr!                          172 bytes DataType                                                           
  Parser.#int_from_bytes                 172 bytes DataType                                                           
  Parser.#isjsondigit                    172 bytes DataType                                                           
  Parser.#isjsonspace                    172 bytes DataType                                                           
  Parser.#maxsize_buffer                 172 bytes DataType                                                           
  Parser.#number_from_bytes              172 bytes DataType                                                           
  Parser.#parse                          172 bytes DataType                                                           
  Parser.#parse##kw                      172 bytes DataType                                                           
  Parser.#parse_array                    172 bytes DataType                                                           
  Parser.#parse_jsconstant               172 bytes DataType                                                           
  Parser.#parse_number                   172 bytes DataType                                                           
  Parser.#parse_object                   172 bytes DataType                                                           
  Parser.#parse_string                   172 bytes DataType                                                           
  Parser.#parse_value                    172 bytes DataType                                                           
  Parser.#parsefile                      172 bytes DataType                                                           
  Parser.#parsefile##kw                  172 bytes DataType                                                           
  Parser.#predict_string                 172 bytes DataType                                                           
  Parser.#read_four_hex_digits!          172 bytes DataType                                                           
  Parser.#read_unicode_escape!           172 bytes DataType                                                           
  Parser.#skip!                          172 bytes DataType                                                           
  Parser.#unparameterize_type            172 bytes DataType                                                           
  Parser.#utf16_get_supplementary        172 bytes DataType                                                           
  Parser.#utf16_is_surrogate             172 bytes DataType                                                           
  Parser.ParserState                     172 bytes DataType                                                           
  Serializations.#eval                   172 bytes DataType                                                           
  Serializations.#include                172 bytes DataType                                                           
  Serializations.CommonSerialization     172 bytes DataType                                                           
  Serializations.Serialization           172 bytes DataType                                                           
  Serializations.StandardSerialization   172 bytes DataType                                                           
  Writer.##show_json#3                   172 bytes DataType                                                           
  Writer.#1#2                            172 bytes DataType                                                           
  Writer.#begin_array                    172 bytes DataType                                                           
  Writer.#begin_object                   172 bytes DataType                                                           
  Writer.#delimit                        172 bytes DataType                                                           
  Writer.#end_array                      172 bytes DataType                                                           
  Writer.#end_object                     172 bytes DataType                                                           
  Writer.#eval                           172 bytes DataType                                                           
  Writer.#include                        172 bytes DataType                                                           
  Writer.#indent                         172 bytes DataType                                                           
  Writer.#json                           172 bytes DataType                                                           
  Writer.#lower                          172 bytes DataType                                                           
  Writer.#print                          172 bytes DataType                                                           
  Writer.#separate                       172 bytes DataType                                                           
  Writer.#show_element                   172 bytes DataType                                                           
  Writer.#show_json                      172 bytes DataType                                                           
  Writer.#show_json##kw                  172 bytes DataType                                                           
  Writer.#show_key                       172 bytes DataType                                                           
  Writer.#show_null                      172 bytes DataType                                                           
  Writer.#show_pair                      172 bytes DataType                                                           
  Writer.#show_string                    172 bytes DataType                                                           
  Writer.CS                              172 bytes DataType                                                           
  Writer.JSONContext                     172 bytes DataType                                                           
  Writer.SC                              172 bytes DataType                                                           
  Writer.StructuralContext               172 bytes DataType                                                           
  Parser.ParserContext                   160 bytes UnionAll                                                           
  Writer.IsPrintedAsString               160 bytes Union                                                              
  Common.E_BAD_CONTROL                    41 bytes 33-codeunit String                                                 
  Parser.StreamingParserState             40 bytes UnionAll                                                           
  Writer.CompactContext                   40 bytes UnionAll                                                           
  Writer.CompositeTypeWrapper             40 bytes UnionAll                                                           
  Writer.PrettyContext                    40 bytes UnionAll                                                           
  Writer.StringContext                    40 bytes UnionAll                                                           
  Common.E_LEADING_ZERO                   38 bytes 30-codeunit String                                                 
  Common.E_BAD_ESCAPE                     31 bytes 23-codeunit String                                                 
  Common.E_UNEXPECTED_EOF                 31 bytes 23-codeunit String                                                 
  Common.E_EXPECTED_EOF                   29 bytes 21-codeunit String                                                 
  Common.E_UNEXPECTED_CHAR                28 bytes 20-codeunit String                                                 
  Common.E_BAD_KEY                        26 bytes 18-codeunit String                                                 
  Common.E_BAD_NUMBER                     22 bytes 14-codeunit String                                                 
  Common.ARRAY_BEGIN                        1 byte UInt8                                                              
  Common.ARRAY_END                          1 byte UInt8                                                              
  Common.BACKSLASH                          1 byte UInt8                                                              
  Common.BACKSPACE                          1 byte UInt8                                                              
  Common.DECIMAL_POINT                      1 byte UInt8                                                              
  Common.DELIMITER                          1 byte UInt8                                                              
  Common.DIGIT_NINE                         1 byte UInt8                                                              
  Common.DIGIT_ZERO                         1 byte UInt8                                                              
  Common.FORM_FEED                          1 byte UInt8                                                              
  Common.LATIN_A                            1 byte UInt8                                                              
  Common.LATIN_B                            1 byte UInt8                                                              
  Common.LATIN_E                            1 byte UInt8                                                              
  Common.LATIN_F                            1 byte UInt8                                                              
  Common.LATIN_I                            1 byte UInt8                                                              
  Common.LATIN_L                            1 byte UInt8                                                              
  Common.LATIN_N                            1 byte UInt8                                                              
  Common.LATIN_R                            1 byte UInt8                                                              
  Common.LATIN_S                            1 byte UInt8                                                              
  Common.LATIN_T                            1 byte UInt8                                                              
  Common.LATIN_U                            1 byte UInt8                                                              
  Common.LATIN_UPPER_A                      1 byte UInt8                                                              
  Common.LATIN_UPPER_E                      1 byte UInt8                                                              
  Common.LATIN_UPPER_F                      1 byte UInt8                                                              
  Common.LATIN_UPPER_I                      1 byte UInt8                                                              
  Common.LATIN_UPPER_N                      1 byte UInt8                                                              
  Common.LATIN_Y                            1 byte UInt8                                                              
  Common.MINUS_SIGN                         1 byte UInt8                                                              
  Common.NEWLINE                            1 byte UInt8                                                              
  Common.OBJECT_BEGIN                       1 byte UInt8                                                              
  Common.OBJECT_END                         1 byte UInt8                                                              
  Common.PLUS_SIGN                          1 byte UInt8                                                              
  Common.RETURN                             1 byte UInt8                                                              
  Common.SEPARATOR                          1 byte UInt8                                                              
  Common.SOLIDUS                            1 byte UInt8                                                              
  Common.SPACE                              1 byte UInt8                                                              
  Common.STRING_DELIM                       1 byte UInt8                                                              
  Common.TAB                                1 byte UInt8                                                              
  eval                                     0 bytes typeof(JSON.eval)                                                  
  include                                  0 bytes typeof(JSON.include)                                               
  json                                     0 bytes typeof(json)                                                       
  Common.eval                              0 bytes typeof(JSON.Common.eval)                                           
  Common.include                           0 bytes typeof(JSON.Common.include)                                        
  Parser.#parse#1                          0 bytes JSON.Parser.var"##parse#1"                                         
  Parser.#parse#2                          0 bytes JSON.Parser.var"##parse#2"                                         
  Parser.#parsefile#3                      0 bytes JSON.Parser.var"##parsefile#3"                                     
  Parser.DEFAULT_PARSERCONTEXT             0 bytes JSON.Parser.ParserContext{Dict{String, Any}, Int64, false, nothing}
  Parser._count_before                     0 bytes typeof(JSON.Parser._count_before)                                  
  Parser._error                            0 bytes typeof(JSON.Parser._error)                                         
  Parser._error_expected_char              0 bytes typeof(JSON.Parser._error_expected_char)                           
  Parser._get_parsercontext                0 bytes typeof(JSON.Parser._get_parsercontext)                             
  Parser.advance!                          0 bytes typeof(JSON.Parser.advance!)                                       
  Parser.byteat                            0 bytes typeof(JSON.Parser.byteat)                                         
  Parser.chomp_space!                      0 bytes typeof(JSON.Parser.chomp_space!)                                   
  Parser.current                           0 bytes typeof(JSON.Parser.current)                                        
  Parser.eval                              0 bytes typeof(JSON.Parser.eval)                                           
  Parser.float_from_bytes                  0 bytes typeof(JSON.Parser.float_from_bytes)                               
  Parser.hasleadingzero                    0 bytes typeof(JSON.Parser.hasleadingzero)                                 
  Parser.hasmore                           0 bytes typeof(JSON.Parser.hasmore)                                        
  Parser.include                           0 bytes typeof(JSON.Parser.include)                                        
  Parser.incr!                             0 bytes typeof(JSON.Parser.incr!)                                          
  Parser.int_from_bytes                    0 bytes typeof(JSON.Parser.int_from_bytes)                                 
  Parser.isjsondigit                       0 bytes typeof(JSON.Parser.isjsondigit)                                    
  Parser.isjsonspace                       0 bytes typeof(JSON.Parser.isjsonspace)                                    
  Parser.maxsize_buffer                    0 bytes typeof(JSON.Parser.maxsize_buffer)                                 
  Parser.number_from_bytes                 0 bytes typeof(JSON.Parser.number_from_bytes)                              
  Parser.parse                             0 bytes typeof(JSON.Parser.parse)                                          
  Parser.parse_array                       0 bytes typeof(JSON.Parser.parse_array)                                    
  Parser.parse_jsconstant                  0 bytes typeof(JSON.Parser.parse_jsconstant)                               
  Parser.parse_number                      0 bytes typeof(JSON.Parser.parse_number)                                   
  Parser.parse_object                      0 bytes typeof(JSON.Parser.parse_object)                                   
  Parser.parse_string                      0 bytes typeof(JSON.Parser.parse_string)                                   
  Parser.parse_value                       0 bytes typeof(JSON.Parser.parse_value)                                    
  Parser.parsefile                         0 bytes typeof(JSON.Parser.parsefile)                                      
  Parser.predict_string                    0 bytes typeof(JSON.Parser.predict_string)                                 
  Parser.read_four_hex_digits!             0 bytes typeof(JSON.Parser.read_four_hex_digits!)                          
  Parser.read_unicode_escape!              0 bytes typeof(JSON.Parser.read_unicode_escape!)                           
  Parser.skip!                             0 bytes typeof(JSON.Parser.skip!)                                          
  Parser.unparameterize_type               0 bytes typeof(JSON.Parser.unparameterize_type)                            
  Parser.utf16_get_supplementary           0 bytes typeof(JSON.Parser.utf16_get_supplementary)                        
  Parser.utf16_is_surrogate                0 bytes typeof(JSON.Parser.utf16_is_surrogate)                             
  Serializations.eval                      0 bytes typeof(JSON.Serializations.eval)                                   
  Serializations.include                   0 bytes typeof(JSON.Serializations.include)                                
  Writer.#show_json#3                      0 bytes JSON.Writer.var"##show_json#3"                                     
  Writer.begin_array                       0 bytes typeof(JSON.Writer.begin_array)                                    
  Writer.begin_object                      0 bytes typeof(JSON.Writer.begin_object)                                   
  Writer.delimit                           0 bytes typeof(JSON.Writer.delimit)                                        
  Writer.end_array                         0 bytes typeof(JSON.Writer.end_array)                                      
  Writer.end_object                        0 bytes typeof(JSON.Writer.end_object)                                     
  Writer.eval                              0 bytes typeof(JSON.Writer.eval)                                           
  Writer.include                           0 bytes typeof(JSON.Writer.include)                                        
  Writer.indent                            0 bytes typeof(JSON.Writer.indent)                                         
  Writer.json                              0 bytes typeof(json)                                                       
  Writer.lower                             0 bytes typeof(JSON.Writer.lower)                                          
  Writer.print                             0 bytes typeof(JSON.Writer.print)                                          
  Writer.separate                          0 bytes typeof(JSON.Writer.separate)                                       
  Writer.show_element                      0 bytes typeof(JSON.Writer.show_element)                                   
  Writer.show_json                         0 bytes typeof(JSON.Writer.show_json)                                      
  Writer.show_key                          0 bytes typeof(JSON.Writer.show_key)                                       
  Writer.show_null                         0 bytes typeof(JSON.Writer.show_null)                                      
  Writer.show_pair                         0 bytes typeof(JSON.Writer.show_pair)                                      
  Writer.show_string                       0 bytes typeof(JSON.Writer.show_string)

Which took

0.015613 seconds (54.33 k allocations: 2.737 MiB)

As a contrast, this the most varinfo can see currently

julia> varinfo(JSON)
  name            size summary     
  –––––––– ––––––––––– ––––––––––––
  JSON     295.247 KiB Module      
  JSONText   188 bytes DataType    
  json         0 bytes typeof(json)

An extreme test is:

julia> @time varinfo(DifferentialEquations, all = true, recursive = true, sortby=:size)
 92.196971 seconds (308.91 M allocations: 17.873 GiB, 3.97% gc time, 95.05% compilation time)

which works but the table formatting wraps around the terminal given the long package names and deps tree

IanButterworth · 2020-10-22T18:17:30Z

@vchuravy what do you think about the additional recursive mode?

Could this go into https://github.com/JuliaLang/julia/milestones/1.6%20features ?

IanButterworth · 2020-10-26T16:09:29Z

bump

IanButterworth · 2020-10-29T04:44:28Z

@timholy what do you think? Provides a fair bit more introspection into module memory usage

add options for varinfo to show all and imported, and sort by size

1024fd4

extend varinfo tests

d05721e

IanButterworth force-pushed the ib/varinfo_enhancements branch from 531754d to d05721e Compare October 16, 2020 22:02

vchuravy reviewed Oct 19, 2020

View reviewed changes

stdlib/InteractiveUtils/src/InteractiveUtils.jl Outdated Show resolved Hide resolved

vchuravy requested a review from JeffBezanson October 19, 2020 17:53

better sort

126ebc7

switch to sortby with colnames

003e88a

IanButterworth changed the title ~~Add options for varinfo() to show all and imported, and sort by size~~ Add options for varinfo() to show all and imported, and sort by each column Oct 20, 2020

add option to inspect objects from sub-modules recursively

8ec6ad2

IanButterworth requested a review from vchuravy October 22, 2020 06:59

IanButterworth changed the title ~~Add options for varinfo() to show all and imported, and sort by each column~~ Add options for varinfo() to show all and imported, sort by each column, search modules recursively Oct 22, 2020

KristofferC merged commit 5c47690 into JuliaLang:master Oct 29, 2020

IanButterworth deleted the ib/varinfo_enhancements branch October 29, 2020 11:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add options for varinfo() to show all and imported, sort by each column, search modules recursively #38042

Add options for varinfo() to show all and imported, sort by each column, search modules recursively #38042

IanButterworth commented Oct 15, 2020 •

edited

Loading

Octogonapus commented Oct 15, 2020

IanButterworth commented Oct 16, 2020

IanButterworth commented Oct 18, 2020

vchuravy left a comment

fredrikekre commented Oct 19, 2020

IanButterworth commented Oct 19, 2020

rfourquet commented Oct 20, 2020

IanButterworth commented Oct 20, 2020

IanButterworth commented Oct 22, 2020 •

edited

Loading

IanButterworth commented Oct 22, 2020 •

edited

Loading

IanButterworth commented Oct 26, 2020

IanButterworth commented Oct 29, 2020

Add options for varinfo() to show all and imported, sort by each column, search modules recursively #38042

Add options for varinfo() to show all and imported, sort by each column, search modules recursively #38042

Conversation

IanButterworth commented Oct 15, 2020 • edited Loading

Octogonapus commented Oct 15, 2020

IanButterworth commented Oct 16, 2020

IanButterworth commented Oct 18, 2020

vchuravy left a comment

Choose a reason for hiding this comment

fredrikekre commented Oct 19, 2020

IanButterworth commented Oct 19, 2020

rfourquet commented Oct 20, 2020

IanButterworth commented Oct 20, 2020

IanButterworth commented Oct 22, 2020 • edited Loading

IanButterworth commented Oct 22, 2020 • edited Loading

IanButterworth commented Oct 26, 2020

IanButterworth commented Oct 29, 2020

IanButterworth commented Oct 15, 2020 •

edited

Loading

IanButterworth commented Oct 22, 2020 •

edited

Loading

IanButterworth commented Oct 22, 2020 •

edited

Loading