Add static_methods function #8

MasonProtter · 2020-01-17T22:46:00Z

This is the result of some explorations I took in finding all the methods of a function at compile time, with back-edges built in.

julia> using Tricks

julia> f(x) = x + 1
f (generic function with 1 method)

julia> static_methods(f)
# 1 method for generic function "f":
[1] f(x) in Main at REPL[2]:1

julia> @btime static_methods(f);
  43.324 ns (0 allocations: 0 bytes)

julia> @btime methods(f);
  2.250 μs (12 allocations: 720 bytes)

As you can see it happens at compile time, but also gets recompiled when new methods are added:

julia> f(::Int) = 1
f (generic function with 2 methods)

julia> static_methods(f)
# 2 methods for generic function "f":
[1] f(::Int64) in Main at REPL[6]:1
[2] f(x) in Main at REPL[2]:1

julia> f(x, y) = x / y
f (generic function with 3 methods)

julia> static_methods(f, Tuple{Int, Float64})
# 1 method for generic function "f":
[1] f(x, y) in Main at REPL[9]:1

I messed with the indentation to reflect regular module organization, but if you don't like that I can revert it.

src/Tricks.jl

test/runtests.jl

oxinabox · 2020-01-17T23:29:35Z

What if rather than dooing all this stuff to create a CodeInfo for a function.
we just made a function that does what we want:

julia> @generated function _neomethods(f)
       methods(f.instance)
       end
_neomethods (generic function with 1 method)

julia> _neomethods(identity)
# 1 method for generic function "identity":
[1] identity(x) in Base at operators.jl:502

julia> @code_typed _neomethods(identity)
CodeInfo(
1 ─     return # 1 method for generic function "identity":
[1] identity(x) in Base at operators.jl:502
) => Base.MethodList

julia> Base.identity(::Float64) = 3

julia> _neomethods(identity)
# 1 method for generic function "identity":
[1] identity(x) in Base at operators.jl:502

So that, as you see does indeed compute the methods at compile time, but it is missing the hooks.

But then we do the same thing we do in static_hasmethod, and copy its code info
(we make it have the same signature as what ever outer method has)
and then attached hooks.

Probably extract out some common functionality between the two into helpers.

src/Tricks.jl

NHDaly · 2020-01-18T14:30:01Z

So what happened with your investigation into whether the optimizer is already doing this?

oxinabox · 2020-01-18T15:39:11Z

So what happened with your investigation into whether the optimizer is already doing this?

The optimizer won't do this, it might do the thing that was Mason's original use case for this though

MasonProtter · 2020-01-18T21:54:18Z

I'm not sure I understand exactly what you mean here

But then we do the same thing we do in static_hasmethod, and copy its code info
(we make it have the same signature as what ever outer method has)
and then attached hooks.

or how to achieve it.

oxinabox · 2020-01-19T04:40:20Z

Like in my example we define at the start :

@generated _neomethods(f, t) = methods(f,t)

Then in our static_methods we copy it's codeinfo, and add edges to it before returning it.

This we don't need to do the fiddly expr to codeinfo thing

NHDaly · 2020-01-19T19:26:20Z

Yes, exactly!

Then your whole function would basically just be like:

@generated function static_methods(@nospecialize(m::Module), @nospecialize(f) , @nospecialize(_T::Type{T})) where {T <: Tuple}
        # Re-use the body of _neomethods:
        ci_orig = uncompressed_ast(typeof(_neomethods).name.mt.defs.func)
        ci = ccall(:jl_copy_code_info, Ref{CodeInfo}, (Any,), ci_orig)

        # And just add the edges so if a method is added to `f`, this recompiles
        mt = f.name.mt
        ci.edges = Core.Compiler.vect(mt, Tuple{Vararg{Any}})
        return ci
end

Or something else simple, more or less like that. :)

I'm actually not even sure if _neomethods needs to be @generated like you have it, @oxinabox

MasonProtter · 2020-01-19T19:50:29Z

Unfortunately, if _neomethods is not @generated, then static_methods will run at runtime, not compiletime.

If _neomethods is @generated, then uncompressed_ast doesn't work and julia suggests to use code_lowered instead, but when I switch to code_lowered(_neomethods, Tuple{f, T}) I get

julia> begin
       using Tricks
       f(::Int) = 1
       static_methods(f, Tuple{Any})
       end
ERROR: Could not expand generator for `@generated` method MethodInstance for _neomethods(::typeof(f), ::Tuple{Any}). This can happen if the provided argument types (Tuple{typeof(f),Tuple{Any}}) are not leaf types, but the `generated` argument is `true`.
Stacktrace:
 [1] error(::String, ::Core.MethodInstance, ::String, ::String, ::Type, ::String, ::String) at ./error.jl:42
 [2] (::Base.var"#12#13"{Bool,DataType})(::Core.MethodInstance) at ./reflection.jl:806
 [3] _collect(::Array{Core.MethodInstance,1}, ::Base.Generator{Array{Core.MethodInstance,1},Base.var"#12#13"{Bool,DataType}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./generator.jl:47
 [4] collect_similar(::Array{Core.MethodInstance,1}, ::Base.Generator{Array{Core.MethodInstance,1},Base.var"#12#13"{Bool,DataType}}) at ./array.jl:564
 [5] map(::Function, ::Array{Core.MethodInstance,1}) at ./abstractarray.jl:2073
 [6] #code_lowered#11 at ./reflection.jl:801 [inlined]
 [7] code_lowered at ./reflection.jl:793 [inlined]
 [8] #s15#7 at /Users/mason/.julia/dev/Tricks/src/Tricks.jl:69 [inlined]
 [9] #s15#7(::Any, ::Any, ::Any, ::Any) at none:0
 [10] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at ./boot.jl:524
 [11] top-level scope at REPL[1]:4

where the relevant code from Tricks is

@generated _neomethods(@nospecialize(f), @nospecialize(t)) = methods(f, t)

@generated function static_methods(@nospecialize(f) , @nospecialize(_T::Type{T})) where {T <: Tuple}
    # Re-use the body of _neomethods:
    ci_orig = code_lowered(_neomethods, Tuple{f, T})
    ci = ccall(:jl_copy_code_info, Ref{CodeInfo}, (Any,), ci_orig)

    # And just add the edges so if a method is added to `f`, this recompiles
    mt = f.name.mt
    ci.edges = Core.Compiler.vect(mt, Tuple{Vararg{Any}})
    return ci
end

NHDaly · 2020-01-19T20:05:40Z

Unfortunately, if _neomethods is not @generated, then static_methods will

run at runtime, not compiletime. Is that right? Even though `static_methods` itself is a @generated function?

…

On Sun, Jan 19, 2020 at 2:50 PM Mason Protter ***@***.***> wrote: Unfortunately, if _neomethods is not @generated, then static_methods will run at runtime, not compiletime. If _neomethods is @generated, then uncompressed_ast doesn't work and julia suggests to use code_lowered instead, but when I switch to code_lowered(_neomethods, Tuple{f, T}) I get julia> begin using Tricks f(::Int) = 1 static_methods(f, Tuple{Any}) end ERROR: Could not expand generator for ***@***.***` method MethodInstance for _neomethods(::typeof(f), ::Tuple{Any}). This can happen if the provided argument types (Tuple{typeof(f),Tuple{Any}}) are not leaf types, but the `generated` argument is `true`. Stacktrace: [1] error(::String, ::Core.MethodInstance, ::String, ::String, ::Type, ::String, ::String) at ./error.jl:42 [2] (::Base.var"#12#13"{Bool,DataType})(::Core.MethodInstance) at ./reflection.jl:806 [3] _collect(::Array{Core.MethodInstance,1}, ::Base.Generator{Array{Core.MethodInstance,1},Base.var"#12#13"{Bool,DataType}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./generator.jl:47 [4] collect_similar(::Array{Core.MethodInstance,1}, ::Base.Generator{Array{Core.MethodInstance,1},Base.var"#12#13"{Bool,DataType}}) at ./array.jl:564 [5] map(::Function, ::Array{Core.MethodInstance,1}) at ./abstractarray.jl:2073 [6] #code_lowered#11 at ./reflection.jl:801 [inlined] [7] code_lowered at ./reflection.jl:793 [inlined] [8] #s15#7 at /Users/mason/.julia/dev/Tricks/src/Tricks.jl:69 [inlined] [9] #s15#7(::Any, ::Any, ::Any, ::Any) at none:0 [10] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at ./boot.jl:524 [11] top-level scope at REPL[1]:4 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8?email_source=notifications&email_token=AAMCIELIYXSXUC3OTOMGSM3Q6SVINA5CNFSM4KIPDOJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJK2XSI#issuecomment-576039881>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMCIEMPXMXTPXNC5LSM5F3Q6SVINANCNFSM4KIPDOJQ> .

MasonProtter · 2020-01-19T20:37:24Z

That’s what I saw testing it myself. It’s possible I did something incorrect though.

oxinabox · 2020-01-19T21:00:09Z

hmm, I will have a play with this for a big see if i can make it work.

oxinabox · 2020-01-22T09:59:41Z

I couldn't work it out either.
Lets make the change to get rid of the ::Module argument,
and update the readme,
and then we can merge it.

MasonProtter · 2020-01-22T15:53:56Z

Okay, will do.

MasonProtter · 2020-01-22T19:30:56Z

Okay, so I'd updated the docstring, switched to @__MODULE__ in expr_to_code_info and added a test case where I make sure we're capturing the method-table for functions living in other modules.

Tests pass locally.

Maybe we should get Travis set up for this package?

src/Tricks.jl

oxinabox · 2020-01-22T19:57:44Z

I have added Travis. If you rebase off master is should run.

Can you also bump the version in the Project.toml so i can tag a release?
See:
https://white.ucc.asn.au/2019/09/28/Continuous-Delivery-For-Julia-Packages.html#what-if-i-dont-want-to-release-right-now--dev-versions

NHDaly

Unfortunately, if _neomethods is not @generated, then static_methods will run at runtime, not compiletime.

Is that right? Even though static_methods itself is a @generated function?

Oh, wait, right, duh, of course that's right. Because now the method you're generating has as its body, a runtime call to methods(), rather than having as its body the result of a call to methods from within the generated function. ✔️

I think the code as you have it is the best way to do what you want: you call methods() from within the body of the generated function, then you create a new CodeInfo that does nothing but return that result, and attach edges to it! ✔️

src/Tricks.jl

Co-Authored-By: Lyndon White <[email protected]>

…ks.jl into static_methods

MasonProtter · 2020-01-22T21:36:11Z

I think the CI failures are unrelated.

oxinabox · 2020-01-23T10:00:24Z

urg, those test failures are unrelated yes.

I need to work out a better way to test these thins

oxinabox · 2020-01-23T10:15:17Z

@MasonProtter can you take a look at the commit I just mad and see that i didn't break anything?

I also note that since sparams is never nothing
we could simplify to

    expr = Expr(Symbol("with-static-parameters"),
        Expr(:lambda,
            argnames,
            Expr(Symbol("scope-block"),
                Expr(:block,
                    Expr(:return, value),
                )
            )
        )
        spnames...
    )

But I think that might make the code less clear

MasonProtter · 2020-01-23T17:39:03Z

Your commit looks good to me and the tests pass locally.

MasonProtter added 2 commits January 17, 2020 15:37

change indentation of file

f7bb806

add static_methods

2a05b29

oxinabox reviewed Jan 17, 2020

View reviewed changes

src/Tricks.jl Outdated Show resolved Hide resolved

oxinabox reviewed Jan 17, 2020

View reviewed changes

src/Tricks.jl Outdated Show resolved Hide resolved

oxinabox reviewed Jan 17, 2020

View reviewed changes

test/runtests.jl Show resolved Hide resolved

MasonProtter added 2 commits January 17, 2020 16:46

update tests and fix indentation

0cf2f80

Always attach backedges to method table

992659a

oxinabox reviewed Jan 18, 2020

View reviewed changes

src/Tricks.jl Outdated Show resolved Hide resolved

MasonProtter added 2 commits January 22, 2020 12:26

fixes

883cb77

remove some outdated comments

ff014c2

oxinabox reviewed Jan 22, 2020

View reviewed changes

src/Tricks.jl Outdated Show resolved Hide resolved

NHDaly approved these changes Jan 22, 2020

View reviewed changes

src/Tricks.jl Show resolved Hide resolved

MasonProtter and others added 4 commits January 22, 2020 14:26

Update src/Tricks.jl

863865d

Co-Authored-By: Lyndon White <[email protected]>

Merge https://github.com/oxinabox/Tricks.jl into static_methods

1dbaf49

rename expr_to_codeinfo and trim

46cd853

Merge branch 'static_methods' of https://github.com/MasonProtter/Tric…

3d70fe2

…ks.jl into static_methods

some remove unused calls and adjust style

608dcd5

oxinabox merged commit dd68bf2 into oxinabox:master Jan 23, 2020

oxinabox mentioned this pull request Jan 25, 2020

make tests robust #10

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add static_methods function #8

Add static_methods function #8

MasonProtter commented Jan 17, 2020 •

edited

Loading

oxinabox commented Jan 17, 2020

NHDaly commented Jan 18, 2020

oxinabox commented Jan 18, 2020

MasonProtter commented Jan 18, 2020

oxinabox commented Jan 19, 2020

NHDaly commented Jan 19, 2020

MasonProtter commented Jan 19, 2020 •

edited

Loading

NHDaly commented Jan 19, 2020 via email

MasonProtter commented Jan 19, 2020

oxinabox commented Jan 19, 2020

oxinabox commented Jan 22, 2020

MasonProtter commented Jan 22, 2020

MasonProtter commented Jan 22, 2020

oxinabox commented Jan 22, 2020

NHDaly left a comment

MasonProtter commented Jan 22, 2020

oxinabox commented Jan 23, 2020

oxinabox commented Jan 23, 2020

MasonProtter commented Jan 23, 2020

Add static_methods function #8

Add static_methods function #8

Conversation

MasonProtter commented Jan 17, 2020 • edited Loading

oxinabox commented Jan 17, 2020

NHDaly commented Jan 18, 2020

oxinabox commented Jan 18, 2020

MasonProtter commented Jan 18, 2020

oxinabox commented Jan 19, 2020

NHDaly commented Jan 19, 2020

MasonProtter commented Jan 19, 2020 • edited Loading

NHDaly commented Jan 19, 2020 via email

MasonProtter commented Jan 19, 2020

oxinabox commented Jan 19, 2020

oxinabox commented Jan 22, 2020

MasonProtter commented Jan 22, 2020

MasonProtter commented Jan 22, 2020

oxinabox commented Jan 22, 2020

NHDaly left a comment

Choose a reason for hiding this comment

MasonProtter commented Jan 22, 2020

oxinabox commented Jan 23, 2020

oxinabox commented Jan 23, 2020

MasonProtter commented Jan 23, 2020

MasonProtter commented Jan 17, 2020 •

edited

Loading

MasonProtter commented Jan 19, 2020 •

edited

Loading