Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segfault (during precompilation) with map(zip(a, b)) do (a, b) inside a macro #50538

Closed
thevolatilebit opened this issue Jul 13, 2023 · 4 comments · Fixed by #50541
Closed

Segfault (during precompilation) with map(zip(a, b)) do (a, b) inside a macro #50538

thevolatilebit opened this issue Jul 13, 2023 · 4 comments · Fixed by #50541
Labels
bug Indicates an unexpected problem or unintended behavior macros @macros

Comments

@thevolatilebit
Copy link

I have encountered segmentation faults in Julia 1.9 in the following specific test case: a module MWE2 makes a call to a macro that's defined in another module, MWE1, where the macro features a construct of the form map(zip(a, b)) do (a, b).

Interestingly, the same construct works as expected when wrapped as a function (defined in MWE1) or when the exact same macro, but defined in MWE2, is called. It also works fine in Julia 1.8; or if you disable precompilation in 1.9 using __precompile__(false).

Minimal Reproducible Example

module MWE1

macro map_test()
    quote
        map(zip(1:2, 1:2)) do (x1, x2)
            x1*x2
        end
    end
end

function map_test()
    map(zip(1:2, 1:2)) do (x1, x2)
        x1*x2
    end
end

end
module MWE2

using MWE1

x1 = MWE1.@map_test # seg fault
x2 = MWE1.map_test() # fine

macro map_test_internal()
    quote
        map(zip(1:2, 1:2)) do (x1, x2)
            x1*x2
        end
    end
end

x3 = @map_test_internal # fine

end

Version

It fails in Julia 1.9, but works in Julia 1.8.

Julia Version 1.9.2
Commit e4ee485e909 (2023-07-05 09:39 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 10 × Apple M1 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1)
  Threads: 1 on 8 virtual cores

Julia Version 1.8.5
Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.5.0)
  CPU: 10 × Apple M1 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 1 on 8 virtual cores

Modules Setup

For local setup of the module structure, you may use the following convenience script:

import Pkg

Pkg.generate("MWE1")
write("MWE1/src/MWE1.jl", 
    """
    module MWE1
    
    macro map_test()
        quote
            map(zip(1:2, 1:2)) do (x1, x2)
                x1*x2
            end
        end
    end

    function map_test()
        map(zip(1:2, 1:2)) do (x1, x2)
            x1*x2
        end
    end
    
    end""")

Pkg.generate("MWE2")

write("MWE2/src/MWE2.jl", 
    """
    module MWE2

    using MWE1

    x1 = MWE1.@map_test # seg fault
    x2 = MWE1.map_test() # fine

    macro map_test_internal()
        quote
            map(zip(1:2, 1:2)) do (x1, x2)
                x1*x2
            end
        end
    end

    x3 = @map_test_internal # fine

    end""")

Pkg.activate("MWE2")
Pkg.develop(path="MWE1")

using MWE2
@thevolatilebit thevolatilebit mentioned this issue Jul 13, 2023
Closed
@vtjnash
Copy link
Member

vtjnash commented Jul 13, 2023

I don't see a segfault, I just get a regular error. Could you post what you see so I can investigate that? Meanwhile, I will look into what triggered this error.

julia> using MWE2
Precompiling MWE2
  ✗ MWE2
  1 dependency successfully precompiled in 2 seconds

ERROR: The following 1 direct dependency failed to precompile:

MWE2 [771e6b25-5dee-434a-baf4-34b15c29549c]

Failed to precompile MWE2 [771e6b25-5dee-434a-baf4-34b15c29549c] to "/home/vtjnash/.julia/compiled/v1.11/MWE2/jl_aFVTbq".
fatal: error thrown and no exception handler available.
ErrorException("Binding cannot be serialized")
ijl_error at /home/vtjnash/julia1/src/rtutils.c:41
jl_write_values at /home/vtjnash/julia1/src/staticdata.c:1236 [inlined]
jl_save_system_image_to_stream at /home/vtjnash/julia1/src/staticdata.c:2518
ijl_create_system_image at /home/vtjnash/julia1/src/staticdata.c:2760
ijl_write_compiler_output at /home/vtjnash/julia1/src/precompile.c:121
ijl_atexit_hook at /home/vtjnash/julia1/src/init.c:251
jl_repl_entrypoint at /home/vtjnash/julia1/src/jlapi.c:732
main at /home/vtjnash/julia1/cli/loader_exe.c:58
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at /home/vtjnash/julia1/usr/bin/julia (unknown line)

@vtjnash
Copy link
Member

vtjnash commented Jul 13, 2023

To some extent, this is just your run-of-the-mill typical bug in macroexpand.scm hygiene pass not understanding that parameters can be grouped in argument lists so it converts those into globals:

julia> @macroexpand @map_test
quote
    #= REPL[11]:3 =#
    ((Main.x1, Main.x2),)->begin
            #= REPL[11]:3 =#
            return
        end
end

@JeffBezanson JeffBezanson added bug Indicates an unexpected problem or unintended behavior macros @macros labels Jul 13, 2023
vtjnash added a commit that referenced this issue Jul 13, 2023
@vtjnash
precompile: ensure globals are not accidentally created where disallowed
22502b8 
Usually this is caught by use of `eval`, but we should try to move away
from that broad rule to specific functions such as this one, such that
eventually we can remove that rule from `eval`.

Fix #50538
@vtjnash vtjnash mentioned this issue Jul 13, 2023
Merged
@thevolatilebit
Copy link
Author

Sure, thank you for investigating that error.

julia> using MWE2
[ Info: Precompiling MWE2 [707808d4-a326-492c-9359-fd411e7f8296]

[13044] signal (11.2): Segmentation fault: 11
in expression starting at REPL[8]:1
jl_restore_system_image_from_stream_ at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
jl_restore_package_image_from_stream at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
ijl_restore_package_image_from_file at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
_include_from_serialized at ./loading.jl:1052
_tryrequire_from_serialized at ./loading.jl:1442
unknown function (ip: 0x11aaf4993)
_require at ./loading.jl:1816
_require_prelocked at ./loading.jl:1660
macro expansion at ./loading.jl:1648 [inlined]
macro expansion at ./lock.jl:267 [inlined]
require at ./loading.jl:1611
jfptr_require_47948 at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
ijl_apply_generic at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
eval_import_path at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
jl_toplevel_eval_flex at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
jl_toplevel_eval_flex at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
jl_toplevel_eval_flex at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
jl_toplevel_eval_flex at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
ijl_toplevel_eval_in at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
eval at ./boot.jl:370 [inlined]
eval_user_input at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:153
repl_backend_loop at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:249
#start_repl_backend#46 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:234
start_repl_backend at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:231
ijl_apply_generic at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
#run_repl#59 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:379
run_repl at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:365
jfptr_run_repl_59922 at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
ijl_apply_generic at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
#1017 at ./client.jl:421
jfptr_YY.1017_40761 at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
ijl_apply_generic at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
jl_f__call_latest at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
#invokelatest#2 at ./essentials.jl:816 [inlined]
invokelatest at ./essentials.jl:813 [inlined]
run_main_repl at ./client.jl:405
exec_options at ./client.jl:322
_start at ./client.jl:522
jfptr__start_43625 at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
ijl_apply_generic at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
true_main at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
jl_repl_entrypoint at /Applications/Julia-1.9.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.9.dylib (unknown line)
Allocations: 3869086 (Pool: 3866378; Big: 2708); GC: 6
zsh: segmentation fault  julia1.9

@vtjnash
Copy link
Member

vtjnash commented Jul 13, 2023

Ah, right, we only added this error in v1.10, but it just crashes in v1.9

vtjnash added a commit that referenced this issue Jul 17, 2023
@vtjnash
precompile: ensure globals are not accidentally created where disallo…
3a9345c 
…wed (#50541)

Usually this is caught by use of `eval`, but we should try to move away
from that broad rule to specific functions such as this one, such that
eventually we can remove that rule from `eval`.

Fix #50538
KristofferC pushed a commit that referenced this issue Jul 18, 2023
@vtjnash @KristofferC
precompile: ensure globals are not accidentally created where disallo…
2c3f79b 
…wed (#50541)

Usually this is caught by use of `eval`, but we should try to move away
from that broad rule to specific functions such as this one, such that
eventually we can remove that rule from `eval`.

Fix #50538

(cherry picked from commit 3a9345c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Indicates an unexpected problem or unintended behavior macros @macros
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

precompile: ensure globals are not accidentally created where disallowed JuliaLang/julia
3 participants
@vtjnash @JeffBezanson @thevolatilebit