Defining new Lexer result in LoadError due to evaluation in module Lexers

[chengchingwen]

Hi, I'm trying to define a Lexer in a local module and during the precompilation I get LoadError:

ERROR: LoadError: LoadError: Evaluation into the closed module `Lexers` breaks incremental compilation because the side effects will not be permanent. This is likely due to some other module mutating `Lexers` with `eval` during precompilation -
 don't do this.

[MichaelHatherly]

Could you provide some additional details for this? versioninfo(), as well as a self-contained example (if possible), a stacktrace (or is that the full extent of the error.

(I've not touched the codebase in a very long time... it's likely to be something to do with

Highlights.jl/src/Compiler.jl

Line 119 in 6db0259

Core.eval(mod, func)

.)

[MichaelHatherly]

@chengchingwen could you see whether the following patch fixes things for you?

diff --git a/src/Lexers.jl b/src/Lexers.jl
index a808ff4..5dfbc63 100644
--- a/src/Lexers.jl
+++ b/src/Lexers.jl
@@ -48,7 +48,7 @@ macro lexer(T, dict)
         #     @generated $(Compiler).lex!{S}(ctx::Context, ::Type{$tx}, ::State{S}) =
         #         compile($tx, S, data)
         # end
-        $(Compiler).compile_lexer($(@__MODULE__), $tx)
+        $(Compiler).compile_lexer($(__module__), $tx)
         $tx
     end
 end

[chengchingwen]

I'm using Julia v1.6

julia> versioninfo()
Julia Version 1.6.1-pre.1
Commit 9283b6c7a7 (2021-04-01 07:14 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-7800X CPU @ 3.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake-avx512)
Environment:
  JULIA_NUM_THREADS = 8

The error can be reproduce by simply pasting the CommentLexer from the document in a new generated package.

And here is the full error log:

julia> using Testhi                                                                                                       
[ Info: Precompiling Testhi [4875d125-e8d5-46ab-9eb5-d1b166ab85ea]                                                        
ERROR: LoadError: UndefVarError: CommentLexer not defined                                                                 
Stacktrace:                                                                                                               
 [1] top-level scope                                                                                                      
   @ ~/.julia/packages/Highlights/NdBSa/src/Lexers.jl:45                                                                  
 [2] include                                                                                                              
   @ ./Base.jl:386 [inlined]                                                                                              
 [3] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, 
load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)                              
   @ Base ./loading.jl:1213                                                                                               
 [4] top-level scope                                                                                                      
   @ none:1                                                                                                               
 [5] eval                                                                                                                 
   @ ./boot.jl:360 [inlined]                                                                                              
 [6] eval(x::Expr)                                                                                                        
   @ Base.MainInclude ./client.jl:446                                                                                     
 [7] top-level scope                                                                                                      
   @ none:1                                                                                                               
in expression starting at /home/peter/peter/repo/Serpent/Testhi/src/Testhi.jl:1                                           
ERROR: Failed to precompile Testhi [4875d125-e8d5-46ab-9eb5-d1b166ab85ea] to /home/peter/.julia/compiled/v1.6/Testhi/jl_LD
WE0r.                                                                                                                     
Stacktrace:                                                                                                               
 [1] error(s::String)                                                                                                     
   @ Base ./error.jl:33                                                                                                   
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::Base.TTY, internal_stdout::Base.TTY)                    
   @ Base ./loading.jl:1360                                                                                               
 [3] compilecache(pkg::Base.PkgId, path::String)  
   @ Base ./loading.jl:1306
 [4] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1021
 [5] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:914
 [6] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:901

and the file:

module Testhi

using Highlights
using Highlights.Lexers
using Highlights.Tokens

abstract type CommentLexer <: AbstractLexer end

@lexer CommentLexer Dict(
    :name => "Comments",
    :description => "A C-style comment lexer.",
    :tokens => Dict(
        :root => [
            (r"//.*\n", COMMENT_SINGLE),
            (r"/\*",    COMMENT_MULTILINE,   :multiline_comments),
            (r"[^/]+",  TEXT)
        ],
        :multiline_comments => [
            (r"/\*",     COMMENT_MULTILINE,  :__push__),
            (r"\*/",     COMMENT_MULTILINE,  :__pop__),
            (r"[^/\*]+", COMMENT_MULTILINE),
        ],
    )
)

greet() = print("Hello World!")

end # module

[chengchingwen]

could you see whether the following patch fixes things for you?

@MichaelHatherly Yes, this patch seems to fix the things.

[MichaelHatherly]

Yup, fixes it on my side too.

[MichaelHatherly]

It'll be out in JuliaRegistries/General#35136 shortly @chengchingwen.

[chengchingwen]

Thanks!