ArgumentError: input string is empty or only contains whitespace

[svilupp]

We've noticed the above error in some of our scheduled jobs using QuartoNotebookRunner.jl.
Error: ArgumentError: input string is empty or only contains whitespace

This seems similar to: JuliaPluto/Malt.jl#85 where it says

Turns out when a worker crashes before a port is written back, stdout is read back as "" so you get an error when parsing that as the port number.

We haven't figured out how to reliably reproduce it + we cannot debug it without insight into the Malt worker failure.
It failed several times but always at a different point (different notebooks, so it's not data problem).

Temporary solution: Remove outer asyncmap and run serially

Oddly, it was running fine for 7 days after upgrading to Julia 1.11 (and quarto 1.7.2, containing QuartoNotebookRunner 11.6?).

Reproduction:
Our setup is similar to the following (notebooks are in different folders not to clash in Quarto production):

asyncmap(1:100; ntasks=3) do i
    run(`quarto render notebook_$(i)/notebook.qmd`)
end

ERROR: Julia server returned error after receiving "run" command:
Failed to run notebook: /tmp/job/notebook_5.qmd
ERROR: ArgumentError: input string is empty or only contains whitespace
Stacktrace:
  [1] tryparse_internal(::Type{UInt16}, s::String, startpos::Int64, endpos::Int64, base_::Int64, raise::Bool)
    @ Base ./parse.jl:115
  [2] #parse#619
    @ ./parse.jl:254 [inlined]
  [3] parse
    @ ./parse.jl:253 [inlined]
  [4] Malt.Worker(; env::Vector{String}, exeflags::Vector{String})
    @ Malt ~/.julia/packages/Malt/YJ2Ml/src/Malt.jl:118
  [5] Worker
    @ ~/.julia/packages/Malt/YJ2Ml/src/Malt.jl:103 [inlined]
  [6] #1
    @ ~/.julia/packages/QuartoNotebookRunner/eYIiC/src/server.jl:24 [inlined]
  [7] cd(f::QuartoNotebookRunner.var"#1#2"{Vector{String}, Vector{String}}, dir::String)
    @ Base.Filesystem ./file.jl:112
  [8] QuartoNotebookRunner.File(path::String, options::Dict{String, Any})
    @ QuartoNotebookRunner ~/.julia/packages/QuartoNotebookRunner/eYIiC/src/server.jl:24
  [9] (::QuartoNotebookRunner.var"#41#44"{Bool, Dict{String, Any}, Server, String})()
    @ QuartoNotebookRunner ~/.julia/packages/QuartoNotebookRunner/eYIiC/src/server.jl:1155
 [10] lock(f::QuartoNotebookRunner.var"#41#44"{Bool, Dict{String, Any}, Server, String}, l::ReentrantLock)
    @ Base ./lock.jl:232
 [11] borrow_file!(f::QuartoNotebookRunner.var"#32#36"{Nothing, String, Bool, Dict{String, Any}, QuartoNotebookRunner.var"#chunk_callback#67"{TCPSocket}, Server}, server::Server, path::String; optionally_create::Bool, options::Dict{String, Any})
    @ QuartoNotebookRunner ~/.julia/packages/QuartoNotebookRunner/eYIiC/src/server.jl:1148
 [12] borrow_file!
    @ ~/.julia/packages/QuartoNotebookRunner/eYIiC/src/server.jl:1139 [inlined]
 [13] #run!#31
    @ ~/.julia/packages/QuartoNotebookRunner/eYIiC/src/server.jl:1108 [inlined]
 [14] _handle_response(socket::TCPSocket, notebooks::Server, request::@NamedTuple{type::String, content::Union{Dict{String, Any}, String}}, showprogress::Bool)
    @ QuartoNotebookRunner ~/.julia/packages/QuartoNotebookRunner/eYIiC/src/socket.jl:273
 [15] (::QuartoNotebookRunner.var"#59#66"{Bool, Base.RefValue{Bool}, Server, Base.UUID})()
    @ QuartoNotebookRunner ~/.julia/packages/QuartoNotebookRunner/eYIiC/src/socket.jl:201
ERROR: Internal julia server error

[jkrumbiegel]

In my case it was always invalid exeflags that caused this error, are you setting any?

[svilupp]

No, we're not setting any exeflags. The process runs and has been running for a week.
Today was the first day it failed with this error (and repeatedly). It's just weird flukes that happened at different times/files.

We produce c. 30 notebooks, each takes ~1-2 mins and we were running 3 in parallel (ntasks=3), so it seems unlikely they all hit some file/lock exactly at the same time.

Our header, if anyone comes across it as well:

---
format:
  html:
    output-file: report.html
    toc: true
    toc-depth: 3
    self-contained: true
    css: styles.css
    page-layout: full
    scrollable: false
execute:
  echo: false
engine: julia
---

[jkrumbiegel]

You could try cloning QuartoNotebookRunner somewhere, then dev'ing JuliaPluto/Malt.jl#85 into it, then using it for quarto by setting QUARTO_JULIA_PROJECT="the/path/to/quartonotebookrunner". That might give you the specific underlying error?

[MichaelHatherly]

#232 now uses a modified vendored version of the Malt code that tries to handle this kind of error better by providing improved error messages when no port is returned, ie. when the worker fails to start correctly. We now get a print out of the stacktrace, so hopefully it'll make tracking this one down a bit easier.