Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nbconvert with Julia kernel fails inside Pkg.test #1031

Open
bencottier opened this issue Jan 6, 2022 · 5 comments
Open

nbconvert with Julia kernel fails inside Pkg.test #1031

bencottier opened this issue Jan 6, 2022 · 5 comments

Comments

@bencottier
Copy link

bencottier commented Jan 6, 2022 •
edited
Loading

I have a Julia package with a function that runs nbconvert - it's actually using Weave.notebook but I've since found a MWE.

I want to test the function that runs nbconvert but I hit an error when nbconvert runs the Julia kernel.

MWE setup

For the MWE I created a package/folder IJuliaPkgTest.jl with the following structure:

.
├── Project.toml
├── src
│   └── IJuliaPkgTest.jl
├── test
│   └── runtests.jl
└── test.ipynb

Project.toml:

name = "IJuliaPkgTest"
uuid = "f2c0e409-174f-4158-90ff-9978d98f62c8"

[deps]
IJulia = "7073ff75-c697-5162-941a-fcdaad2a7d2a"

src/IJuliaPkgTest.jl just contains module IJuliaPkgTest end.

test/runtests.jl:

using IJulia

run(`jupyter nbconvert --ExecutePreprocessor.timeout=-1 --to notebook --execute ../test.ipynb --debug --output ../test_executed.ipynb`)

Finally, test.ipynb:

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-06T08:04:32.314433Z",
     "iopub.status.busy": "2022-01-06T08:04:31.594744Z",
     "iopub.status.idle": "2022-01-06T08:04:33.800845Z",
     "shell.execute_reply": "2022-01-06T08:04:33.800037Z"
    }
   },
   "outputs": [],
   "source": [
    "println(\"Hello!\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.6.2",
   "language": "julia",
   "name": "julia-1.6"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

The problem

Here is the error when I run Pkg.test. The key thing is this after the julia kernel starts:

ERROR: LoadError: ArgumentError: Package Revise not found in current path:
- Run `import Pkg; Pkg.add("Revise")` to install the Revise package.

Stacktrace:
 [1] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:893
in expression starting at /Users/bencottier/.julia/config/startup.jl:1

As you can see I have a startup.jl file, which contains using Revise. I have Revise.jl in my base environment, but the base environment doesn't seem to be available here.

Full logs:

(IJuliaPkgTest) pkg> test
     Testing IJuliaPkgTest
      Status `/private/var/folders/g6/w_cfz8c507j6mbt5mdv7wt_h0000gn/T/jl_b8DcZa/Project.toml`
  [7073ff75] IJulia v1.23.2
  [f2c0e409] IJuliaPkgTest v0.0.0 `~/projects/testing/IJuliaPkgTest`
      Status `/private/var/folders/g6/w_cfz8c507j6mbt5mdv7wt_h0000gn/T/jl_b8DcZa/Manifest.toml`
  [8f4d0f93] Conda v1.6.0
  [7073ff75] IJulia v1.23.2
  [f2c0e409] IJuliaPkgTest v0.0.0 `~/projects/testing/IJuliaPkgTest`
  [692b3bcd] JLLWrappers v1.3.0
  [682c06a0] JSON v0.21.2
  [739be429] MbedTLS v1.0.3
  [69de0a69] Parsers v2.1.3
  [21216c6a] Preferences v1.2.3
  [b85f4697] SoftGlobalScope v1.1.0
  [81def892] VersionParsing v1.2.1
  [c2297ded] ZMQ v1.2.1
  [8f1865be] ZeroMQ_jll v4.3.4+0
  [a9144af2] libsodium_jll v1.0.20+0
  [0dad84c5] ArgTools `@stdlib/ArgTools`
  [56f22d72] Artifacts `@stdlib/Artifacts`
  [2a0f44e3] Base64 `@stdlib/Base64`
  [ade2ca70] Dates `@stdlib/Dates`
  [f43a241f] Downloads `@stdlib/Downloads`
  [7b1f6079] FileWatching `@stdlib/FileWatching`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [b27032c2] LibCURL `@stdlib/LibCURL`
  [76f85450] LibGit2 `@stdlib/LibGit2`
  [8f399da3] Libdl `@stdlib/Libdl`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [a63ad114] Mmap `@stdlib/Mmap`
  [ca575930] NetworkOptions `@stdlib/NetworkOptions`
  [44cfe95a] Pkg `@stdlib/Pkg`
  [de0858da] Printf `@stdlib/Printf`
  [3fa0cd96] REPL `@stdlib/REPL`
  [9a3f8284] Random `@stdlib/Random`
  [ea8e919c] SHA `@stdlib/SHA`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [6462fe0b] Sockets `@stdlib/Sockets`
  [fa267f1f] TOML `@stdlib/TOML`
  [a4e569a6] Tar `@stdlib/Tar`
  [8dfed614] Test `@stdlib/Test`
  [cf7118a7] UUIDs `@stdlib/UUIDs`
  [4ec0a83e] Unicode `@stdlib/Unicode`
  [deac9b47] LibCURL_jll `@stdlib/LibCURL_jll`
  [29816b5a] LibSSH2_jll `@stdlib/LibSSH2_jll`
  [c8ffd9c3] MbedTLS_jll `@stdlib/MbedTLS_jll`
  [14a3606d] MozillaCACerts_jll `@stdlib/MozillaCACerts_jll`
  [83775a58] Zlib_jll `@stdlib/Zlib_jll`
  [8e850ede] nghttp2_jll `@stdlib/nghttp2_jll`
  [3f19e933] p7zip_jll `@stdlib/p7zip_jll`
     Testing Running tests...
[NbConvertApp] Searching ['/Users/bencottier/projects/testing/IJuliaPkgTest/test', '/Users/bencottier/.jupyter', '/Users/bencottier/.julia/conda/3/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[NbConvertApp] Looking for jupyter_config in /etc/jupyter
[NbConvertApp] Looking for jupyter_config in /usr/local/etc/jupyter
[NbConvertApp] Looking for jupyter_config in /Users/bencottier/.julia/conda/3/etc/jupyter
[NbConvertApp] Looking for jupyter_config in /Users/bencottier/.jupyter
[NbConvertApp] Looking for jupyter_config in /Users/bencottier/projects/testing/IJuliaPkgTest/test
[NbConvertApp] Looking for jupyter_nbconvert_config in /etc/jupyter
[NbConvertApp] Looking for jupyter_nbconvert_config in /usr/local/etc/jupyter
[NbConvertApp] Looking for jupyter_nbconvert_config in /Users/bencottier/.julia/conda/3/etc/jupyter
[NbConvertApp] Looking for jupyter_nbconvert_config in /Users/bencottier/.jupyter
[NbConvertApp] Looking for jupyter_nbconvert_config in /Users/bencottier/projects/testing/IJuliaPkgTest/test
[NbConvertApp] Converting notebook ../test.ipynb to notebook
[NbConvertApp] Notebook name is '../test'
[NbConvertApp] Applying preprocessor: ExecutePreprocessor
[NbConvertApp] Starting kernel (async): ['/Applications/Julia-1.6.app/Contents/Resources/julia/bin/julia', '-i', '--color=yes', '--project=@.', '/Users/bencottier/.julia/packages/IJulia/e8kqU/src/kernel.jl', '/private/var/folders/g6/w_cfz8c507j6mbt5mdv7wt_h0000gn/T/tmpkzynyxq_.json']
[NbConvertApp] Connecting to: tcp://127.0.0.1:65133
[NbConvertApp] connecting shell channel to tcp://127.0.0.1:65130
[NbConvertApp] Connecting to: tcp://127.0.0.1:65130
[NbConvertApp] connecting iopub channel to tcp://127.0.0.1:65131
[NbConvertApp] Connecting to: tcp://127.0.0.1:65131
[NbConvertApp] connecting stdin channel to tcp://127.0.0.1:65132
[NbConvertApp] Connecting to: tcp://127.0.0.1:65132
[NbConvertApp] connecting heartbeat channel to tcp://127.0.0.1:65134
[NbConvertApp] connecting control channel to tcp://127.0.0.1:65133
[NbConvertApp] Connecting to: tcp://127.0.0.1:65133
ERROR: LoadError: ArgumentError: Package Revise not found in current path:
- Run `import Pkg; Pkg.add("Revise")` to install the Revise package.

Stacktrace:
 [1] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:893
in expression starting at /Users/bencottier/.julia/config/startup.jl:1
Traceback (most recent call last):
  File "/Users/bencottier/.julia/conda/3/bin/jupyter-nbconvert", line 11, in <module>
    sys.exit(main())
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/jupyter_core/application.py", line 254, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/traitlets/config/application.py", line 845, in launch_instance
    app.start()
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 350, in start
    self.convert_notebooks()
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 524, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 489, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 418, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 181, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 199, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/exporters/notebook.py", line 32, in from_notebook_node
    nb_copy, resources = super().from_notebook_node(nb, resources, **kw)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 143, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 318, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb, resources)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/preprocessors/execute.py", line 79, in preprocess
    self.execute()
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbclient/util.py", line 74, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbclient/util.py", line 53, in just_run
    return loop.run_until_complete(coro)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbclient/client.py", line 524, in async_execute
    async with self.async_setup_kernel(**kwargs):
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/async_generator/_util.py", line 34, in __aenter__
    return await self._agen.asend(None)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbclient/client.py", line 483, in async_setup_kernel
    await self.async_start_new_kernel_client()
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbclient/client.py", line 410, in async_start_new_kernel_client
    await ensure_async(self.kc.wait_for_ready(timeout=self.startup_timeout))
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbclient/util.py", line 85, in ensure_async
    result = await obj
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/jupyter_client/asynchronous/client.py", line 139, in wait_for_ready
    raise RuntimeError('Kernel died before replying to kernel_info')
RuntimeError: Kernel died before replying to kernel_info
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbclient/client.py", line 351, in _async_cleanup_kernel
    assert self.km is not None
AssertionError
ERROR: LoadError: failed process: Process(`jupyter nbconvert --ExecutePreprocessor.timeout=-1 --to notebook --execute ../test.ipynb --debug --output ../test.ipynb`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error
   @ ./process.jl:525 [inlined]
 [2] run(::Cmd; wait::Bool)
   @ Base ./process.jl:440
 [3] run(::Cmd)
   @ Base ./process.jl:438
 [4] top-level scope
   @ ~/projects/testing/IJuliaPkgTest/test/runtests.jl:4
 [5] include(fname::String)
   @ Base.MainInclude ./client.jl:444
 [6] top-level scope
   @ none:6
in expression starting at /Users/bencottier/projects/testing/IJuliaPkgTest/test/runtests.jl:4
ERROR: Package IJuliaPkgTest errored during testing

My guess is this has something to do with Pkg.test running julia with --startup-file=no and the IJulia kernel running with the default --startup-file=yes. The Pkg.test command looks like this:

/Applications/Julia-1.6.app/Contents/Resources/julia/bin/julia -Cnative -J/Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib --depwarn=yes --check-bounds=yes -g1 --color=yes --startup-file=no --project=/var/folders/g6/w_cfz8c507j6mbt5mdv7wt_h0000gn/T/jl_GYXYn9 -e <code>

But it's not just that, because this seems to run OK:

/Applications/Julia-1.6.app/Contents/Resources/julia/bin/julia -Cnative -J/Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib --depwarn=yes --check-bounds=yes -g1 --color=yes --startup-file=no --project=/var/folders/g6/w_cfz8c507j6mbt5mdv7wt_h0000gn/T/jl_GYXYn9 -e "run(`/Applications/Julia-1.6.app/Contents/Resources/julia/bin/julia -i --color=yes --project=@. /Users/bencottier/.julia/packages/IJulia/e8kqU/src/kernel.jl`)"

versioninfo

julia> versioninfo()
Julia Version 1.6.2
Commit 1b93d53fc4 (2021-07-14 15:36 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.7.0)
  CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)
Environment:
  JULIA_PKG_DEVDIR = /Users/bencottier/JuliaEnvs
  JULIA_EDITOR = code
@bencottier
Copy link
Author

bencottier commented Jan 6, 2022 •
edited
Loading

I think the reason it doesn't work with Pkg.test in particular is that Pkg.test ultimately runs this to remove the julia base environment from LOAD_PATH.

The reason I made an issue here is that I'd expect the IJulia kernel to inherit the --startup-file option from any parent process, unless there is particular reason not to. This kind of thing can be done using Base.julia_cmd(). But I guess there are special reasons for overriding other julia options, like -i?

@bencottier
Copy link
Author

bencottier commented Jan 6, 2022 •
edited
Loading

From the git blame of

IJulia.jl/deps/kspec.jl

Line 97 in 8710cd4

kernelcmd_array = String[julia.exec..., "-i", "--color=yes"]

I found JuliaLang/Pkg.jl#1815 which is a reason to not specify --startup-file 😕

@bencottier
Copy link
Author

Ah, I now realise the command to start the kernel is normally set one time in installkernel, so that makes it difficult to change the startup-file option.

@phillip-buelow
Copy link

@bencottier, are you able to nbconvert within the Julia REPL?

@bencottier
Copy link
Author

IIRC, yes I was able to nbconvert within the Julia REPL. (Sorry, I'm no longer working with Julia so this issue is no longer relevant to me.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bencottier @phillip-buelow