Kernel error: jupyter_client/connect.py AssertionError

[dahjan]

Error message

Traceback (most recent call last):
File "/home/djan/anaconda3/lib/python3.7/site-packages/tornado/web.py", line 1699, in _execute
result = await result
File "/home/djan/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/home/djan/anaconda3/lib/python3.7/site-packages/notebook/services/sessions/handlers.py", line 72, in post
type=mtype))
File "/home/djan/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/home/djan/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/home/djan/anaconda3/lib/python3.7/site-packages/notebook/services/sessions/sessionmanager.py", line 88, in create_session
kernel_id = yield self.start_kernel_for_session(session_id, path, name, type, kernel_name)
File "/home/djan/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/home/djan/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/home/djan/anaconda3/lib/python3.7/site-packages/notebook/services/sessions/sessionmanager.py", line 101, in start_kernel_for_session
self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name)
File "/home/djan/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/home/djan/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/home/djan/anaconda3/lib/python3.7/site-packages/notebook/services/kernels/kernelmanager.py", line 168, in start_kernel
super(MappingKernelManager, self).start_kernel(**kwargs)
File "/home/djan/anaconda3/lib/python3.7/site-packages/jupyter_client/multikernelmanager.py", line 110, in start_kernel
km.start_kernel(**kwargs)
File "/home/djan/anaconda3/lib/python3.7/site-packages/jupyter_client/manager.py", line 240, in start_kernel
self.write_connection_file()
File "/home/djan/anaconda3/lib/python3.7/site-packages/jupyter_client/connect.py", line 547, in write_connection_file
kernel_name=self.kernel_name
File "/home/djan/anaconda3/lib/python3.7/site-packages/jupyter_client/connect.py", line 212, in write_connection_file
with secure_write(fname) as f:
File "/home/djan/anaconda3/lib/python3.7/contextlib.py", line 112, in enter
return next(self.gen)
File "/home/djan/anaconda3/lib/python3.7/site-packages/jupyter_client/connect.py", line 105, in secure_write
assert '0600' == oct(stat.S_IMODE(os.stat(fname).st_mode)).replace('0o', '0')
AssertionError

System information

Running Jupyter (v. 1.0.0) on a Linux server, with anaconda3 (v. 2019.07) 64-bit.

Available Kernels

Available kernels:
python3 /home/djan/.local/share/jupyter/kernels/python3
nn /home/djan/.local/share/jupyter/kernels/nn

Reproducing the issue

1. Install the newest anaconda3 version.
2. Run Jupyter notebook in the base version --> Kernel runs fine
3. Create a new environment with python: conda create -n test python=3.7
4. After activating the environment, Jupyter notebook is not available, thus I must install it via Conda: conda install -c anaconda jupyter
5. Run Jupyter notebook in the test environment --> Kernel breaks down as explained above

What I tried so far

There seems to be an issue with file permissions when running Jupyter on a server inside a Conda environment. I tried to give all files inside my anaconda3 folder the requested permissions (0600), but this did not help.

[dahjan]

Update

It turns out that the newest version of jupyter_client, available under anaconda as version 5.3.3, breaks the Jupyter environment on a server.

Installing the previous version of jupyter_client via following command fixes this issue:

conda install -c anaconda jupyter_client=5.3.1

[mirekphd]

The AssertionError message I get when an assertion fails in line 105 of connect.py of the two latest versions of jupyter_client affected by this bug is the same or rather varies slightly in its string contents:

File "/opt/conda/lib/python3.6/site-packages/jupyter_client/connect.py", line 105, in secure_write
assert '0600' == oct(stat.S_IMODE(os.stat(fname).st_mode)).replace('0o', '0')
AssertionError

The downgrade to 5.3.1 proposed by @dahjan worked for me as well. This means that the bug was introduced in 5.3.3 and remains in 5.3.4.

@dahjan - please reopen the issue - downgrading the pkg is just a temp workaround, and we need a future-proof solution.

Appeal to the developers: please remove BOTH broken versions of jupyter_client (5.3.4 and 5.3.3) from BOTH pypi and Anaconda's conda-forge channel.

More info

I suspect the bug will be a long-lasting one because client-server connections to python 3.6 kernels in Jupyter Notebook and Lab work under some circumstances. For instance, they work correctly under docker run on a rather insecure server (with all ports open) but fail with AssertionError under my corporate (Red Hat) Openshift installation where only ports 8888 and 80 are open between the client and server and there is no routing to the internet (the most relevant differences to this bug I think), so 99% of testers will fail to reproduce it (who in the open source community tests Jupyter kernel connections on on-prem Openshift installations..?:)

My suspicion is confirmed by the fact that it now applies to TWO historical versions of jupyter_client, so this standard downgrade-by-one trick did not work for me when applied to version 5.3.4 (because 5.3.3 is affected too).

[kevin-bates]

Hi @mirekphd - I'm sorry about the inconvenience. The latest release of jupyter_client (5.3.4) did two things.

1. It moved the code for secure_write into the jupyter_core (4.6.0) package
2. It addressed a slippery win32 issue (in the moved code in jupyter_core)

I agree that there is likely a non-win32 issue here relative to the "old" statement you show from jupyter_client. However, that particular statement is no longer an assertion, but a raise with a detailed message containing the current permissions - which would prove extremely helpful in this particular case.

Could you please confirm that you're using the true latest versions of each library and try again so that we can gain access to what the permissions happen to be on this particular platform?

cc: @MSeal

[ketozhang]

Could you please confirm that you're using the true latest versions of each library

Can you clarify again which libraries? I understand jupyter_core is one.

[kevin-bates]

jupyter_client is the other.

[scivm]

I made some comments on the original PR that caused this issue. It is still happening in all the latest docker images in the Jupyter project (build 1 month ago.) I have to add:

RUN conda install --yes jupyter_client==5.3.1

in all my Dockerfiles.

[bollwyvl]

So is an updated jupyter_client bottom pin of jupyter_core >=4.6.1 now appropriate?

[scivm]

Yes, my issue is related to Azurefile cifs filesystem permissions. There has been some work to solve the issue in Jupyter core which resulted in 4.6.1 allowing execute bit on owner. CIF mounts are setting the execute bit on owner which is difficult to prevent. I am now looking at trying to change some mount options related to group and world which came as default with jupyterhub azurefile storage type I am using.

jupyter/jupyter_core#172

[a114383]

After downgrading 5.3.1 I recieve this error message:

$> jupyter --version
jupyter core     : 4.7.0
jupyter-notebook : 6.1.6
qtconsole        : not installed
ipython          : 7.16.1
ipykernel        : 5.4.3
jupyter client   : 5.3.1
jupyter lab      : not installed
nbconvert        : 6.0.7
ipywidgets       : not installed
nbformat         : 5.0.8
traitlets        : 4.3.3

$> jupyter lab --port=18777
Traceback (most recent call last):
  File "/home/a114383/miniconda3/envs/pfft/bin/jupyter-lab", line 5, in <module>
    from jupyterlab.labapp import main
  File "/home/a114383/miniconda3/envs/pfft/lib/python3.6/site-packages/jupyterlab/__init__.py", line 7, in <module>
    from .labapp import LabApp
  File "/home/a114383/miniconda3/envs/pfft/lib/python3.6/site-packages/jupyterlab/labapp.py", line 16, in <module>
    from jupyterlab_server import slugify, WORKSPACE_EXTENSION
  File "/home/a114383/miniconda3/envs/pfft/lib/python3.6/site-packages/jupyterlab_server/__init__.py", line 4, in <module>
    from .app import LabServerApp
  File "/home/a114383/miniconda3/envs/pfft/lib/python3.6/site-packages/jupyterlab_server/app.py", line 11, in <module>
    from jupyter_server.extension.application import ExtensionApp, ExtensionAppJinjaMixin
  File "/home/a114383/miniconda3/envs/pfft/lib/python3.6/site-packages/jupyter_server/extension/application.py", line 21, in <module>
    from jupyter_server.serverapp import ServerApp
  File "/home/a114383/miniconda3/envs/pfft/lib/python3.6/site-packages/jupyter_server/serverapp.py", line 75, in <module>
    from .services.kernels.kernelmanager import MappingKernelManager, AsyncMappingKernelManager
  File "/home/a114383/miniconda3/envs/pfft/lib/python3.6/site-packages/jupyter_server/services/kernels/kernelmanager.py", line 20, in <module>
    from jupyter_client.multikernelmanager import MultiKernelManager, AsyncMultiKernelManager
ImportError: cannot import name 'AsyncMultiKernelManager'

Tried to upgrade to the most recent as suggested here, but arrived back here:

$> conda update jupyter_client
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.8.4
  latest version: 4.9.2

Please update conda by running

    $ conda update -n base conda



## Package Plan ##

  environment location: /home/a114383/miniconda3/envs/pfft

  added / updated specs:
    - jupyter_client


The following packages will be UPDATED:

  jupyter_client                                 5.3.1-py_0 --> 6.1.11-pyhd8ed1ab_1


Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

$> jupyter lab --port=18777
[I 2021-01-12 16:19:57.398 ServerApp] jupyterlab | extension was successfully linked.
[I 2021-01-12 16:19:58.239 ServerApp] nbclassic | extension was successfully linked.
[I 2021-01-12 16:19:58.459 LabApp] JupyterLab extension loaded from /home/a114383/miniconda3/envs/pfft/lib/python3.6/site-packages/jupyterlab
[I 2021-01-12 16:19:58.459 LabApp] JupyterLab application directory is /home/a114383/miniconda3/envs/pfft/share/jupyter/lab
[I 2021-01-12 16:19:58.464 ServerApp] jupyterlab | extension was successfully loaded.
[I 2021-01-12 16:19:58.476 ServerApp] nbclassic | extension was successfully loaded.
[I 2021-01-12 16:19:58.476 ServerApp] Serving notebooks from local directory: /home/a114383/PLP-PFFT
[I 2021-01-12 16:19:58.476 ServerApp] Jupyter Server 1.2.1 is running at:
[I 2021-01-12 16:19:58.476 ServerApp] http://localhost:18777/lab?token=0ba569d54d0c2e662ec88e17dfbd09896cef10f7c054f089
[I 2021-01-12 16:19:58.476 ServerApp]  or http://127.0.0.1:18777/lab?token=0ba569d54d0c2e662ec88e17dfbd09896cef10f7c054f089
[I 2021-01-12 16:19:58.476 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Traceback (most recent call last):
  File "/home/a114383/miniconda3/envs/pfft/bin/jupyter-lab", line 8, in <module>
    sys.exit(main())
  File "/home/a114383/miniconda3/envs/pfft/lib/python3.6/site-packages/jupyter_server/extension/application.py", line 504, in launch_instance
    serverapp.start()
  File "/home/a114383/miniconda3/envs/pfft/lib/python3.6/site-packages/jupyter_server/serverapp.py", line 1972, in start
    self.start_app()
  File "/home/a114383/miniconda3/envs/pfft/lib/python3.6/site-packages/jupyter_server/serverapp.py", line 1933, in start_app
    self.write_server_info_file()
  File "/home/a114383/miniconda3/envs/pfft/lib/python3.6/site-packages/jupyter_server/serverapp.py", line 1823, in write_server_info_file
    with secure_write(self.info_file) as f:
  File "/home/a114383/miniconda3/envs/pfft/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/a114383/miniconda3/envs/pfft/lib/python3.6/site-packages/jupyter_server/utils.py", line 291, in secure_write
    assert '0600' == oct(stat.S_IMODE(os.stat(fname).st_mode)).replace('0o', '0')
AssertionError

[kevin-bates]

Hi @a114383 - updating jupyter_client (in the first half of your comment) was the correct thing to do. Unfortunately, it exposed an issue in jupyter_server in which the older secure_write method is still in place and should be replaced with the secure_write method in jupyter_core. This update will likely still trigger a similar failure, but the messaging should be a bit more helpful.

I suspect you'd be able to simulate that error by attempting to run jupyter notebook in this environment - since notebook uses the correct variant of secure_write. Could you please try that? I'd like to be able to close this issue here.

I will open a new issue in jupyter_server regarding my previous comment.

[a114383]

Thanks @kevin-bates - Looks like I can run jupyter notebook just not jupyter labs

[kevin-bates]

Thanks for the update @a114383 - that is very helpful. I'm curious why you didn't encounter the same security-related error when running against the notebook server. Are you setting the env JUPYTER_ALLOW_INSECURE_WRITES to bypass the security checks due to the filesystem's configuration? (This is another one of the differences provided by the code in jupyter_core.)

I suspect you'd be able to run lab using a release < 3.x since it's only the 3.x releases that rely on jupyter_server. That said, we could probably get a jupyter_server release built that uses the core library for this relatively soon.

[anupsaha1]

Having similar issue here is the doc file i included everything in details please have a look at it and respond please really struggling for past few days
https://docs.google.com/document/d/1reYwQGEtynpjdxrFkSUHHA1WSMX0Qs90mVy2248PIbY/edit

[kevin-bates]

Hi @anupsaha1 - this is a different issue related pyzmq on Windows. Please see this issue for guidance: zeromq/pyzmq#1505.

[anupsaha1]

Hi @anupsaha1 - this is a different issue related pyzmq on Windows. Please see this issue for guidance: zeromq/pyzmq#1505.
the pyzmq installation was incomplete and I am getting the error mind if you have a loon into it zeromq/pyzmq#1505

[anupsaha1]

thanks mate

…

Sent from my Mi phone Jan 25, 2022 12:11 AM wrote on Kevin Bates ***@***.***>: Hi @anupsaha1<https://github.com/anupsaha1> - this is a different issue related pyzmq on Windows. Please see this issue for guidance: zeromq/pyzmq#1505<zeromq/pyzmq#1505>. — Reply to this email directly, view it on GitHub<#4937 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ARDNBUX3UXTE3BXBWPSGSK3UXV22JANCNFSM4I4YBOBA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[kevin-bates]

Glad to hear you're moving forward @anupsaha1.

I'm going to close this issue since the original issue is referring to a source code location that has since been moved and seems to be addressed, and the issue is starting to splinter into unrelated areas.

[sugamkarki]

I'm still getting the same error even after downgrading jupyter_client to 5.3.1. My error message is slightly different than the original error posted.

Traceback (most recent call last):
  File "G:\anaconda\envs\tf\lib\runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "G:\anaconda\envs\tf\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\sk_np\AppData\Roaming\Python\Python38\site-packages\ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "G:\anaconda\envs\tf\lib\site-packages\traitlets\config\application.py", line 846, in launch_instance
    app.start()
  File "C:\Users\sk_np\AppData\Roaming\Python\Python38\site-packages\ipykernel\kernelapp.py", line 477, in start
    ioloop.IOLoop.instance().start()
  File "G:\anaconda\envs\tf\lib\site-packages\tornado\platform\asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "G:\anaconda\envs\tf\lib\asyncio\windows_events.py", line 314, in run_forever
    assert self._self_reading_future is None
AssertionError

[kevin-bates]

Hi @sugamkarki - this appears to be unrelated to other issues in this thread. Yes, it's an assertion error, but that's about the only relationship.

I don't think downgrading jupyter_client to 5.3.1 was the correct move here. That's a very old release and the issue is coming from ipykernel - not the notebook server.

This smells like a Windows event loop issue to me. Instead, I would make sure you're running the latest jupyter_client (certainly >= 6.x), ipykernel, and ipython. It would also be good to know which version of python is in use (along with the other packages just mentioned).

[sugamkarki]

@kevin-bates I've recreated the env with [email protected] , [email protected], [email protected]. The env is with [email protected]. No matter how many times I reinstall the environment, the issue still persists. Shall I open a new issue for this?

[kevin-bates]

Thank you for the version information.

Shall I open a new issue for this?

Probably so. However, the issue should be filed in https://github.com/ipython/ipykernel/issues since that's where the application (ipykernel) resides. Prior to that, I would suggest installing its latest version (6.9.0) since there may have been some fixes for this and see if anything changes.