Fix launching non python kernels from a windows store installed jupyter

[rchiodo]

Fixes #567

The root cause of this problem is the path of the connection file that's passed to an external process.

From within the window's store python (where jupyter_client is running), the path looks like so:

C:\Users\rchiodo.REDMOND\AppData\Roaming\jupyter\runtime\kernel-e62fdf2f-ca02-4611-b38d-a363a8cdff10.json

However this path doesn't actually exist. It's a redirected path caused by the window's store version of python (see this bug here: https://bugs.python.org/issue41196)

What the actual path is this:

C:\Users\rchiodo.REDMOND\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\Roaming\jupyter\runtime\kernel-e62fdf2f-ca02-4611-b38d-a363a8cdff10.json

This change 'fixes' the path to the real path before launching the kernel.

Sorry but not sure how I'd write a test for this other than manually testing it with a Windows store python.

[davidbrochart]

Note that cmd represents Python code, not all elements of the list are paths. I don't think it is safe to assume that an element that is not a path won't exist as a path by chance.

[davidbrochart]

Thanks for reporting the issue and submitting a PR @rchiodo. I agree replacing the path to the connection file with the "real path" can solve the issue, however it might be better to do it in:

jupyter_client/jupyter_client/manager.py

Lines 175 to 208 in 778fa85

	def format_kernel_cmd(self, extra_arguments=None):
	"""replace templated args (e.g. {connection_file})"""
	extra_arguments = extra_arguments or []
	if self.kernel_cmd:
	cmd = self.kernel_cmd + extra_arguments
	else:
	cmd = self.kernel_spec.argv + extra_arguments
	if cmd and cmd[0] in {'python',
	'python%i' % sys.version_info[0],
	'python%i.%i' % sys.version_info[:2]}:
	# executable is 'python' or 'python3', use sys.executable.
	# These will typically be the same,
	# but if the current process is in an env
	# and has been launched by abspath without
	# activating the env, python on PATH may not be sys.executable,
	# but it should be.
	cmd[0] = sys.executable
	ns = dict(connection_file=self.connection_file,
	prefix=sys.prefix,
	)
	if self.kernel_spec:
	ns["resource_dir"] = self.kernel_spec.resource_dir
	ns.update(self._launch_args)
	pat = re.compile(r'\{([A-Za-z0-9_]+)\}')
	def from_ns(match):
	"""Get the key out of ns if it's there, otherwise no change."""
	return ns.get(match.group(1), match.group())
	return [ pat.sub(from_ns, arg) for arg in cmd ]

[rchiodo]

Thanks for reporting the issue and submitting a PR @rchiodo. I agree replacing the path to the connection file with the "real path" can solve the issue, however it might be better to do it in:

jupyter_client/jupyter_client/manager.py

Lines 175 to 208 in 778fa85

	def format_kernel_cmd(self, extra_arguments=None):
	"""replace templated args (e.g. {connection_file})"""
	extra_arguments = extra_arguments or []
	if self.kernel_cmd:
	cmd = self.kernel_cmd + extra_arguments
	else:
	cmd = self.kernel_spec.argv + extra_arguments
	if cmd and cmd[0] in {'python',
	'python%i' % sys.version_info[0],
	'python%i.%i' % sys.version_info[:2]}:
	# executable is 'python' or 'python3', use sys.executable.
	# These will typically be the same,
	# but if the current process is in an env
	# and has been launched by abspath without
	# activating the env, python on PATH may not be sys.executable,
	# but it should be.
	cmd[0] = sys.executable
	ns = dict(connection_file=self.connection_file,
	prefix=sys.prefix,
	)
	if self.kernel_spec:
	ns["resource_dir"] = self.kernel_spec.resource_dir
	ns.update(self._launch_args)
	pat = re.compile(r'\{([A-Za-z0-9_]+)\}')
	def from_ns(match):
	"""Get the key out of ns if it's there, otherwise no change."""
	return ns.get(match.group(1), match.group())
	return [ pat.sub(from_ns, arg) for arg in cmd ]

I'm not sure that would work (I can try) but when I did something similar in multikernelmanager.py it broke python kernels (as I believe other spots must have the connection_file with the old path).

I think it would be better to just change the code in the launcher.py to instead verify it's the kernel-.json file. Then there wouldn't be a problem with possibly changing paths on things that aren't files.

[rchiodo]

No your suggested change did work. Thanks. I'll update to that.

[MSeal]

Thanks for tackling this! I didn't have a good answer for the issue last time it was raised so I'm glad someone understood it better and had a solution that's simple in mind.

Any chance we could update or add a test to cover functionality? The code below the change does some complex manipulations and I think it'd be a risk that someone refactoring that could break behavior unintentionally for windows store installations and not know.

[rchiodo]

@MSeal I'd be happy to add a test but it would have to install the window's store python (or perhaps mock realpath to force it to something different for a unit test).

[rchiodo]

Do you have any suggestions on where to start with say a unit test? That seems like the easier way to verify this doesn't regress.

[MSeal]

I would mock realpath to set the path to a fake location and test that the command returned by format_kernel_cmd has an expected realpath mocked value present. Should be a fairly small test to cover behavior though you will need to set kernel_cmd, connection_file on the tested object to ensure the function can complete.

There's no existing tests on format_kernel_cmd or pre_start_kernel which calls it. But the https://github.com/jupyter/jupyter_client/blob/master/jupyter_client/tests/test_kernelmanager.py file has some other examples testing adjacent or parent functions.

[MSeal]

Thanks for adding the test