How should distributed deal with mismatches between Python or package versions?

[hendrikmakait]

There are currently a few issues with the way we handle mismatches between Python or package versions:

• Client.get_versions(check=True) not behaving as expected from docs #5731 highlights that Client.get_versions(check=True) does not work as suggested in the docstring and never raises.
  • This behavior has changed multiple times in the past (Err hard when Dask has mismatched versions or lz4 installed #3936, Add compression, pickle protocol to comm contexts #4019)
• SystemError: unknown opcode with SchedulerPlugin when client uses Python3.10 and scheduler Python3.9 #7016 shows that using different Python versions between client and cluster may result in a SystemError or segfault.
• Client._ensure_connected contains dormant functionality to raise an ImportError in that is currently not triggered due to the implementation of versions.error_message (see point 1):

  distributed/distributed/client.py

  Lines 1306 to 1307 in 0a0cfb8

  	if msg[0].get("error"):
  	raise ImportError(msg[0]["error"])

• versions.error_message does not consider the Python version of the scheduler (fix in Detect mismatching Python version in scheduler #7018)

This raises the question how distributed should deal with version mismatches in general and how tolerant it should be, if at all. My current thoughts are that we should be strict about Python, required and optional packages again due to #7016 and the fragility of using different versions of dask-related packages. I don't know if we should necessarily raise per default in Client._ensure_connected, but we should at least make sure that we can raise using a flag (similar to Client.get_versions(check=True)).

Notes

Cloudpickle can only be used to send objects between the exact same version of Python.

https://github.com/cloudpipe/cloudpickle

[shughes-uk]

I'd be happy if everything exploded with a sensible error on mismatches for critical packages (and python). The errors that can creep in otherwise are a nightmare to debug if you haven't encountered them before. The more devious ones can be damn near impossible for anyone.

I suspect most people on the Dask dev team can recognize the usual suspects near instantly (a deserialization exception is usually a dead giveaway) , but for a new dask user they can be totally inscrutable.

I'm also very happy to walk people through the internals of the Coiled package sync feature and how it does package version detection too, it can be non-trivial. I believe right now dask is kinda cheating using things<pkg>.__version__ and a lot of try: except: