client.run() causing TypeError: TLS expects a ssl_context

[scottyhq]

@TomAugspurger, @jcrist, @consideRatio - We're still having some trouble with dask-gateway 0.7.1. Thought I'd create a dedicated issue because this has come up in a few different comments.

• Bump to Dask Gateway 0.7.1 #584 (comment)
• Planning issue multicloud-demo#1 (comment)

Any client.run command for example client.run(os.getpid) fails on aws-uswest2 and hydro currently. This does work on hub.pangeo.io with older dask versions...

dask                      2.14.0                     py_0    conda-forge
dask-core                 2.14.0                     py_0    conda-forge
dask-gateway              0.7.1            py37hc8dfbb8_0    conda-forge
dask-kubernetes           0.10.1                     py_0    conda-forge
dask-labextension         2.0.1                      py_0    conda-forge
pangeo-dask               2020.04.14                    0    conda-forge

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-34-6ebc4bcc7c73> in <module>
----> 1 client.run(os.getpid)

/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/client.py in run(self, function, *args, **kwargs)
   2430         >>> c.run(print_state, wait=False)  # doctest: +SKIP
   2431         """
-> 2432         return self.sync(self._run, function, *args, **kwargs)
   2433 
   2434     def run_coroutine(self, function, *args, **kwargs):

/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/client.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    778         else:
    779             return sync(
--> 780                 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
    781             )
    782 

/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
    345     if error[0]:
    346         typ, exc, tb = error[0]
--> 347         raise exc.with_traceback(tb)
    348     else:
    349         return result[0]

/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/utils.py in f()
    329             if callback_timeout is not None:
    330                 future = asyncio.wait_for(future, callback_timeout)
--> 331             result[0] = yield future
    332         except Exception as exc:
    333             error[0] = sys.exc_info()

/srv/conda/envs/notebook/lib/python3.7/site-packages/tornado/gen.py in run(self)
    733 
    734                     try:
--> 735                         value = future.result()
    736                     except Exception:
    737                         exc_info = sys.exc_info()

/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/client.py in _run(self, function, nanny, workers, wait, *args, **kwargs)
   2359             ),
   2360             workers=workers,
-> 2361             nanny=nanny,
   2362         )
   2363         results = {}

/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/core.py in send_recv_from_rpc(**kwargs)
    748             name, comm.name = comm.name, "ConnectionPool." + key
    749             try:
--> 750                 result = await send_recv(comm=comm, op=key, **kwargs)
    751             finally:
    752                 self.pool.reuse(self.addr, comm)

/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/core.py in send_recv(comm, reply, serializers, deserializers, **kwargs)
    547         if comm.deserialize:
    548             typ, exc, tb = clean_exception(**response)
--> 549             raise exc.with_traceback(tb)
    550         else:
    551             raise Exception(response["text"])

/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/core.py in handle_comm()
    401                             result = asyncio.ensure_future(result)
    402                             self._ongoing_coroutines.add(result)
--> 403                             result = await result
    404                     except (CommClosedError, CancelledError) as e:
    405                         if self.status == "running":

/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/scheduler.py in broadcast()
   2982 
   2983         results = await All(
-> 2984             [send_message(address) for address in addresses if address is not None]
   2985         )
   2986 

/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/utils.py in All()
    235     while not tasks.done():
    236         try:
--> 237             result = await tasks.next()
    238         except Exception:
    239 

/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/scheduler.py in send_message()
   2975         async def send_message(addr):
   2976             comm = await connect(
-> 2977                 addr, deserialize=self.deserialize, connection_args=self.connection_args
   2978             )
   2979             comm.name = "Scheduler Broadcast"

/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/comm/core.py in connect()
    226                 with ignoring(TimeoutError):
    227                     comm = await asyncio.wait_for(
--> 228                         future, timeout=min(deadline - time(), 1)
    229                     )
    230                     break

/srv/conda/envs/notebook/lib/python3.7/asyncio/tasks.py in wait_for()
    440 
    441         if fut.done():
--> 442             return fut.result()
    443         else:
    444             fut.remove_done_callback(cb)

/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/comm/tcp.py in connect()
    334         self._check_encryption(address, connection_args)
    335         ip, port = parse_host_port(address)
--> 336         kwargs = self._get_connect_args(**connection_args)
    337 
    338         try:

/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/comm/tcp.py in _get_connect_args()
    373 
    374     def _get_connect_args(self, **connection_args):
--> 375         ctx = _expect_tls_context(connection_args)
    376         return {"ssl_options": ctx}
    377 

/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/comm/tcp.py in _expect_tls_context()
    311             "TLS expects a `ssl_context` argument of type "
    312             "ssl.SSLContext (perhaps check your TLS configuration?)"
--> 313             "  Instead got %s" % str(ctx)
    314         )
    315     return ctx

TypeError: TLS expects a `ssl_context` argument of type ssl.SSLContext (perhaps check your TLS configuration?)  Instead got None

[TomAugspurger]

Ah I ran into this in the pangeo multi-cloud repo. So I’ll be debugging it tomorrow anyway.

[scottyhq]

Well we tagged a new set of docker images today and the error has gone away! I don't plan on digging into why it was occurring but using pangeo/pangeo-notebook:2020.05.10 (or latest) does the trick. Here is the complete list of packages that changed between the last version in case somebody wants to figure out what the issue was:
https://github.com/pangeo-data/pangeo-docker-images/compare/2020.05.04..2020.05.10#diff-fc143938f9485967d3be2239526ec787

[TomAugspurger]

Confirmed that it's working in https://github.com/pangeo-data/multicloud-demo as well. I'm also not sure what changed, but happy to remain ignorant.

Good to close?