Handling exceptions in Worker plugins

[pentschev]

I have noticed that Worker plugins exceptions are currently not handled, but they are checked and returned as a message to the caller in

distributed/distributed/worker.py

Lines 2392 to 2398 in a192b22

	try:
	result = plugin.setup(worker=self)
	if isawaitable(result):
	result = await result
	except Exception as e:
	msg = error_message(e)
	return msg

Specifically, plugins are awaited in Worker.start():

distributed/distributed/worker.py

Lines 1142 to 1144 in a192b22

	await asyncio.gather(
	*[self.plugin_add(plugin=plugin) for plugin in self._pending_plugins]
	)

I'm not sure what is supposed to happen with the error message, is it supposed to be passed to the scheduler somehow? Currently it seems to me that these message are silently ignored and not logged at all, but I am still not sure what is the expected way to handle this. For me, it looks as if we should check the status of message of each plugin, and if it raised an exception we should re-raise them within Worker.start() itself. Is that the right solution or is there something else we should do here? I'm happy to work on a fix either way, but I wanted to check what's the right way to fix it before spending the time on a potentially flawed fix.

[mrocklin]

Failing on start seems sensible to me.

…

On Wed, Dec 2, 2020 at 3:42 AM Peter Andreas Entschev < ***@***.***> wrote: I have noticed that Worker plugins exceptions are currently not handled, but they are checked and returned as a message to the caller in https://github.com/dask/distributed/blob/a192b22dece37b9e151b71aaaa129a409f5bc630/distributed/worker.py#L2392-L2398 Specifically, plugins are awaited in Worker.start(): https://github.com/dask/distributed/blob/a192b22dece37b9e151b71aaaa129a409f5bc630/distributed/worker.py#L1142-L1144 I'm not sure what is supposed to happen with the error message, is it supposed to be passed to the scheduler somehow? Currently it seems to me that these message are silently ignored and not logged at all, but I am still not sure what is the expected way to handle this. For me, it looks as if we should check the status of message of each plugin, and if it raised an exception we should re-raise them within Worker.start() itself. Is that the right solution or is there something else we should do here? I'm happy to work on a fix either way, but I wanted to check what's the right way to fix it before spending the time on a potentially flawed fix. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4297>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTG4GBJBBLZPLEPDTQTSSYR4HANCNFSM4UKL6HMQ> .