tui: prevent hang on shutdown

[oliver-sanders]

It has been reported that Tui can hang on shutdown.

The easiest way to reproduce this is to press "q" as soon as starting Tui. Sometimes this can cause it to hang.

$ cylc vip <workflow>
$ cylc tui <workflow>
q

This PR adds a timeout on the updater shutdown to prevent any hanging operation stopping Tui from exiting.

This isn't really the "right" way to solve the problem, getting the updater to exit cleanly in all circumstances would be nicer. However, it's a reasonable fail safe and since the updater is a read-only process (i.e. the only "resources" it's managing are Cylc clients which have timeouts built in) there are no ill effects of terminating it.

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Not reasonably testable.
• CHANGES.md entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

[wxtim]

I have struggled to repeat the original bug, but this looks ok.

[oliver-sanders]

@wxtim, the original bug isn't reproducing as easily as it originally did. To simulate this bug, try changing break to continue in this code:

cylc-flow/cylc/flow/tui/updater.py

Lines 148 to 159 in 67a6ba5

	async def run(self, filters):
	"""Start the updater in an existing asyncio.loop.
	The tests call this within the same process.
	"""
	with suppress_logging():
	self._update_filters(filters)
	while True:
	ret = await self._update()
	if ret == self.SIGNAL_TERMINATE:
	break
	self.update_queue.put(ret)

That will prevent the updater from shutting down. Before this commit, Tui should hang indefinitely, after this PR it should exit after 4 seconds (due to the timeout).

[wxtim]

try changing break to continue in this code

Looks like I can now repeat the problem. Pleasingly this branch fixes it!