Terminated session viewable and joinable in webui after restarting Node

[Joerger]

Expected behavior:

Restart Teleport Node (^C) -> No active sessions on restart.

Current behavior:

Any session which was open during the restart will show up in the WebUI. You can also join the session on the WebUI, which will create a new session with the same ID, since the session doesn't actually exist. This was a weird user experience, and initially I thought there was a file leak on the Teleport Node.

Bug details:

• Teleport version - Always allow session owners to join own sessions + only list active trackers in WebUI #13660
• Recreation steps
  • Open a session with tsh ssh
  • Restart the Teleport Node
  • See the session still listed in the WebUI.
• Debug logs - N/A

Proposal

We have a few other similar issues which occur with non-graceful restart ^C

• Forward Agent and X11 unix sockets don't get cleaned up - Improve non-graceful restarts #10075
• Files are created and not removed - Warning - Skipped session recording #13640 (comment)

And we have some related fixes already in place:

• UploadCompleter uploads abandoned session uploads
  • It also emits a session.end event if it is missing

Improving graceful restart would be good, but it still wouldn't handle the case where the process is killed, etc.

Instead, we can expand the UploadCompleter to be a SessionCompleter and handle all related issues. It could exist in the Session service and work more closely with the SessionRegistry. This will also solve some racy issues with the UploadCompleter as it is now.

For now, it would find session trackers which don't have a corresponding session in the session registry, then, if applicable:

• Close associated unix sockets (x11 forwarding and agent forwarding)
• Complete uploading of the session recording
• Delete loose files (like the session upload place holder part file)
• Emit a session end event
• Delete the session tracker

For this to work, we'd need to remove the session tracker expiration (1 hour, extending during session lifetime), and instead track the sessions manually. This way, an abandoned upload would have an abandoned tracker, so we could move the upload completion logic to the SessionCompleter. To make this backwards compatible, we'd need to deprecate UploadCompleter over the next major version, which shouldn't be a problem since it won't overlap with the SessionCompleter.

[xacrimon]

Note: This became evident in v10 as we started to query sessions using trackers instead of the legacy system. This wasn't noticed too much as the legacy system had a very short heartbeat that traded heavy backend load for this issue. The reason for avoiding such a short timeout is that the auth could drop the session if the node lost connection or the network was bad.