Machine tokens become invalid when workspaces are recovered

[sleshchenko]

Description

Now machine tokens are stored in memory by workspace server. So after restarting of a workspace server, all machines token become invalid so workspace agent has no ability to request workspace server.
It is needed to investigate how is issue can be solved.
Possible solutions:

• there should be a mechanism to provide a new valid token to recovered workspaces;
• workspace server should persist machine tokens somehow not in memory not to lose them after restarting;
• declare current MACHINE_TOKEN env var as something like INIT_MACHINE_TOKEN, rework AgentHttpJsonRequestFactory to use the token of the current subject instead of initing machine token, generate new machine token after workspace recovering.

The third one is preferable.

It is also required to rework a bit generating of machine tokens to make it while token requesting instead of interception of workspace start method.

[akorneta]

The third solution looks good, but in this case, it is not fully clear what to do with the bootstrapper which takes a machine token from an environment and use it to publish logs. So we need to figure out possible ways, how to put bootstrapper a valid token.

[akorneta]

The goal that we are trying to achieve is to be able to provide a stateless mechanism of interaction between Che master and workspaces, so it means that we need to rework existing codebase related to machine authentication, managing of machine tokens.
One of the possible options is a replacement of machine token (that is stored on a Che master side in runtime storage) to JWT token which would be signed on master and placed with a public part in an environment of the workspace.
In this way, we avoid +1 request for token verification on machine side and we also don't need any storage for the tokens on Che master side. When new key pair generated all the workspaces must be restarted(this also may be discussed and proposed solutions welcome).

[akorneta]

Draft with JWT tokens that are used for authentication between workspace agents and Che master.