Investigate how to run IDE tooling in separate containers

[garagatyi]

Related to SPI branch only.
There is a concept called sidecar. It suggests starting wsagent, terminal, exec and other servers as a separate containers. We need to investigate how it can work with design of new SPI

[gorkem]

@benoitf can you please add the details from your PoC here?

[benoitf]

@gorkem
Adding link to the document on side car containers : https://docs.google.com/document/d/1HaiVOLQlBqYWNJ9tALpFF1xkv7CQ2wl6IyqTBMJdn2k/edit

[gazarenkov]

@garagatyi let's consider something practical here.
I would say that the best PoC for the time is making some of the LS work on dedicated machine.
To make it work with IDE It requires both:

• server communicating with ws-agent
• sharing (mounting) projects folder
  I would suggest to sync with @evidolob

[gazarenkov]

look at the @eivantsov 's answer #5634 as well

[garagatyi]

I investigated the usage of LSs as separate containers (sidecars). Here are findings, tasks, and questions that should be addressed to implement sidecar concept in Docker infra:

1. We can't say for sure that installers won't be needed and all the cases can be covered by sidecars, so we can't remove installers workflow for now. But some infrastructure may forbid usage of installers or, for example, not provide sudo, so installers that require sudo will fail and workspace start will fail. Because of that sidecar and installer concepts should coexist for now.
2. We don't have any specific place to describe sidecar in workspace config. So we need to decide how it would look like in the workspace model.

The current model of the environment is:

{
    "recipe" : {...}, // Environment specific recipe file, such as Docker Compose YAML, or some k8s specs.
    "machines" : {
        "mymachine1" : {
            "installers" : [
                "org.eclipse.che.terminal",
                "org.eclipse.che.exec"
            ],
            "servers" : {
                "server1" : {
                    "port" : "8080",
                    "protocol" : "http",
                    "path" : "/my/path"
                }
            },
            "attributes" : {
                "some.attribute" : "attribute value",
                ...
            }
        },
        ...
    }
}

3. It is not clear how to define dependencies between sidecars and other machines. What resources can they expose and consume? Looks like language servers needs only projects folder, but it is not clear on how to identify which path is needed for a sidecar. What about other paths (not projects folder), can sidecars use such paths? How sidecar can change container configuration of other machines/sidecars.
   For example, an LS needs the projects folder. It also needs to pass some configuration to wsagent, such as files patterns, its URL, etc. If it is a maven LS it might want to share maven local repo also. And other LSs might use other locations.

4. We don't have a place for defining sidecar's docker image (in a case of other infras it can be something else).

5. Make user configure RAM limits for each sidecar is bad UX, so we need to figure out how to set RAM limit for a set of sidecars (wsagent, LSs, etc).

6. The current implementation of LS protocol in Che doesn't support TCP socket connection. It also identifies to which LSs it should connect by searching for binaries of LSs in certain paths. And configuration of LS is hard coded into Java code, so an addition of new LS would require recompilation of wsagent which is bad UX. To fix that it would be better to pass LSs configs to wsagent, so it will know how to configure communication with an LS that will be used in a certain workspace.

[ghost]

@garagatyi can you elaborate on #1? What is the use case where an installer is needed? Can't we have the exact required environment in the image that LS will run in? I mean it is LS owner/author who's responsible for LS deps and runtime.

[benoitf]

I agree with @eivantsov
I would say that if we don't need installer for docker and openshift/k8s as usage of sidecar containers leverage the feature, then it's something (the bootstrap installer) that is specific to a hypothetical implementation so it could be removed from the core.

[garagatyi]

As @TylerJewell said in one of the discussions we can't remove installers unless we are 100% sure that sidecars will cover all the cases where installers can be used. So, for now, it is not the thing we should argue about.

[benoitf]

@garagatyi if it's for a different implementation and 'that we may not even need it at all' it means it should no longer be part of the core. And could come back only through a specific implementation.

[gazarenkov]

I guess I do not totally understand what kind of problem you're trying to solve @garagatyi
Could you please describe what did you do with details to let understand the real problems?

I do believe there is something wrong in the implementation if you need to analyze existence of installers.
The way of provisioning soft (recipe vs image vs installer...) should not matter AFAIK.
I am really confused with your comments...

[garagatyi]

I tried to move LS to a separate container. These are notes about issues I faced during my investigation in this sprint.
The existence of installers had not influenced what I did.

[TylerJewell]

I think what Gennady and Florent are saying is that agent installers and sidecar agents are separate concepts. They may or may not be used by the type of infrastructure chosen. So should we consider them orthogonal for architecture design?

[garagatyi]

I'm just referring to your phrase

The potential and promise of sidecars to remove 100% requirements for installing anything into a workspace runtime is a great aspiration. However, I think we should not quite "commit" as this being the only path for the future until we have prototypes that demonstrate all of our use cases and that performance will not suffer.

And we haven't prooved that sidecar is acceptable in any case yet.

[ghost]

All I wanted to know is the use case when we need an installer. Right now, we don't know in what environment LS is going to start. Thus, we try to auto detect OS, install dependencies and configure runtime if necessary. Maybe @garagatyi has a real life example based on his experience?

[garagatyi]

I don't know any particular use case, but I also haven't tried yet Docker PID namespaces for exec and terminal servers.

[gazarenkov]

@garagatyi said

I tried to move LS to a separate container. These are notes about issues I faced during my investigation in this sprint.
The existence of installers had not influenced what I did.

OK, then let's leave alone installers and focus on usecase you mentioned.
As I understand the only problem is that you could not find "right" LS which is able to work using remote protocol (not stdin/stdout only). Is that correct or there is something else preventing your POC?

[garagatyi]

It's not preventing my POC - I just added concerns on the usage of sidecars and issues I faced (it doesn't mean I can not solve some of them).
I can create an image with LS that works over TCP socket. But our code is not designed yet to use it. And I can rework the code to deal with TCP connection instead of input/output.

[benoitf]

@gazarenkov

OK, then let's leave alone installers

I think it should not be leave "alone" as it's part of the objectives : to not need it for Docker and OpenShift/K8s implementation. There is no usecase for having them for these implementations.

[gazarenkov]

@benoitf
Yes, installers are definitely out of scope since they do not block the case discussed (as Alex mentioned).
And, no, I do not consider them as useless if you will.

@garagatyi OK, thanks for explanations, so just configure your workspace with images/recipes/servers and volumes (LS on standalone container). Would it work?

[benoitf]

@gazarenkov the title of the issue is "Investigate how to run IDE tooling in separate containers", not being specific about LS, so installers are not out of the scope.
It's you that is trying to restrict the case to language server and try to move out of the scope installers.
the description is about "wsagent, terminal, exec and other servers", not about LS.

So for wsagent, terminal and exec, they have runtime dependencies that may require to install something new and it's not possible without sudo/root access
So, it means installer is useless for all these cases as if you try to use for example "ubuntu" image on OpenShift it will fail with installers. Then, why try to keep in the core something that can definitely not work by design.

As sidecar is working on docker and OS/K8s then installers are not out of the scope, they should not be there at all in the core feature. If someone need to have it in a specific implementation then it could be there but not as part of the default.

[gazarenkov]

@benoitf what I was saying is exactly the scope we discussed with @garagatyi
it is what I expect to see as a practical POC for "how to run IDE tooling in separate containers".

[benoitf]

@gazarenkov IMHO it's not because it's what you expect that your view is matching the full description issue.
Also POC was working on my document so it's only a matter of implementing it.

[skabashnyuk]

As a conclusion, I can say that the idea is potentially good. But due to described points #5623 (comment) that extend development, we do not consider it in close future.