Expand support of devfile v2 in Che to include definition of Editors and Theia Plugins

[l0rd]

Is your enhancement related to a problem? Please describe.

Devfile plugins are not suited to specify che-theia plugins.

Describe the solution you'd like

Beyond recommendations in .vscode/extensions.json(covered by this issue) Che workspaces should support those configuration files in .che folder:

.che
  |--- che-theia-plugins.yaml
  |___ che-editor.yaml

And it should be possible to inline those files as annotations in a devfile (there is a separate issue for that):

schemaVersion: 2.0.0
attributes:
  che-editor.yaml:
    <yaml content>
  che-theia-plugins.yaml:
    <yaml content>

Additional context

Editors and theia plugins, can be specified in 3 different ways in a che-editor.yaml or che-theia.yaml file:

• by id: the definition is in the Che plugin registry and referenced by the id field (mandatory)
• by url: the definition is published at the URL specified in the reference field (mandatory)
• inline: the spec is provided inline

Che Theia Plugin Definition Syntax

A Che Theia plugin is specified using the file che-theia-plugins.yaml of the che-plugin-registry syntax.

Che Editor Definition Syntax

A Che editor is specified using the file che-editors.yaml syntax (but we should move che-editors to devfile v2 syntax)

Overriding fields

Overriding of a registry or web reference is allowed through json merge patch.

Examples

che-editor.yaml

# Contains the spec of just ONE editor. 
# Here I am providing 3 examples separted with ---
---
# EXAMPLE 1 - registry reference
id: eclipse/che-theia/nigthly         # mandatory
registryUrl: https://my-registry.com  # optional
override:                             # optional
  containers:
    - name: theia-ide
      memoryLimit: 1280Mi
---
# EXAMPLE 2 - web reference
reference: https://gist.github.com/.../che-editor.yaml # mandatory
override:                                              # optional
  containers:
      - name: theia-ide
        memoryLimit: 1280Mi 
---
# EXAMPLE 3 - inline
inline:        # <--- that's current syntax of https://github.com/eclipse/che-plugin-registry/blob/master/che-editors.yaml
               #      but we should use devfile v2 instead 
  endpoints:
    - name: "theia"
      public: true
      targetPort: 3100
      attributes:
        protocol: http
        type: ide
        secure: true
        cookiesAuthEnabled: true
        discoverable: false
    (...)
  containers:
    - name: theia-ide
      image: "quay.io/eclipse/che-theia:next"
      env:
        - name: THEIA_PLUGINS
          value: local-dir:///plugins
      volumeMounts:
        - name: plugins
          path: "/plugins"
        - name: theia-local
          path: "/home/theia/.theia"
      mountSources: true
      ports:
        - exposedPort: 3100
      memoryLimit: "512M"
      cpuLimit: 1000m
      cpuRequest: 100m
  initContainers:
    - name: remote-runtime-injector
      image: "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
      volumeMounts:
        - name: remote-endpoint
          path: "/remote-endpoint"
          ephemeral: true
      env:
        - name: PLUGIN_REMOTE_ENDPOINT_EXECUTABLE
          value: /remote-endpoint/plugin-remote-endpoint
        - name: REMOTE_ENDPOINT_VOLUME_NAME
          value: remote-endpoint

che-theia-plugins.yaml

- id: redhat/java8/latest               # mandatory 
  registryUrl: https://my-registry.com  # optional 
  override:                             # optional 
    sidecar:
      memoryLimit: 1280Mi
- reference: https://gist.github.com/.../che-theia-plugin.yaml # mandatory 
  override:                                                    # optional 
    sidecar:
      memoryLimit: 1280Mi 
- inline:       # <--- same syntax as https://github.com/eclipse/che-plugin-registry/blob/master/che-theia-plugins.yaml
    sidecar:
      image: 'quay.io/eclipse/che-plugin-sidecar:go-c939ba4'
      name: vscode-go
      memoryLimit: 512Mi
      env:
        - name: GOPATH
           value: '/go:$(CHE_PROJECTS_ROOT)'
      extensions:
        - 'https://github.com/golang/vscode-go/releases/download/v0.16.1/go-0.16.1.vsix'

Subtasks

• Provides high-level library for managing DevWorkspace objects for clients
• Allow resolver API to resolve other files than [.]devfile.yaml
• Generate fragment / che-theia-plugin.yaml for each plug-in in the registry
• Dashboard should resolve editor and plugins from repo during factory flow Dashboard should resolve editor and plugins from repo during factory flow #19927
• Support of definitions in .che/*.yaml files
  • Support for .che/che-theia-plugins.yaml files
    • che server
    • dashboard
    • che-devworkspace-client
  • Support for .che/che-editor.yaml file
    • che server
    • dashboard
    • che-devworkspace-client
• Inline definition support (issue):
  • Support for che-theia-plugins.yaml devfile attribute
    • dashboard
    • che-devworkspace-client
  • Support for che-editor.yaml devfile attribute
    • dashboard
    • che-devworkspace-client
• Adapt QE tests

[sleshchenko]

Is che/che-theia-plugins.yaml supposed to be just a recommendation as well?
Like the following use-case vscode/extensions.json:

{
   "recommendations": [
       "redhat.java"
   ]
}

.che/che-theia-plugins.yaml

recommendations:
- id: redhat/java8/latest
  sidecar:
    memoryLimit: 1280Mi

Theia detects both files and proposes a user to:

• to enable redhat java
  • the recommended configuration(1280);
  • or default (that came from registry);
• or procced without recommended extensions.

[l0rd]

Yes, from Che-Theia point of view it's just a recommendation.

We didn't thought about specifying explicitly the recommendations section in .che/che-theia-plugins.yaml but it may make sense. It makes things clear. On the other hand the file is more verbose and less consistent with che-theia-plugins.yaml in the plugin registry. I am ok with both approaches though. cc @benoitf

Anyway, even if Che-Theia should consider those as recommendations only, Che can and should be smarter than that: when only a git URL is provided Che can look for a devfile.yaml, for che-theia-plugins.yaml (if Che-Theia is the editor) and for extensions.json (if the editor supports vscode extensions) and "resolve" the corresponding components before creating the DevWorkspace CR. This "devfile resolver" is an optimization and is not part of this issue.

[tsmaeder]

If the user can override the recommendations (aka remove and add plugins she wants), where would this state be saved? If it's not in the devfile, the workspace is no longer recreatable nor shareable.

[benoitf]

@tsmaeder it would be stored in .che/che-theia-plugins.yaml

and one goal/main usecase of devworkspace' is that you share a repository that contains a devfile, not a devfile referencing a repository.
so if you share a link to your repository/fork/branch where you've a customization in a .che folder, then it would be shareable.

[azatsarynnyy]

Isn't it the same or a sub-task of #18274?
If not, I don't see how it differs.

[l0rd]

I have updated the description and make it clear how this issue is different from the che-theia plugins recommendations one. I have included both as subtask of #18669

[amisevsk]

I was under the impression that the che-theia-plugins.yaml and che-editors.yaml formats were implemented as tools to help generate meta.yamls (and didn't know about the design process therein until it was merged). As tools to help generating a more complete spec, they make sense, but this proposal leaves a gap between the short-form of e.g. che-theia-plugins and the ultimate outcome, which is

• For devfile v1, the format needs to be converted to a ChePlugin as understood by the Che server (this is done by the plugin broker)
• For devfile v2, the format needs to be converted to a DevWorkspaceTemplate, as understood by the DevWorkspace Operator.

The che-editors.yaml definition is already very similar to the DevWorkspaceTemplate format, so it seems strange to me to introduce yet another API in between the definition and implementation.

[ericwill]

I was under the impression that the che-theia-plugins.yaml and che-editors.yaml formats were implemented as tools to help generate meta.yamls (and didn't know about the design process therein until it was merged). As tools to help generating a more complete spec, they make sense, but this proposal leaves a gap between the short-form of e.g. che-theia-plugins and the ultimate outcome, which is

* For devfile v1, the format needs to be converted to a `ChePlugin` as understood by the Che server (this is done by the plugin broker)

* For devfile v2, the format needs to be converted to a `DevWorkspaceTemplate`, as understood by the DevWorkspace Operator.

My understanding is that there will be a translation system (like the plugin broker) that will convert the che-theia-plugins.yaml format into a DevWorkspaceTemplate. The benefit there is that users only ever have to worry about one format -- the che-theia-plugins.yaml format. If there are improvements to that format we should probably discuss it sometime soon, though.

The che-editors.yaml definition is already very similar to the DevWorkspaceTemplate format, so it seems strange to me to introduce yet another API in between the definition and implementation.

If there are no feature gaps, we should probably use the DevWorkspaceTemplate format for the che-editors.yaml file.

[l0rd]

If there are no feature gaps, we should probably use the DevWorkspaceTemplate format for the che-editors.yaml file.

I have created an issue for that #19034

[nickboldt]

This issue appears to be done except for

• Adapt QE tests

When will that be completed?

[sleshchenko]

^ @dmytro-ndp WDYT?

[l0rd]

I have created a dedicated issue for the remaining subtask #20385 and included it in STEP 3 milestone.

I am closing this issue and including it back to STEP2 milestone.