Ability for Codewind CLI to run extra commands when creating project

[makandre]

Codewind version:
OS:

Che version:
IDE extension version:
IDE version:
Kubernetes cluster:

Description of the enhancement:

When the CLI is used to create a new project, it would be good if it can run extra commands on the project depending on the type.

A use case for this is for appsody support. For appsody projects, we need to run the appsody init command to setup the local development environment for the project (e.g. copying a parent pom into the maven cache).

Proposed solution:

In the appsody extension's codewind.yaml, there's a commands array which is not currently used: https://github.com/eclipse/codewind-appsody-extension/blob/master/codewind.yaml#L5

We could defined a type of command there that is meant to run at certain point in the project creation workflow (to keep things generic and flexible for other extensions)

For example, something like this:

commands:
  - name: postProjectValidate
    command: appsody init

So when Codewind CLI sees that, it runs the command after the project is validated.

Of course, this would require for the appsody CLI to be available on system's path. I will open a separate issue (#271) to track how we would ensure that.

Edit: used to be postProjectBind but just found out bind is not done by the CLI

[makandre]

As discussed with @tobespc. Please tag as appropriate.

[makandre]

We would need appsody init to run after project bind validation to ensure the experience
is consistent for the project import workflow as well

[makandre]

Note that because of requirements coming up from #292, we likely need to define 2 ways to run appsody init:

1. For creating a project from a template (where .appsody-config.yaml exists), we would run appsody init without arguments

2. For binding a vanilla project as an appsody project, we would run appsody init <stack> --no-template

[mattcolegate]

So currently hardcoding seems to be the way forward to get it done quickly for release.
I put some effort in yesterday thinking about how to architect this, and I came up with the following:

Extension

Each extension has an identifier (currently name), a detection file or path that would be in the project root that uniquely identifies projects that would use this extension, and an array of Commands as outlined above, using names that would be documented in the Codewind Docs.

Projects

Each project would store an extendedBy array of extension identifiers in it's .inf file

Functions

Each function (such as bind, create, stop etc) is assigned a hard-coded extensionPoint (e.g. onProjectBind). Probably have to have pre- and post- versions too. These extension points would be externally documented

How it works

Discovery

• On every project create or import, during verification the list of installed extensions is iterated over, using the detection path to see if the project should be extended by that extension. If it is, add the extension identifier to the extendedBy field
• On every extension installation, the known list of projects is gone through, testing if they qualify to be extended by the new extension. If it is, add the extension identifier to the extendedBy field
• On every extension removal, the known list of projects is gone through and that extension identifier is removed from the extendedBy field.

Function extension

Every function that has an extensionPoint will, at the correct stage (pre, post or during) see if the project it is operating on has anything in it's extendedBy field. If it does, the function looks up the relevant extension and checks the Commands array for it's extensionPoint. If it finds it, it will exec the command contained therein.

This design has the plus point of keeping the commands in the Extension, thereby allowing new extensions to automatically be incorporated or supplanted without keeping a close eye on the contents of every individual project.inf.

The major problem of implementing this at the moment is of course that everything is hardcoded for Appsody already, and the decision has already been made to jam the entire extension yaml into the project.inf. I would very much like to know why that decision was made - certainly I'm open to persuasion as to why that is the correct way of doing it!

Are we also expecting these systems to operate in tandem with the current system or will we be supplanting it?

Do we have an existing list of extension points that stakeholders require?

[makandre]

Re jamming the extension yaml into project.inf. While I don't know the decision for why that is done, it does mean that projects returned by GET /api/v1/projects will have the extension metadata, some of which is used by IDEs (e.g. https://github.com/eclipse/codewind-appsody-extension/blob/master/codewind.yaml#L8)

At the very least that saves the IDE from making an extra API call to query the extension metadata

[mattcolegate]

That's a good use case - I was thinking that querying the extension might provide better service in terms of ease of updating extension versions (which might have extra command structures, so every existing project will need to have it's project.inf updated with the new commands). I wonder if that would be something the current API call could be altered to provide, rather than maintaining (hopefully) identical sets of data in two different files.

Do you know where in the current process the extension.yaml is inserted into the project.inf?

So for this work should we squarely focus on the use case in hand (calling appsody init) in a generic way by just having the one extension point (`onProjectBind')? When during the bind process would be the best time to make the call?

[makandre]

Perhaps it is here: https://github.com/eclipse/codewind/blob/master/src/pfe/portal/routes/projects/binding.route.js#L107

I agree that rather than writing a copy of the extension metadata into project.inf, it's better that at the GET /api/v1/projects API, the extension metadata is appended to the projects in the response.

I was thinking the call to appsody init happens after project is bound because at that point you will have the extension metadata loaded and know what command to call.

[mattcolegate]

I'm struggling to think of a way to include #292 in this - any ideas I have rely on knowledge of the project.inf file, and I don't think we want extension providers to necessarily know the inner workings of that in case it gets corrupted.

[makandre]

Couldn't we just define 2 commands? For appsody for example, extension would define:

commands:
  - name: postProjectValidate
    command: "appsody init"
  - name: postVanillaProjectValidate (better name TBD)
    command: "appsody init <stack> --no-template"

I mentioned in #292 (comment) that we would need to pass the <stack> name to bind validate command for the second case; but then it's just a matter of Codewind CLI calling one of the commands depending on whether <stack> was passed in.

I'm not sure why the extension would need to know about project.inf file.

That said, I discussed with @sghung and given the timeline we don't want to rush #292 in so I think hardcoding the call to appsody init is fine for this release. But for #292 after this, we would want to have a proper extensible way to define the commands.

[mattcolegate]

We could do it like that for sure, but then it stops being generic because we're adding custom extension points solely for Appsody. We need a way to tell the two different types of Appsody project appart. Perhaps modify the codewind.yaml of the extension to be able to define more than one projectType where each type has it's own set of commands, detection and config?

I was thinking we'd store some kind of templatePath in the project.inf which could be used to determine if the project was created via a template.

[makandre]

It's not really a custom extension point for Appsody. The upcoming ODO extension would make use of the postVanillaProjectValidate command as well.

They would define something like:

commands:
  - name: postVanillaProjectValidate
    command: "odo create <component_type>`

(they only support the second scenario)

cc: @jingfuwang @ssh24

[ssh24]

Similar to Appsody projects, a vanilla project can be binded as an ODO project in codewind. Once a vanilla project is binded it creates an odo config file. The command Andrew provided above will be similar to what we are designing it as well.

[GeekArthur]

On every project create or import, during verification the list of installed extensions is iterated over, using the detection path to see if the project should be extended by that extension. If it is, add the extension identifier to the extendedBy field

For my understanding extendedBy field is an array of extension identifiers in .inf file for each project and its functionality is to tell us which extension is available to use for the project. Given that, I think only install & uninstall extension can modify extendedBy field, then during the project bind, if we find the detection path we don't need to add the extension identifier to the extendedBy field, just directly call the corresponding commands.

Also can you please define the detection path? I assume it's either .odo (odo case) or .appsody-config.yaml (appsody case) under project directory.
Difference between odo and appsody here is: appsody runs appsody init if .appsody-config.yaml exists, odo doesn't need to do anything if .odo exists

We could do it like that for sure, but then it stops being generic because we're adding custom extension points solely for Appsody. We need a way to tell the two different types of Appsody project appart. Perhaps modify the codewind.yaml of the extension to be able to define more than one projectType where each type has it's own set of commands, detection and config?

I think if we already had appsody cli and odo cli in the system's path, we can get the generic command from codewind.yaml then get the component/stack to form the complete command using appsody/odo cli
eg.
For odo, uses odo catalog list to get the available components then run odo create <component>
For appsody, uses appsody list to get the available stacks then run appsody init <stack> --no-template

[tobespc]

So lets just quickly go back over how the codewind extension mechanism is intended to work. Each extension can define a list of commands that can be run at certain steps in the development process. For example, you could define a'build' step and supply the command you wish to run. What would then happen is during each development process in codewind, we would look to see if any registered extensions were available for the project type that were registered against that step and then execute that command.

For appsody init, We should be adding a 'postcreate' step to the extension that we would then call during creation. That is where we would want the 'appsody init' step to defined

[makandre]

Updated some previous comments to change bind to validate; since just found out bind is still a REST API call (runs in container), but as part of the process of binding a project to Codewind, validate will be called which will be a CLI command (runs out of container).

Running extension commands post-validate is fine; it should cover what needed for appsody and ODO.

[GeekArthur]

I just realize that appsody has the mount issue for running appsody init inside container, and that’s the root reason why appsody needs run CLI on the host OS, odo doesn’t have the limitation. Given that and also it’s a lot pain to support multiple OS if run CLI on the host OS, I think odo can only run everything inside PFE container. So I think after 0.3.1 appsody and odo may work on slightly different path because odo doesn’t need the CLI.

[sghung]

Andrew came up with a more generic way to run commands that has a scope beyond project creation. Closing for now as the work required for 0.4.0 is done