pluggable pakcages

[nabuskey]

closes: #5

[nimakaviani]

Kapp doesnt need OCI images. Kapp + imgpkg allows to use OCI images.

[cmoulliard]

Is it the best approach to use Argocd Hooks to execute kubernetes jobs pre/post sync of resources vs using a real supply chain/pipeline tool as Tekton or Argo workflow ? A drawback to use hook is that they are executed for every resource sync and don't offer a kind a singleton hook which is only executed when a new application is installed.

[cmoulliard]

I would like also to mention that we should as much as we can use the argoCd pattern App of Apps - https://medium.com/dzerolabs/turbocharge-argocd-with-app-of-apps-pattern-and-kustomized-helm-ea4993190e7c
Using the ApplicationSet and their generators will also help us to customize the Applications CR generated using by example the version of the pluggable package to be installed like its Helkm chart, etc

[nimakaviani]

we can make Argo CD hooks run Argo workflows.

ideally actions taken as part of executing argo workflows will be no-ops in subsequent executions. but if that happens not to be the case, then maybe we consider triggering and running argo workflows separately. either way, I agree that Argo Workflows are better suited for doing imperative work.

[nabuskey]

Yep Application Sets are a great idea to use with the in-cluster git server, especially for reference implementation stuff.

[cmoulliard]

"we can make Argo CD hooks run Argo workflows."
There are some limitations using hooks = kubernetes jobs as ArgoCD delete them end like the pods of their execution. So, if users would like to view the logs, then this is impossible. If the job fails, then again it is also very difficult to access/vies the log of the newly pod created from a job as argocd kill/recreate them -every x seconds

[cmoulliard]

If we include jobs, how such a reconciliation will work as installing a package will require to of course let argoCd to install the resources within the target cluster but also that what we call jobs succeeded.

[cmoulliard]

Having such a mechanism is definitively a plus as today creating a kubernetes platform is really a pain :-(
Even VMWare Tanzu TAP don't propose such a great solution top of their kapp + ytt + catalog solution

[cmoulliard]

Remarks @nabuskey

• Title should be renamed to "Pluggable and configurable packages proposal"
• Proposal should also include a section "Alternatives" where we could also evaluate technology like to compose
  • KUDO: https://kudo.dev/docs/what-is-kudo.html#main-concepts
  • KRATIX: https://kratix.io/docs/main/value-of-kratix
  • etc

[greghaynes]

Some added support for this - our internal CI is a go CLI which depends on idpbuilder and implements exactly this :).

[jessesanford]

Should we consider requiring signed commits for our reference "packages" as argo cd apps so that the idpbuilder can be built to trust only official reference packages? https://argo-cd.readthedocs.io/en/release-1.8/user-guide/gpg-verification/ we can also allow users to configure their own gpg keys to use for verification as flag to idpbuilder create? https://argo-cd.readthedocs.io/en/release-1.8/user-guide/gpg-verification/#configure-using-declarative-setup

[nabuskey]

Should we consider requiring signed commits for our reference "packages" as argo cd apps so that the idpbuilder can be built to trust only official reference packages?

Did not know about this. We should 100% do this for our reference packages.

My current thinking around git server stuff:

1. I think embedded FS is quite useful for providing the reference implementation and possibly for use in air gapped environments. Great way to tie manifests to a idpbuilder release with no external dependencies or secrets to worry about.

2. As discussed in [suggestion] Use gitea as embedded git server #32, Gitea is a great idea. This allows us to pull stuff from a remote repository using user credentials, push it to the in-cluster git server. We will need to handle kustomize remote references and helm subcharts to ensure everything we need is pushed to the in-cluster git server.

[nabuskey]

Ok I've incorporated ideas posted above.

TLDR:

• Use gittea.
• Support syncing local files to git
• Replace repo urls with local repo urls in application spec.
• No support for imperative pipelining spec until concrete use cases arise.

[nabuskey]

About Git server contents:

Is mirroring content from remotes to in-cluster git server the right way to go? This becomes a bit more complicated once you start dealing with nested remotes. Especially for helm. We pretty much need to:

1. Get the root chart tgz
2. Extract, check dependencies.
3. If there are subcharts, pull them.
4. Repeat 1-3
5. Replace urls in dependencies for all subcharts. Tarball.
6. Push all to cluster.

This breaks signing on charts as well, but not sure if it matters for local.

If we are to render them, things are a bit simpler.

1. Get values.
2. Render. e.g. helm template
3. Push manifests to the cluster.
4. Update Argo App to use directory instead.

This means we have to make sure we are using the same process to render manifests as ArgoCD does. So more libraries, dependencies, and potentially deal with rendering logic. We are now pairing a specific version of ArgoCD to a release. We can't guarantee manifests are render the same if a different version of ArgoCd is deployed to the cluster than the one in use by idpbuilder

This also pretty much won't allow you to take advantage of FluxCD source mechanisms IF we decide to support Flux in the future.

Also note that to support ArgoCD app of Apps, we need to render locally. Because app of apps could be inside a Helm Chart, Kustomize, or plain manfiests.

We can also pass local credentials to the cluster but I am not sure if that's a good practice. I can't be sure which credentials should be passed. Passing all would not be a good idea imo.

[jessesanford]

About Git server contents:

Is mirroring content from remotes to in-cluster git server the right way to go? This becomes a bit more complicated once you start dealing with nested remotes. Especially for helm. We pretty much need to:

1. Get the root chart tgz
2. Extract, check dependencies.
3. If there are subcharts, pull them.
4. Repeat 1-3
5. Replace urls in dependencies for all subcharts. Tarball.
6. Push all to cluster.

This breaks signing on charts as well, but not sure if it matters for local.

If we are to render them, things are a bit simpler.

1. Get values.
2. Render. e.g. helm template
3. Push manifests to the cluster.
4. Update Argo App to use directory instead.

This means we have to make sure we are using the same process to render manifests as ArgoCD does. So more libraries, dependencies, and potentially deal with rendering logic. This also pretty much won't allow you to take advantage of FluxCD source mechanisms IF we decide to support Flux in the future.

Also note that to support ArgoCD app of Apps, we need to render locally. Because app of apps could be inside a Helm Chart, Kustomize, or plain manfiests.

We can also pass local credentials to the cluster but I am not sure if that's a good practice. I can't be sure which credentials should be passed. Passing all would not be a good idea imo.

I don't see any drawback to rendering and then pushing to local gitserver. Am I missing something? Is it simply that we would have a dependency on helm as a lib in idpbuilder?

[nabuskey]

I don't see any drawback to rendering and then pushing to local gitserver. Am I missing something? Is it simply that we would have a dependency on helm as a lib in idpbuilder?

Yeah pretty much. Dependencies on Helm and Kustomize libraries. We'll probably have to get the rendering logic from ArgoCD so this will add more dependencies.

Thinking about it. Another concern is that ArgoCD doesn't do rendering with this approach. So we have to make sure rendering logic is the same as ArgoCD. This could introduce a case where the library version of ArgoCD doesn't match the version of ArgoCD installed.

[greghaynes]

@nabuskey Would it make sense only to mirror contents which are referenced by a local path? In the case of an argo app which already has a remote it might be ok to just use that remote directly. The main win for the gitserver IMO is removing the dependency on having to push packages to some external mirror, but if the package is already pushed this shouldnt be an issue.

[jessesanford]

@nabuskey Would it make sense only to mirror contents which are referenced by a local path? In the case of an argo app which already has a remote it might be ok to just use that remote directly. The main win for the gitserver IMO is removing the dependency on having to push packages to some external mirror, but if the package is already pushed this shouldnt be an issue.

I agree with this, I think the only issue is how much we can keep the code paths between local and remote as unified as possible. I would hate to have to handle two separate "pipelines" of artifact herding when there is a design choice that could result in only 1 being necessary. However, based on @nabuskey comment about needing to match the rendering to argo's means that the tradeoff complexity might be more of a burden. I will take a back seat on this decision as you guys know this plumbing better than me. My only concern is that we have a great local workflow for developers that doesn't require pushing to a remote just to test changes during quick iteration.

[greghaynes]

One minor nit comments, LGTM

[greghaynes]

I wonder if we could expand on this - I think we should make ArgoCD a hard requirement of our installers (at least for the time being), but we should also maintain separation between packages and CD tooling. e.g. I'd expect there to be separate backstage packages (helm/kustomize) from the Argo apps which install them. With this we leave the door open to supporting CD alternatives in the future. If this sounds reasonable could we include a line about "we will strive to maintain separation between packages and specific CD technologies (argo) to allow for future CD plug-ability.

[nabuskey]

Sounds reasonable to me. For this implementation, we will hard require ArgoCD but there is no reason this cannot be changed in the future.

[cmoulliard]

4. Update Argo App to use directory instead.

I dont think that argocd supports accessing local directory as resources are managed by the argocd repo server (shards - redis).

[cmoulliard]

To extend the proposition of @nabuskey to use internally a git server (= gitea), I would like to mention these points too:

• Make the git server accessible from admin users externally (UI or SSH)
• Clone ALL the resources within the git server (core components: ingress, argocd & cert manager + pluggable). Why ? To let an admin user to change a resource when problem took place during deployment and to avoid to rebuild idpbuilder CLI, etc
• Refactor the localbuilds.idpbuilder.cnoe.io CRD to better describe a package to be installed.
• Component or Package CRD definition:
  • Name, labels
  • Source: URL, OCI image, bucket
  • Type: core or pluggable (we should find a better name)
  • Packaging type: helm, plain, kustomize
  • Config: = parameters
• Use a Package or Component CRD to deploy everything on the platform including core and pluggable
• Convert the Package or Component CRD to their GitOps equivalent. Why: to be able to support different implementations: argocd, fluxcd, Crossplane, Terraform etc. So our CRD could be translated for ArgocD to an Application CRD, for FluxCD to a GitRepository, etc

[nabuskey]

* Make the git server accessible from admin users externally (UI or SSH)

100% agreed. Being able to visualize and interact with git is valuable.

* Clone ALL the resources within the git server (core components: ingress, argocd & cert manager + pluggable). Why ? To let an admin user to change a resource when problem took place during deployment and to avoid to rebuild `idpbuilder CLI`, etc

This was my intention as well. We will need to make ArgoCD manage itself but should be easy. I will clarify that in the doc.

* Refactor the `localbuilds.idpbuilder.cnoe.io` CRD to better describe a package to be installed.

* Component or Package CRD definition:
  
  * Name, labels
  * Source: URL, OCI image, bucket
  * Type: core or pluggable (we should find a better name)
  * Packaging type: helm, plain, kustomize
  * Config: = parameters

* Use a Package or Component CRD to deploy everything on the platform including core and pluggable

* Convert the Package or Component CRD to their GitOps equivalent. Why: to be able to support different implementations: argocd, fluxcd, Crossplane, Terraform etc. So our CRD could be translated for ArgocD to an Application CRD, for FluxCD to a GitRepository, etc

I think these make sense as this project matures. I have a few concerns at this stage where we want to focus on use cases in local settings.

1. We can provide abstraction for these implementations but there must be a well defined limit. These technologies vary widely between how they are used and how they work. We have to put a lot more thoughts on these alone.

2. I also think this is getting a bit close to application modeling. Application modeling is hard. There has been attempts at modeling applications like OAM with Kubevela. As far as I am aware, there are no solutions that the industry is converging to. We need to be careful abut introducing another.

3. Dependencies and overheads. We will have to import a lot of dependencies and match rendering logic versions.

These are great discussion points to follow up after this PR. I think we should keep it simple and nail the local experience first. I think it's a low hanging fruit that we can immediately see its values on.