Add contributing doc #112

nabuskey · 2023-12-13T00:15:01Z

Added some information on how to get started as a contributor and described internals.

Signed-off-by: Manabu Mccloskey <[email protected]>

CONTRIBUTING.md

nimakaviani · 2023-12-13T01:09:25Z

CONTRIBUTING.md

+    ```
+    ~/idpbuilder$ make build
+    test -s /home/ubuntu/idpbuilder/bin/controller-gen && /home/ubuntu/idpbuilder/bin/controller-gen --version | grep -q v0.12.0 || \
+    GOBIN=/home/ubuntu/idpbuilder/bin go install sigs.k8s.io/controller-tools/cmd/[email protected]
+    /home/ubuntu/idpbuilder/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./api/..." output:crd:artifacts:config=pkg/controllers/resources
+    /home/ubuntu/idpbuilder/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
+    go fmt ./...
+    go vet ./...
+    go build -o idpbuilder main.go  
+    ```


I think the indentation for triple backticks is off, no? should start from the beginning of the line

If it starts from the beginning, the code block doesn't line up well with bullet points.

CONTRIBUTING.md

nimakaviani · 2023-12-13T01:14:33Z

CONTRIBUTING.md

+This command creates the following resources on your machine. 
+
+1. [Kind](https://kind.sigs.k8s.io/) cluster.
+2. [ArgoCD](https://argo-cd.readthedocs.io/en/stable/) resources.
+3. [Gitea](https://about.gitea.com/) resources.
+4. [Crossplane](https://www.crossplane.io/) deployment. No providers.
+5. [Backstage](https://backstage.io/) resources.
+
+They are deployed as ArgoCD Applications with the Gitea repositories set as their sources. 
+
+UIs for Backstage, Gitea, and ArgoCD are accessible on the machine:
+* Gitea: http://gitea.cnoe.localtest.me:8443/explore/repos
+* Backstage: http://backstage.cnoe.localtest.me:8880/
+* ArgoCD: https://argocd.cnoe.localtest.me:8443/applications
+
+ArgoCD username is `admin` and the password can be obtained with 
+```
+kubectl -n argocd get secret argocd-initial-admin-secret -o go-template='{{ range $key, $value := .data }}{{ printf "%s: %s\n" $key ($value | base64decode) }}{{ end }}'
+```
+
+Gitea admin credentials can be obtained with 
+```
+kubectl get secrets -n gitea gitea-admin-secret -o go-template='{{ range $key, $value := .data }}{{ printf "%s: %s\n" $key ($value | base64decode) }}{{ end }}'
+```
+
+All ArgoCD applications should be synced and healthy. You can check them in the UI or 
+```
+kubectl get application -n argocd
+```
+
+## idpbuilder Overview
+
+idpbuilder is made of two phases: CLI and controllers.
+
+### CLI
+
+During this phase: 
+1. This is the phase where command flags are parsed and translated into relevant Go structs' fields. Most notably the [`LocalBuild`](https://github.com/cnoe-io/idpbuilder/blob/main/api/v1alpha1/localbuild_types.go) struct.
+2. Create a Kind cluster, then update the kubeconfig file.
+3. Once the kind cluster is started and relevant fields are populated, Kubernetes controllers are started:
+    *  `LocalbuildReconciler` responsible for bootstrapping the cluster with absolute necessary packages. Creates Custom Resources (CRs) and installs embedded manifests.
+    *  `RepositoryReconciler` responsible for creating and managing Gitea repository and repository contents.
+    *  `CustomPackageReconciler` responsible for managing custom packages.  
+4. They are all managed by a single Kubernetes controller manager.
+5. Once controllers are started, CRs corresponding to these controllers are created. For example for Backstage, it creates a GitRepository CR and ArgoCD Application.
+6. CLI then waits for these CRs to be ready.
+
+### Controllers
+
+During this phase, controllers act on CRs created by the CLI phase.
+
+#### LocalbuildReconciler
+
+`LocalbuildReconciler` bootstraps the cluster using embedded manifests. Embedded manifests are yaml files that are baked into the binary at compile time.
+1. Install Bootstrap apps. They are essential services that are needed for the user experiences we want to enable:
+    * Gitea. This is the in-cluster Git server that hosts Git repositories.
+    * Ingress-nginx. This is necessary to expose services inside the cluster to the users.
+    * ArgoCD. This is used as the packaging mechanism. Its primary purpose is to deploy manifests from gitea repositories. 
+2. Once they are installed, create `GitRepository` CRs for bootstrap apps.
+3. Create ArgoCD applications for the apps. Point them to the Gitea repositories. From here on, ArgoCD manages the bootstrap apps.
+
+Once bootstrap apps are installed, it creates the other embedded applications: Backstage and Crossplane.
+1. Create `GitRepository` CRs for the apps.
+2. Create ArgoCD applications for the apps.
+
+
+#### RepositoryReconciler
+
+`RepositoryReconciler` creates Gitea repositories.
+The content of the repositories can either be sourced from Embedded file system or local file system.
+
+#### CustomPackageReconciler
+
+`CustomPackageReconciler` parses the specified ArgoCD application files. If they specify repository URL with the scheme `cnoe://`,
+it creates `GitRepository` CR with source specified as local, then creates ArgoCD application with the repository URL replaced.
+
+For example, if an ArgoCD application is specified as the following.
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+spec:
+  source:
+    repoURL: cnoe://busybox
+```


This is more like the general README. I dont think this belongs to Contrib. this can be a higher level systems architecture doc as well.

for contributing.md, I mainly restrict it to how to build and how to test, with a reference to a separate file with the above content that describes the architecture of the idpbuilder.

I agree with @nimakaviani remark

cmoulliard · 2023-12-13T14:45:18Z

CONTRIBUTING.md

+To get started with the project on your machine, you need to install the following tools:
+1. Go 1.21+. See [this official guide](https://go.dev/doc/install) from Go authors.
+2. Make. You can install it through a package manager on your system. E.g. Install `build-essential` for Ubuntu systems.
+3. Docker. Similar to Make, you can install it through your package manager.


Suggestion: Add link to https://www.docker.com/products/docker-desktop/
Remark: We should also mention or reference podman

I didn't want to mention podman because we don't support it yet :(. I personally would love to support podman. Dcoker desktop kills my laptop battery.

cmoulliard · 2023-12-13T14:46:32Z

CONTRIBUTING.md

+Then change your current working directory to the repository root. e.g. `cd idpbuilder`.
+
+All subsequent commands described in this document assumes they are executed from the repository root.
+


There is a step which is missing: Docker desktop is running ;-)

It doesn't necessary need to be docker desktop. For example, my main development machine for this project is a headless Ubuntu EC2 instance.

CONTRIBUTING.md

cmoulliard · 2023-12-13T14:53:42Z

CONTRIBUTING.md

+4. [Crossplane](https://www.crossplane.io/) deployment. No providers.
+5. [Backstage](https://backstage.io/) resources.
+
+They are deployed as ArgoCD Applications with the Gitea repositories set as their sources. 


Remarks:

Only backstage and Crossplane packages are deployed as ArgoCD Applications !

We should explain in a few words what gitea do and what a developer will be able to do using it ;-)

CONTRIBUTING.md

cmoulliard · 2023-12-13T15:12:36Z

CONTRIBUTING.md

+2. Once they are installed, create `GitRepository` CRs for bootstrap apps.
+3. Create ArgoCD applications for the apps. Point them to the Gitea repositories. From here on, ArgoCD manages the bootstrap apps.
+
+Once bootstrap apps are installed, it creates the other embedded applications: Backstage and Crossplane.


I suggest "Once the core packages are installed and running, then the controller will deploy some additional packages: Backstage and Crossplane using GitRepository and Application CR resources

Remark: Ideally such additional packages should be installed using the GitOps way. To be reviewed and discussed as the real value of this tool/project will be to propose an approach for the platform engineers to define what they would like to provision as their IDP and deploy it:

For company A, they could be interested to install the following packages:

"Company A Backstage" implementation relying on: gitlab, jenkins, keycloak

Jenkins

Gitlab

Keycloak

For company B, that could be:

Backstage Community or cnoe or janus-idp having such dependencies: gitlab, keycloak, argocd, dev-spaces, tekton & certficiate manager

Keycloak

Certificate Manager

Tekton

Gitlab

DevSpaces

ArgoCD (= our core package ;-))

cmoulliard · 2023-12-13T15:13:33Z

CONTRIBUTING.md

+#### RepositoryReconciler
+
+`RepositoryReconciler` creates Gitea repositories.
+The content of the repositories can either be sourced from Embedded file system or local file system.


How can a user setup local file system (using --package-dir) ?

We have to add examples for this. In general we need a lot more examples.

CONTRIBUTING.md

cmoulliard · 2023-12-13T15:14:30Z

CONTRIBUTING.md

+This command creates the following resources on your machine. 
+
+1. [Kind](https://kind.sigs.k8s.io/) cluster.
+2. [ArgoCD](https://argo-cd.readthedocs.io/en/stable/) resources.
+3. [Gitea](https://about.gitea.com/) resources.
+4. [Crossplane](https://www.crossplane.io/) deployment. No providers.
+5. [Backstage](https://backstage.io/) resources.
+
+They are deployed as ArgoCD Applications with the Gitea repositories set as their sources. 
+
+UIs for Backstage, Gitea, and ArgoCD are accessible on the machine:
+* Gitea: http://gitea.cnoe.localtest.me:8443/explore/repos
+* Backstage: http://backstage.cnoe.localtest.me:8880/
+* ArgoCD: https://argocd.cnoe.localtest.me:8443/applications
+
+ArgoCD username is `admin` and the password can be obtained with 
+```
+kubectl -n argocd get secret argocd-initial-admin-secret -o go-template='{{ range $key, $value := .data }}{{ printf "%s: %s\n" $key ($value | base64decode) }}{{ end }}'
+```
+
+Gitea admin credentials can be obtained with 
+```
+kubectl get secrets -n gitea gitea-admin-secret -o go-template='{{ range $key, $value := .data }}{{ printf "%s: %s\n" $key ($value | base64decode) }}{{ end }}'
+```
+
+All ArgoCD applications should be synced and healthy. You can check them in the UI or 
+```
+kubectl get application -n argocd
+```
+
+## idpbuilder Overview
+
+idpbuilder is made of two phases: CLI and controllers.
+
+### CLI
+
+During this phase: 
+1. This is the phase where command flags are parsed and translated into relevant Go structs' fields. Most notably the [`LocalBuild`](https://github.com/cnoe-io/idpbuilder/blob/main/api/v1alpha1/localbuild_types.go) struct.
+2. Create a Kind cluster, then update the kubeconfig file.
+3. Once the kind cluster is started and relevant fields are populated, Kubernetes controllers are started:
+    *  `LocalbuildReconciler` responsible for bootstrapping the cluster with absolute necessary packages. Creates Custom Resources (CRs) and installs embedded manifests.
+    *  `RepositoryReconciler` responsible for creating and managing Gitea repository and repository contents.
+    *  `CustomPackageReconciler` responsible for managing custom packages.  
+4. They are all managed by a single Kubernetes controller manager.
+5. Once controllers are started, CRs corresponding to these controllers are created. For example for Backstage, it creates a GitRepository CR and ArgoCD Application.
+6. CLI then waits for these CRs to be ready.
+
+### Controllers
+
+During this phase, controllers act on CRs created by the CLI phase.
+
+#### LocalbuildReconciler
+
+`LocalbuildReconciler` bootstraps the cluster using embedded manifests. Embedded manifests are yaml files that are baked into the binary at compile time.
+1. Install Bootstrap apps. They are essential services that are needed for the user experiences we want to enable:
+    * Gitea. This is the in-cluster Git server that hosts Git repositories.
+    * Ingress-nginx. This is necessary to expose services inside the cluster to the users.
+    * ArgoCD. This is used as the packaging mechanism. Its primary purpose is to deploy manifests from gitea repositories. 
+2. Once they are installed, create `GitRepository` CRs for bootstrap apps.
+3. Create ArgoCD applications for the apps. Point them to the Gitea repositories. From here on, ArgoCD manages the bootstrap apps.
+
+Once bootstrap apps are installed, it creates the other embedded applications: Backstage and Crossplane.
+1. Create `GitRepository` CRs for the apps.
+2. Create ArgoCD applications for the apps.
+
+
+#### RepositoryReconciler
+
+`RepositoryReconciler` creates Gitea repositories.
+The content of the repositories can either be sourced from Embedded file system or local file system.
+
+#### CustomPackageReconciler
+
+`CustomPackageReconciler` parses the specified ArgoCD application files. If they specify repository URL with the scheme `cnoe://`,
+it creates `GitRepository` CR with source specified as local, then creates ArgoCD application with the repository URL replaced.
+
+For example, if an ArgoCD application is specified as the following.
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+spec:
+  source:
+    repoURL: cnoe://busybox
+```


I agree with @nimakaviani remark

Signed-off-by: Manabu Mccloskey <[email protected]>

nabuskey · 2023-12-13T20:53:40Z

Separated architecture information to readme file instead. I've also addressed most comments. Please take a look again when you get a chance.

nimakaviani

LGTM

add contributing doc

412542f

Signed-off-by: Manabu Mccloskey <[email protected]>

nimakaviani requested changes Dec 13, 2023

View reviewed changes

cmoulliard reviewed Dec 13, 2023

View reviewed changes

separate sections into readme and contributing

1966846

Signed-off-by: Manabu Mccloskey <[email protected]>

nabuskey force-pushed the doc/contrib branch from d416c33 to 1966846 Compare December 13, 2023 20:53

nimakaviani self-requested a review December 14, 2023 18:50

nimakaviani approved these changes Dec 14, 2023

View reviewed changes

nimakaviani merged commit 954b7bf into cnoe-io:main Dec 14, 2023
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add contributing doc #112

Add contributing doc #112

nabuskey commented Dec 13, 2023

nimakaviani Dec 13, 2023

nabuskey Dec 13, 2023

nimakaviani Dec 13, 2023

cmoulliard Dec 13, 2023

cmoulliard Dec 13, 2023

nabuskey Dec 13, 2023

cmoulliard Dec 13, 2023

nabuskey Dec 13, 2023

cmoulliard Dec 13, 2023

cmoulliard Dec 13, 2023

cmoulliard Dec 13, 2023

nabuskey Dec 13, 2023

cmoulliard Dec 13, 2023

nabuskey commented Dec 13, 2023

nimakaviani left a comment

		Then change your current working directory to the repository root. e.g. `cd idpbuilder`.

		All subsequent commands described in this document assumes they are executed from the repository root.

Add contributing doc #112

Add contributing doc #112

Conversation

nabuskey commented Dec 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nabuskey commented Dec 13, 2023

nimakaviani left a comment

Choose a reason for hiding this comment