AppStudio Infrastructure Deployments

This repository is an initial set of Argo-CD-based deployments of AppStudio components to a cluster, plus a script to bootstrap Argo CD onto that cluster (to drive these Argo-CD-based deployments, via OpenShift GitOps).

This repository is structured as a GitOps monorepo (e.g. the repository contains the K8s resources for multiple applications), using Kustomize.

The contents of this repository are not owned by any single individual, and should instead be collectively managed and maintained through PRs by individual teams.

How to add your own component

You may use the gitops component as an example for how to add your own component. Here gitops refers to the GitOps service team's K8s resources.

These are the steps to add your own component:

1. Create a new directory for your team's components, under components/(team-name).
2. Add a kustomization.yaml file under that directory, which points to the individual K8s YAML resources you wish to deploy.
   • You may also structure your deployment into directories and files. See the Kustomize documentation for more information, and/or examples below.
   • See components/gitops/backend for an example of this.
3. Create an Argo CD Application resource in argo-cd-apps/base/(team-name).yaml).
   • See gitops.yaml for a template of how this should look.
   • The .spec.source.path value should point to the directory you created in previous step.
   • The .spec.destination.namespace should match the target namespace you wish to deploy your resources to.
   • The .metadata.name should correspond to your (team-name)
4. Add a reference to your new (team-name).yaml file, to argo-cd-apps/base/kustomization.yaml (the reference to your YAML file should be in the resources: list field).
5. Run kustomize build (repo root)/argo-cd-apps/overlays/staging and ensure it passes, and outputs your new Argo CD Application CR.
6. Open a PR for all of the above.

More examples of using Kustomize to drive deployments using GitOps can be found here.

Maintaining your components

Simply update the files under components/(team-name), and open a PR with the changes.

TIP: For development purposes, you can use kustomize build . to output the K8s resources that are being generated for your folder.

Bootstrapping a cluster

Required prerequisites

The prerequisites are:

• You must have kubectl and kustomize installed.
• You must have kubectl pointing to an existing OpenShift cluster, that you wish to deploy to.

Optional: CodeReady Containers Setup

If you don't already have a test OpenShift cluster available, CodeReady Containers is a popular option. It runs a small OpenShift cluster in a single VM on your local workstation.

1. Create or log in using your free Red Hat account, and install CodeReady Containers (CRC).
2. Make sure you have the latest version of CRC: crc version
3. Set up your workstation and command line tools: crc setup
4. Configure the VM using the minimum supported values. You can further increase these values if your workstation can support it: crc config set memory 16384 and crc config set cpus 6
5. Create a new VM after you adjust the memory and cpu allocation: crc delete and confirm with a y.
6. Start the OpenShift cluster: crc start This command will output the OpenShift web console URL as well as the developer and kubeadmin credentials when it's finished.
7. Set up your command line: eval $(crc oc-env)
8. Configure kubectl to use the CRC administrator account: kubectl config use-context crc-admin

Bootstrap App Studio

Steps:

1. Run ./hack/bootstrap-cluster.sh which will bootstrap Argo CD (using OpenShift GitOps) and setup the Argo CD Application Custom Resources (CRs) for each component. This command will output the Argo CD Web UI route when it's finished.
2. Open the Argo CD Web UI to see the status of your deployments. You can use the route from the previous step and login using your OpenShift credentials (using the 'Login with OpenShift' button), or login to the OpenShift Console and navigate to Argo CD using the OpenShift Gitops menu in the Applications pulldown.
3. If your deployment was successful, you should see several applications running, such as "all-components-staging", "gitops", and so on.

Optional: CodeReady Containers Post-Bootstrap Configuration

Even with 6 CPU cores, you will need to reduce the CPU resource requests for each App Studio application. Using kubectl edit argocd/openshift-gitops -n openshift-gitops, reduce the resources.requests.cpu values from 250m to 100m or less. More details are in the FAQ below.

Development mode for your own clusters

Once you bootstrap a cluster above, the root ArgoCD Application and all of the component applications will each point to the upstream repository.

To enable development for a team or individual to test changes on your own cluster, you need to replace the references to https://github.com/redhat-appstudio/infra-deployments.git with references to your own fork.

There are a set of scripts that help with this, and minimize the changes needed in your forks.

There is a development configuration in overlays/development which includes a kustomize overlay that can redirect the default components individual repositorys to your fork. The script also supports branches automatically. If you work in a checked out branch, each of the components in the overlays will mapped to that branch by setting targetRevision:.

Steps:

1. in your forked repository run ./hack/development-mode.sh and this will update the root application on the cluster and all of the git repo references in argo-cd-apps/overlays/development/repo-overlay.yaml

2. you will need to push the updated references in argo-cd-apps/overlays/development/repo-overlay.yaml to your fork. Argo will now sync all the changes from your fork into the cluster

3. You can now make changes to your forked repository and test them via the gitops

4. To submit changes back to the upstream make sure you do not include the modified file argo-cd-apps/overlays/development/repo-overlay.yaml.

One option to prevent accidentally including this modified file, you can run the script ./hack/upstream-mode.sh to reset everything including your cluster to https://github.com/redhat-appstudio/infra-deployments.git and match the upstream config. You can also checkout the current upstream git fetch upstream; git checkout upstream/main -- argo-cd-apps/overlays/development/repo-overlay.yaml to ensure you have the original file.

After you commit your changes you can rerun to ./hack/development-mode.sh and reset your repo to point back to the fork.

Note running these scripts in a clone repo will have no effect as the repo will remain https://github.com/redhat-appstudio/infra-deployments.git

App Studio Build System

The App Studio Build System is composed of the following components:

1. OpenShift Pipelines.
2. AppStudio-specific Pipeline Definitions in build-templates for building images.
3. AppStudio-specific ClusterTasks.

This repository installs all the components and includes a set of example scripts that simplify usage and provide examples of a working system. There are no additiona components needed to use the build system API, however some utilities and scripts are provided to demonstrate functionality.

Quickstart

To try out a pre-configured, follow these steps.

Steps
1. Create project for your pipelines execution. This can be run as any non-admin user (or admin) and is needed to hold your execution pipelines.	oc new-project demo
2. Run build-deploy example with a quarkus app.	./hack/build/build-deploy.sh https://github.com/devfile-samples/devfile-sample-code-with-quarkus
3. View your build on the OpenShift Console under the pipelines page or view the logs via CLI.	./hack/build/ls-builds.sh or tkn.exe pipelinerun logs

Usage

A sample script build.sh is provided which uses the App Studio Build Service API to demonstrate launching a build and inspecting the results. As a proof-of-concept, an optional build-deploy.sh script is included to take the build image and run it. .

./hack/build/build.sh  git-repo-url <optional-pipeline-name>

also the equivalent build but with an associated deploy.

./hack/build/build-deploy.sh  git-repo-url <optional-pipeline-name>

The git-repo-url is the git repository with your source code. The <optional-pipeline-name> is the name of one of the pipelines documented in the App Studio API Contract . This pipeline name can be provide when the automatic build type detection does not find a supported build type. Note: Normally the build type would be done automatically by (by the Component Detection Query) which maps devfile or other markers to a type of build needed. The build currently uses a shim repo-to-pipeline.sh to map file markers to a pipeline type. For testing and experiments the optional-pipeline-name parameter can override the default pipeline name.

The current build types supported are: devfile-build, docker-build, java-buider and node-js-builder.

For a quick "do nothing pipeline" run you can specify the noop buider and have a quick pipeline run that does nothing except print some logs.

./hack/build/build.sh https://github.com/jduimovich/single-container-app noop

Pipelines will be automatically installed when running a build via an OCI bundle mechanism.

To see what builds you have run, use the following examples.

Use ./hack/build/ls-builds.sh to show all builds in the system, and ./hack/build/ls-builds.sh <build-name> to get the stats for a specific build.

Testing

To validate the pipelines are installed and working, you can run ./hack/build/m2-builds script which will build all the samples planned for milestone 2.

To deploy all the builds as they complete, add the -deploy option.

./hack/build/m2-builds -deploy

You can also run the noop build ./hack/build/quick-noop-build.sh, that executes in couple seconds to validate a working install.

Other Build utilties

The build type is identified via temporary hack until the Component Detection Query is available which maps files in your git repo to known build types. See ./hack/build/repo-to-pipeline.sh which will print the repo name and computed builder type.

The system will fill with builds and logs so a utility is provided to prune pipelines and cleanup the associated storage. This is for dev mode only and will be done autatically by App Studio builds. Use ./hack/build/prune-builds.sh for a single cleanup pass, and ./hack/build/prune-builds-loop.sh to run a continuous loop to cleanup extra resources.

Use ./hack/build/utils/check-repo.sh to test your what auto-detect build will return.

./hack/build/utils/check-repo.sh  https://github.com/jduimovich/single-java-app
https://github.com/jduimovich/single-java-app   -> java-builder
  

If you want to check all your repos to see which ones may build you can use this script. You need to set you github id export MY_GITHUB_USER=your-username and it will test your repo for buildable content.

./hack/build/utils/ls-all-my-repos.sh | xargs -n 1 ./hack/build/utils/check-repo.sh

FAQ

Other questions? Ask on #wg-developer-appstudio.

Q: How do I deliver K8s resources in stages? For example, installing a Custom Resource Definition (CRD) first, then installing the Custom Resource (CR) for that CRD.

As long as your resources are declaratively defined, they will eventually be reconciled with the cluster (it just may take Argo CD a few retries). For example, the CRs might fail before the CRDs are applied, but on retry the CRDs will now exist (as they were applied during the previous retry). So now those CRs can progress.

However, this is not true if you are installing an Operator (e.g. Tekton) via OLM Subscription, and then using an operand of that operator (e.g. Pipeline CRs), at the same time. In this case, you will need to add the argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true annotation to the operands (or add it to the parent kustomization.yaml).

See the FAQ question 'the server could not find the requested resource' below for details.

For finer-grained control of resource apply ordering, use Argo CD Sync waves (Here is an example).

Q: What if I want my service's K8s resources in a separate Git repository? (i.e. not this one)

Ultimately, as a team, we should decide on a resource deployment strategy going forward: whether we want every team's K8s resources to be defined within this repository (as a GitOps monorepo), or within individual team's Git repositories. IMHO it is easiest to coordinate deployments within a single Git repository (such as this one), rather than multiple independent repositories.

However, if one or more services want to split off their K8s resource into independent repositories owned by those teams, they can modify the Argo CD Application CR for their service (created in 'How to add your own component' step 3, above) to point to their new repository.

Q: How can I install an Operator to the cluster using Argo CD?

To install an operator, you only need to include the OLM Subscription and OperatorGroup CRs for the operator under your deployed resources folder within this repository. If the operator is not available in OperatorHub, then you need to include also the OLM CatalogSource CR.

For an example of this, see the Red Hat CoP GitOps catalog, for example the Web Terminal operator example.

Q: When Argo CD attempts to synchronize my Argo CD Application, I am seeing 'the server could not find the requested resource' sync error on my custom resources. How can I fix this?

Before Argo CD attempts a synchronize operation (syncing your Git repository with the K8s cluster), it performs a dry-run to ensure that all the K8s resources in your Git repository are valid. If your repository contains custom resources which are not yet defined (for example, Tekton Pipeline CRs), it will refuse to begin the synchronize operation.

This most often occurs when a Git repository contains both the OLM Subscription (which will install the desired operator, e.g. Tekton), and also the operands of that operator (the Pipeline CRs).

The easiest way to solve this is to add this annotation to your custom resources operands: argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true.

For example, we would add this annotation to all of our Pipeline CRs:

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  # Add this annotation to your CRs:
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true 

If you are using Kustomize, you can place the following in your kustomization.yaml file to automatically add it to all resources:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- # (your resources)

# Add these lines to your kustomization.yaml:
commonAnnotations: 
  argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

See the Argo CD docs for more on this sync option. See the redhat-cop/gitops-catalog for examples of this option used with Kustomize and OLM-installed operators.

Q: What is the recommended memory and CPU allocation for CodeReady Containers for development purposes?

We recommend 7+ cores and 24+ GiB (24576 MiB) of memory.

Q: When using CodeReady Containers for development purposes, I am getting an error message similar to: 0/1 nodes available: insufficient memory.

The default worker node memory allocation of 8192 MiB insufficient to run App Studio. Increase the memory to 16 MiB using crc config set memory 16384 and then create a new CRC VM to apply your changes, using crc delete and crc start. Finally, repeat the cluster bootstrapping process.

See the CodeReady Containers docs for more on this configuration option.

Q: When using CodeReady Containers for development purposes, I am getting an error message similar to: 0/1 nodes available: insufficient cpu.

The default 4-CPU allocation will not be sufficient for the CPU resource requests in this repo. Increase number of cores, for example, crc config set cpus 6 if your hardware supports it, and then create a new CRC VM to apply your changes, using crc delete and crc start. Finally, repeat the cluster bootstrapping process.

See the CodeReady Containers docs for more on this configuration option.

Even with 6 CPU cores, you will need to reduce the CPU resource requests for each App Studio application. Using kubectl edit argocd/openshift-gitops -n openshift-gitops, reduce the resources.requests.cpu values from 250m to 100m or less. For example, change each line with

requests:
    cpu: 250m

to

requests:
    cpu: 100m

Then save and exit the editor. The updates will be applied to the cluster immediately, and the App Studio deployment should complete within a few minutes.