This repository has been archived by the owner on Sep 30, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
update campaigns docs to reflect new flow (#11972)
- Loading branch information
Showing
19 changed files
with
669 additions
and
660 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,126 @@ | ||
| # Campaigns design doc | ||
|
|
||
| Why are [campaigns](../user/campaigns/index.md) designed the way they are? | ||
|
|
||
| ## Principles | ||
|
|
||
| - **Declarative API** (not imperative). You declare your intent, such as "lint files in all repositories with a `package.json` file"<!-- TODO(sqs): thorsten had a suggestion to make this quote use non-imperative language but I can't find the comment -->. The campaign figures out how to achieve your desired state. The external state (of repositories, changesets, code hosts, access tokens, etc.) can change at any time, and temporary errors frequently occur when reading and writing to code hosts. These factors would make an imperative API very cumbersome because each API client would need to handle the complexity of the distributed system. | ||
| - **Define a campaign in a file** (not some online API). The source of truth of a campaign's definition is a file that can be stored in version control, reviewed in code review, and re-applied by CI. This is in the same spirit as IaaC (infrastructure as code; e.g., storing your Terraform/Kubernetes/etc. files in Git). We prefer this approach over a (worse) alternative where you define a campaign in a UI with a bunch of text fields, checkboxes, buttons, etc., and need to write a custom API client to import/export the campaign definition. | ||
| - **Shareable and portable.** You can share your campaign specs, and it's easy for other people to use them. A campaign spec expresses an intent that's high-level enough to (usually) not be specific to your own particular repositories. You declare and inject configuration and secrets to customize it instead of hard-coding those values. | ||
| - **Large-scale.** You can run campaigns across 10,000s of repositories. It might take a while to compute and push everything, and the current implementation might cap out lower than that, but the fundamental design scales well. | ||
| - **Accommodates a variety of code hosts and review/merge processes.** Specifically, we don't to limit campaigns to only working for GitHub pull requests. (See [current support list](../user/campaigns/index.md#supported-code-hosts-and-changeset-types).) | ||
|
|
||
| ## Comparison to other distributed systems | ||
|
|
||
| Kubernetes is a distributed system with an API that many people are familiar with. Campaigns is also a distributed system. All APIs for distributed systems need to handle a similar set of concerns around robustness, consistency, etc. Here's a comparison showing how these concerns are handled for a Kubernetes [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) and a Sourcegraph campaign. In some cases, we've found Kubernetes to be a good source of inspiration for the campaigns API, but resembling Kubernetes is **not** an explicit goal. | ||
|
|
||
| <table> | ||
| <tr> | ||
| <td/> | ||
| <th>Kubernetes <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">Deployment</a></th> | ||
| <th>Sourcegraph campaign</th> | ||
| </tr> | ||
| <tr> | ||
| <th>What underlying thing does this API manage?</th> | ||
| <td>Pods running on many (possibly unreliable) nodes</td> | ||
| <td>Branches and changesets on many repositories that can be rate-limited and externally modified (and our authorization can change)</td> | ||
| </tr> | ||
| <tr> | ||
| <th>Spec YAML</th> | ||
| <td> | ||
| <pre><code><em># File: foo.Deployment.yaml</em> | ||
| apiVersion: apps/v1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: nginx-deployment | ||
| spec:<div style="padding:0.5rem;margin:0 -0.5rem;background-color:rgba(0,0,255,0.25)"><strong> # Evaluate this to enumerate instances of...</strong> | ||
| replicas: 2</div> | ||
| <div style="padding:0.5rem;margin:0 -0.5rem;background-color:rgba(255,0,255,0.25)"><strong> # ...this template.</strong> | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: nginx | ||
| spec: | ||
| containers: | ||
| - name: nginx | ||
| image: nginx:1.14.2 | ||
| ports: | ||
| - containerPort: 80</div></pre></code> | ||
| </td> | ||
| <td> | ||
| <pre><code><em># File: hello-world.campaign.yaml</em> | ||
| name: hello-world | ||
| description: Add Hello World to READMEs | ||
|
|
||
| <div style="padding:0.5rem;margin:0 -0.5rem;background-color:rgba(0,0,255,0.25)"><strong># Evaluate this to enumerate instances of...</strong> | ||
| on: | ||
| - repositoriesMatchingQuery: file:README.md | ||
|
|
||
| steps: | ||
| - run: echo Hello | tee -a $(find -name '*.md') | ||
| container: alpine:3 | ||
| </div> | ||
| <div style="padding:0.5rem;margin:0 -0.5rem;background-color:rgba(255,0,255,0.25)"><strong># ...this template.</strong> | ||
| changesetTemplate: | ||
| title: Hello World | ||
| body: My first campaign! | ||
| branch: hello-world | ||
| commit: | ||
| message: Append Hello to .md files | ||
| published: false</pre></code> | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th>How desired state is computed</th> | ||
| <td> | ||
| <ol> | ||
| <li>Evaluate <code>replicas</code>, etc. (blue) to determine pod count and other template inputs</li> | ||
| <li>Instantiate <code>template</code> (pink) once for each pod to produce PodSpecs</li> | ||
| </ol> | ||
| </td> | ||
| <td> | ||
| <ol> | ||
| <li>Evaluate <code>on</code>, <code>steps</code> (blue) to determine list of patches</li> | ||
| <li>Instantiate <code>changesetTemplate</code> (purple) once for each patch to produce ChangesetSpecs | ||
| </li> | ||
| </ol> | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th>Desired state consists of...</th> | ||
| <td> | ||
| <ul> | ||
| <li>DeploymentSpec file (the YAML above)</li> | ||
| <li>List of PodSpecs (template instantiations)</li> | ||
| </ul> | ||
| </td> | ||
| <td> | ||
| <ul> | ||
| <li>CampaignSpec file (the YAML above)</li> | ||
| <li>List of ChangesetSpecs (template instantiations)</li> | ||
| </ul> | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th>Where is the desired state computed?</th> | ||
| <td>The deployment controller (part of the Kubernetes cluster) consults the DeploymentSpec and continuously computes the desired state.</td> | ||
| <td> | ||
| <p>The <a href="https://github.com/sourcegraph/src-cli">Sourcegraph CLI</a> (running on your local machine, not on the Sourcegraph server) consults the campaign spec and computes the desired state when you invoke <code>src campaign apply</code>.</p> | ||
| <p><strong>Difference vs. Kubernetes:</strong> A campaign's desired state is computed locally, not on the server. It requires executing arbitrary commands, which is not yet supported by the Sourcegraph server. See campaigns known issue "<a href="../user/campaigns#server-execution">Campaign steps are run locally...</a>".</p> | ||
| </td> | ||
| </tr> | ||
| <tr> | ||
| <th>Reconciling desired state vs. actual state</th> | ||
| <td>The "deployment controller" reconciles the resulting PodSpecs against the current actual PodSpecs (and does smart things like rolling deploy).</td> | ||
| <td>The "campaign controller" (i.e., our backend) reconciles the resulting ChangesetSpecs against the current actual changesets (and does smart things like gradual roll-out/publishing and auto-merging when checks pass).</td> | ||
| </tr> | ||
| </table> | ||
|
|
||
| These docs explain more about Kubernetes' design: | ||
|
|
||
| - [Kubernetes Object Management](https://kubernetes.io/docs/concepts/overview/working-with-objects/object-management/) | ||
| - [Kubernetes Controllers](https://kubernetes.io/docs/concepts/architecture/controller/) | ||
| - [Desired versus current state](https://kubernetes.io/docs/concepts/architecture/controller/#desired-vs-current) | ||
| - [Kubernetes Architecture](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/architecture.md) | ||
| - [Kubernetes General Configuration Tips](https://kubernetes.io/docs/concepts/configuration/overview/#general-configuration-tips) | ||
| - [Kubernetes Design and Development Explained](https://thenewstack.io/kubernetes-design-and-development-explained/) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,118 @@ | ||
| { | ||
| "$schema": "http://json-schema.org/draft-07/schema#", | ||
| "title": "CampaignSpec", | ||
| "description": "A campaign specification, which describes the campaign and what kinds of changes to make (or what existing changesets to track).", | ||
| "type": "object", | ||
| "additionalProperties": false, | ||
| "required": ["name"], | ||
| "properties": { | ||
| "name": { | ||
| "type": "string", | ||
| "description": "The name of the campaign, which is unique among all campaigns in the namespace. A campaign's name is case-preserving.", | ||
| "pattern": "^[\\w.-]+$" | ||
| }, | ||
| "description": { | ||
| "type": "string", | ||
| "description": "The description of the campaign." | ||
| }, | ||
| "on": { | ||
| "type": "array", | ||
| "description": "The set of repositories (and branches) to run the campaign on, specified as a list of search queries (that match repositories) and/or specific repositories.", | ||
| "items": { | ||
| "title": "OnQueryOrRepository", | ||
| "oneOf": [ | ||
| { | ||
| "title": "OnQuery", | ||
| "type": "object", | ||
| "description": "A Sourcegraph search query that matches a set of repositories (and branches). Each matched repository branch is added to the list of repositories that the campaign will be run on.", | ||
| "additionalProperties": false, | ||
| "required": ["repositoriesMatchingQuery"], | ||
| "properties": { | ||
| "repositoriesMatchingQuery": { | ||
| "type": "string", | ||
| "description": "A Sourcegraph search query that matches a set of repositories (and branches). If the query matches files, symbols, or some other object inside a repository, the object's repository is included.", | ||
| "examples": ["file:README.md"] | ||
| } | ||
| } | ||
| }, | ||
| { | ||
| "title": "OnRepository", | ||
| "type": "object", | ||
| "description": "A specific repository (and branch) that is added to the list of repositories that the campaign will be run on.", | ||
| "additionalProperties": false, | ||
| "required": ["repository"], | ||
| "properties": { | ||
| "repository": { | ||
| "type": "string", | ||
| "description": "The name of the repository (as it is known to Sourcegraph).", | ||
| "examples": ["github.com/foo/bar"] | ||
| }, | ||
| "branch": { | ||
| "type": "string", | ||
| "description": "The branch on the repository to propose changes to. If unset, the repository's default branch is used." | ||
| } | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| }, | ||
| "steps": { | ||
| "type": "array", | ||
| "description": "The sequence of commands to run (for each repository branch matched in the `on` property) to produce the campaign's changes.", | ||
| "items": { | ||
| "title": "Step", | ||
| "type": "object", | ||
| "description": "A command to run (as part of a sequence) in a repository branch to produce the campaign's changes.", | ||
| "additionalProperties": false, | ||
| "required": ["run", "container"], | ||
| "properties": { | ||
| "run": { | ||
| "type": "string", | ||
| "description": "The shell command to run in the container. It can also be a multi-line shell script. The working directory is the root directory of the repository checkout." | ||
| }, | ||
| "container": { | ||
| "type": "string", | ||
| "description": "The Docker image used to launch the Docker container in which the shell command is run.", | ||
| "examples": ["alpine:3"] | ||
| }, | ||
| "env": { | ||
| "type": "object", | ||
| "description": "Environment variables to set in the environment when running this command." | ||
| } | ||
| } | ||
| } | ||
| }, | ||
| "changesetTemplate": { | ||
| "type": "object", | ||
| "description": "A template describing how to create (and update) changesets with the file changes produced by the command steps.", | ||
| "additionalProperties": false, | ||
| "required": ["title", "branch", "commit", "published"], | ||
| "properties": { | ||
| "title": { "type": "string", "description": "The title of the changeset." }, | ||
| "body": { "type": "string", "description": "The body (description) of the changeset." }, | ||
| "branch": { | ||
| "type": "string", | ||
| "description": "The name of the Git branch to create or update on each repository with the changes." | ||
| }, | ||
| "commit": { | ||
| "title": "GitCommitDescription", | ||
| "type": "object", | ||
| "description": "The Git commit to create with the changes.", | ||
| "additionalProperties": false, | ||
| "required": ["message"], | ||
| "properties": { | ||
| "message": { | ||
| "type": "string", | ||
| "description": "The Git commit message." | ||
| } | ||
| } | ||
| }, | ||
| "published": { | ||
| "type": "boolean", | ||
| "description": "Whether to publish the changeset. An unpublished changeset can be previewed on Sourcegraph by any person who can view the campaign, but its commit, branch, and pull request aren't created on the code host. A published changeset results in a commit, branch, and pull request being created on the code host.", | ||
| "$comment": "TODO(sqs): Come up with a way to specify that only a subset of changesets should be published. For example, making `published` an array with some include/exclude syntax items." | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } |
Oops, something went wrong.