Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix menu #287

Merged
merged 3 commits into from
May 21, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 1 addition & 4 deletions docs/intro.md → docs/01-introduction/index.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
---
sidebar_position: 1
title: Welcome to GameCI
---
# Introduction

This site hosts all documentation for GameCI, and will help you setup Continuous Integration for your game projects.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,30 +1,34 @@
# Introduction

## Concept - What Does Cloud Runner Do

**Cloud Runner enables you to run, build and test workflows in the cloud, right from GitHub actions. Builder will automatically provision an environment at a Cloud Provider such as GCP and AWS. It will then send the project to be built and/or tested depending on your workflow configuration.**
**Cloud Runner enables you to run, build and test workflows in the cloud, right from GitHub actions. Builder will automatically provision an environment at a Cloud Provider such as GCP and AWS. It will then send the project to be built and/or tested depending on your workflow configuration.**

**Cloud Runner is especially useful for game development because it supports large projects. Cloud Runner provides first class support for the Unity game engine.**

Cloud Runner uses git to track and transfer your projects and uses native cloud services such as AWS Fargate and Kubernetes to run your jobs. Other version control systems are not actively supported.

## Why cloud runner?

1. Extended options and more control over disk size, memory and CPU. You can build projects of almost any size.
2. Scale up to much larger numbers of builds easily and fully on demand.
3. Run custom jobs and extend the system for any workload.
4. Create resources on-demand, we have made an effort to make sure that it costs you nothing while there are no builds running (no guarantees).

## Why not cloud runner?

1. Your project is small in size. Below 5GB Cloud Runner should not be needed.
2. You already have servers running you can use for capacity.
3. You strongly prefer to avoid the addition of time to your pipeline, slowing down results.
3. You strongly prefer to avoid the addition of time to your pipeline, slowing down results.

Although the speed of a CI pipelines is an important metric to consider, there are real challenges for game development pipelines.
Although the speed of a CI pipelines is an important metric to consider, there are real challenges for game development pipelines.

This solution prefers convenience, ease of use, scalability, throughput and flexibility.
This solution prefers convenience, ease of use, scalability, throughput and flexibility.

Faster solutions exist, but would all involve self-hosted hardware with an immediate local cache of the large project files and working directory and a dedicated server.

## Cloud Runner Release Status

Cloud Runner is in "active development" ⚠️🔨

Cloud Runner overall release status: `preview`
Expand All @@ -35,34 +39,40 @@ Release Stages: `experimental` ➡️ `preview` ➡️ `full release`
You must use a provider with Cloud Runner, each provider's release status is described below. This indicates the stability and support for cloud runner features and workflows.

### Development

_Cloud Runner is actively maintained and kept stable by the Game CI open source project contributors._

💬suggestions, 🐛bugs and ↕️Minor changes are tracked as GitHub issues:
- [Game CI Issues - GitHub](https://github.com/game-ci/unity-builder/labels/cloud-runner)

- [Game CI Issues - GitHub](https://github.com/game-ci/unity-builder/labels/cloud-runner)

You can also explore the [development roadmap page](development).

### Community
__Share your feedback with us!__
- [__Discord Channel__](https://discord.com/channels/710946343828455455/789631903157583923)
- [__Feedback Form__](https://forms.gle/3Wg1gGf9FnZ72RiJ9)

**Share your feedback with us!**

- [**Discord Channel**](https://discord.com/channels/710946343828455455/789631903157583923)
- [**Feedback Form**](https://forms.gle/3Wg1gGf9FnZ72RiJ9)

### Supported Cloud Runner Platforms

```md
| Cloud Provider Platform | Release Status |
| ----------------------- | ------------------------- |
| Cloud Provider Platform | Release Status |
| ----------------------- | ----------------------- |
| Kubernetes | ✔️ experimental release |
| AWS | ✔️ preview release |
| GCP | ⚠ Considered |
| Azure | ⚠ Considered |
| GCP | ⚠ Considered |
| Azure | ⚠ Considered |
```

_Note for Kuberentes support:_
_Usually the cluster needs to be up and running at all times, as starting up a cluster is slow._
_Use Google Cloud's Kubernetes Autopilot you can scale down to the free tier automatically while not in use._

```md
| Git Platform | Release Status |
| --------------------- | -------------------- |
| Git Platform | Release Status |
| --------------------- | ------------------ |
| GitHub | ✔️ preview release |
| GitLab | ✔️ preview release |
| Command Line | ✔️ preview release |
Expand All @@ -71,6 +81,7 @@ _Use Google Cloud's Kubernetes Autopilot you can scale down to the free tier aut
```

## Releases

All cloud runner releases are currently packaged and released with game-ci's unity-builder module:
[Game CI Releases - GitHub](https://github.com/game-ci/unity-builder/releases)

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
position: 2.0
label: Getting started
collapsible: true
collapsed: true
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ Setup the following as `env` variables for cloud runner to use:
- `AWS_DEFAULT_REGION` (should be the same AWS region as the base stack e.g `eu-west-2`)

If you're using GitHub you can use a GitHub Action:

```yaml
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
Expand All @@ -27,6 +28,7 @@ _Note:_
_This enables Cloud Runner access AWS._

## Configuration For AWS Cloud Runner Jobs

Refer to [Configuration page](../configuration) or the [example below](#example).

### Allowed CPU/Memory Combinations
Expand All @@ -35,6 +37,7 @@ There are some limitations to the CPU and Memory parameters. AWS will only accep
[AWS Fargate Documentation, Allowed CPU and memory values (Task Definitions)](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#task_size)

#### Summary Of Format

- Values are represented as 1024:1 GB or CPU.
- Do not include the vCPU or GB suffix.
- 1 CPU can go to a max of 6 GB of memory. 2 CPU's are required to go higher.
Expand All @@ -47,6 +50,7 @@ There are some limitations to the CPU and Memory parameters. AWS will only accep
```

## Example

```yaml
- uses: game-ci/unity-builder@cloud-runner-develop
id: aws-fargate-unity-build
Expand Down Expand Up @@ -79,6 +83,7 @@ There are some limitations to the CPU and Memory parameters. AWS will only accep
- name: awsDefaultRegion
value: eu-west-2
```

_[Custom Steps](../advanced-topics/custom-steps)_

A full workflow example can be seen in builder's [Cloud Runner GitHub sourcecode for AWS Pipeline](https://github.com/game-ci/unity-builder/blob/main/.github/workflows/cloud-runner-aws-pipeline.yml).
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Setup the following as `env` variables for the GitHub build step:
- `kubeConfig` (should be encoded as base64)

## Configuration For Kubernetes Cloud Runner Jobs

Refer to [Configuration page](../configuration) or the [example below](#example).

### Allowed CPU/Memory Combinations
Expand All @@ -25,12 +26,14 @@ Refer to [Configuration page](../configuration) or the [example below](#example)
Do not include the vCPU or GB suffix.

#### Valid CPU and Memory Values

```yaml
- remoteBuildMemory: 2
- remoteBuildCpu: 0.5
```

### Example

```yaml
- uses: game-ci/unity-builder@cloud-runner-develop
id: k8s-unity-build
Expand Down Expand Up @@ -66,6 +69,7 @@ Do not include the vCPU or GB suffix.
- name: awsDefaultRegion
value: eu-west-2
```

_[Custom Steps](../advanced-topics/custom-steps)_

A full workflow example can be seen in builder's [Cloud Runner GitHub sourcecode for AWS Pipeline](https://github.com/game-ci/unity-builder/blob/main/.github/workflows/cloud-runner-k8s-pipeline.yml).
Original file line number Diff line number Diff line change
@@ -1,16 +1,22 @@
# Configuration

### Configuration definition yaml

All accepted parameters given here with a description:

- [https://github.com/game-ci/unity-builder/blob/cloud-runner-develop/action.yml](https://github.com/game-ci/unity-builder/blob/cloud-runner-develop/action.yml)

### GitHub Access Token

Include the following variable in the `with` section of the builder step:

- `githubToken` (should be a GitHub access token with permission to get repositories)

_Note:_
_This enables us to get the repository from the AWS build machine._

### Release branch configuration

_GitHub Action examples to pull from the required release branch_

#### Develop branch
Expand All @@ -26,9 +32,10 @@ _GitHub Action examples to pull from the required release branch_
```

### Provider Configuration

You must specify the Cloud Provider to use via the parameter `cloudRunnerCluster`.

Valid options:

- `aws` [Getting Started - AWS](getting-started/aws)
- `k8s` [Getting Started - Kubernetes](getting-started/kubernetes)

Original file line number Diff line number Diff line change
Expand Up @@ -11,21 +11,27 @@ git clone https://github.com/game-ci/unity-builder.git
yarn install
yarn run cli -m {mode parameter} --projectPath {Your project path} {... other command line parameters}
```

# Planned (does not work currently)

We plan to offer support for Game CI via Deno. This will enable fast, typescript native runtime and you will be able to access this via the following:

```bash
dpx game-ci build
```

# Help

_You can run `yarn run cli -h` or `yarn run cli --help` to List all modes and paramters with descriptions_

# Main Command Parameters

- Default: `cli` (runs a standard build workflow)
- [Garbage Collection Modes](#garbage-collection)

# Avoiding long parameters for commands
You can avoid specifying lots of command line input for credentials (e.g all unity authentication and cloud provider settings) by using environment variables or [the input override feature](advanced-topics/input-override#example) to shorten commands signficantly.

You can avoid specifying lots of command line input for credentials (e.g all unity authentication and cloud provider settings) by using environment variables or [the input override feature](advanced-topics/input-override#example) to shorten commands signficantly.

This enables you to provide a command to pull input, e.g you can pull from a file or from a secret manager.

Expand All @@ -34,19 +40,25 @@ yarn run cli --populateOverride true --readInputFromOverrideList UNITY_EMAIL,UNI
```

## Garbage Collection

### AWS Provider Only

```bash
aws-list-stacks
```

```bash
aws-list-tasks
```

```bash
aws-garbage-collect
```

```bash
aws-garbage-collect-all
```

```bash
aws-garbage-collect-all-1d-older
```
Original file line number Diff line number Diff line change
@@ -1,20 +1,25 @@
# Performance
# Performance

By sending the build to a cloud resource we must accept some overhead in time to do this. Cloud Runner tries to manage and offer good solutions to cut down any overhead. As well as supporting you to use all options to speed up common workflows, such as incremental builds.

## Warm start

⚠️Development roadmap for cloud runner feature: Will enable you to maintain a minimum number of active build resources to handle incoming jobs with shorter delay times to start a job.

## Push Hooks

⚠️Development roadmap for cloud runner feature: Rather than starting a job from your CI process, you can initally start a Cloud Runner job from your local git push hooks, the CI process can then catch up and observe the logs as needed.

## Sidecar

⚠️Development roadmap for cloud runner feature: Continually stream changes to game-ci workloads without needing to push or even commit explicitly (configurable).

## Caching

Caching is used to speed up git clones, Unity imports and builds and can be extended for custom scenarios.

See [Caching](caching).

## Provider Platform's impact

The different Cloud Runner providers all offer different guarenetees and performance for handling your workloads.
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
# Input Override

When running any unity workload you must provide valid unity credentials. In addition to any other credentials this is already quite a lot of input. For this reason, it is common to use the command line mode with input override (link here). This enables you to provide a command to pull input, with this approach you can create a file to store credentials or pull from a secret manager.

## Example

```bash
game-ci -m cli --populateOverride true --readInputFromOverrideList UNITY_EMAIL,UNITY_SERIAL,UNITY_PASSWORD --readInputOverrideCommand="gcloud secrets versions access 1 --secret=\"{0}\""
```

## Required Parameters

- `populateOverride` - Must be true to run the commands.
- `readInputFromOverrideList` - Comma separated list of parameters to read from override command.
- `readInputOverrideCommand` - A command line command to run (The command is formatted to replace "{0}" with the parameter parameter name).
- `readInputOverrideCommand` - A command line command to run (The command is formatted to replace "{0}" with the parameter parameter name).
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,10 @@ Cloud Runner runs the workload and cleans up the resources. It does not expect t
If you cancel a task, the resources will not be cleaned up.

# Full Release - Garbage Collection (Not available yet)

Cloud runner will collect the garbage when it next cleans up. There will be utility functions to clean up directly.
Optional via configuration: Cloud runner can schedule cron jobs to independently guarentee shutdown (already possible via custom hooks).

# Garbage Collection - Command Line Functions

[Command Line Garbage Collection Utilities](../command-line#garbage-collection)
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Logging
Logs are streamed from the workload to the Game CI origin unless you use the

Logs are streamed from the workload to the Game CI origin unless you use the

## Kubernetes

- Native kuberentes logging api
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@
Secrets are transferred to workload containers as secrets via the built-in secrets system the provider being used supports.

## Kubernetes

secrets are created as native kuberentes secret objects and mounted to job containers as env variables.

## AWS

secrets are created as aws secret manager secrets and mounted to fargate tasks from secrets to env variables.
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
position: 5.0
label: Advanced topics
collapsible: true
collapsed: true
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Development

## GitHub Issues

Minor changes, suggestions and bugs are tracked as GitHub issues:
[Game CI Issues - GitHub](https://github.com/game-ci/unity-builder/labels/cloud-runner)

Expand All @@ -17,7 +18,7 @@ Minor changes, suggestions and bugs are tracked as GitHub issues:
| built-in integration for Unity test framework | todo | |
| built-in integration for steam deploy | todo | |
| extract cloud runner input options into encapsulated object | todo | |
| nested github self-hosted runner sample for game-ci | in-progress | |
| nested github self-hosted runner sample for game-ci | in-progress | |
```

## Iteration 1 - GPU Workloads And Capabilities
Expand Down
5 changes: 5 additions & 0 deletions docs/03-github-cloud-runner/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
position: 3.1
label: Github (Cloud Runner)
collapsible: true
collapsed: true
File renamed without changes.
File renamed without changes.
File renamed without changes.
5 changes: 5 additions & 0 deletions docs/03-github/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
position: 3.0
label: GitHub (Default)
collapsible: true
collapsed: true
5 changes: 5 additions & 0 deletions docs/03-github/deployment/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
position: 5.0
label: Deployment
collapsible: true
collapsed: true
File renamed without changes.
5 changes: 5 additions & 0 deletions docs/05-gitlab/_category_.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
position: 4.0
label: GitLab
collapsible: true
collapsed: true
Loading