From b902dd36db4d590b6e9d8d1eb9bece49aa37506e Mon Sep 17 00:00:00 2001 From: oddgrd <29732646+oddgrd@users.noreply.github.com> Date: Wed, 22 Mar 2023 16:13:41 +0100 Subject: [PATCH 1/7] docs: update contributing project structure --- CONTRIBUTING.md | 14 +++++++++----- Cargo.lock | 20 -------------------- 2 files changed, 9 insertions(+), 25 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e4569995c..f3434ea68 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -246,15 +246,19 @@ graph BT user -->|"features = ['codegen']"| service ``` -First, `provisioner`, `gateway`, `deployer`, and `cargo-shuttle` are binary crates with `provisioner`, `gateway` and `deployer` being backend services. The `cargo-shuttle` binary is the `cargo shuttle` command used by users. +First, `provisioner`, `gateway`, `deployer`, `auth`, `admin` and `cargo-shuttle` are binary crates with `provisioner`, `gateway`, `auth` and `deployer` being backend services. The `cargo-shuttle` binary is the `cargo shuttle` command-line interface used by users. The rest are the following libraries: - `common` contains shared models and functions used by the other libraries and binaries. -- `codegen` contains our proc-macro code which gets exposed to user services from `service` by the `codegen` feature flag. The redirect through `service` is to make it available under the prettier name of `shuttle_service::main`. -- `service` is where our special `Service` trait is defined. Anything implementing this `Service` can be loaded by the `deployer` and the local runner in `cargo-shuttle`. - The `codegen` automatically implements the `Service` trait for any user service. -- `proto` contains the gRPC server and client definitions to allow `deployer` to communicate with `provisioner`. +- `codegen` contains our proc-macro code which gets exposed to user services from `runtime`. +The redirect through `runtime` is to make it available under the prettier name of `shuttle_runtime::main`. +- `runtime` contains the `alpha` runtime, which embeds a `Loader` that sets up tracing, provisions resources and starts the users +service. The `runtime` starts a GRPC server to communicate with the `deployer` for deployments or `cargo-shuttle` cli for local runs. The `runtime` crate also contains the `shuttle-next` binary, which is a standalone runtime binary that is started by the `deployer` or the `cargo-shuttle` cli. This also uses a GRPC server to communicate, and takes a users shuttle-next service as a `wasm32-wasi` module. +- `service` is where our special `Service` trait is defined. Anything implementing this `Service` can be loaded by the `deployer` and the local runner in `cargo-shuttle`. The `service` library also defines the `ResourceBuilder` and `Factory` trais +which are used in our codegen to provision resources. +- `proto` contains the gRPC server and client definitions to allow `deployer` to communicate with `provisioner`, and to allow +the `deployer` and `cargo-shuttle` cli to communicate with the `alpha` and `shuttle-next` runtimes. - `e2e` just contains tests which starts up the `deployer` in a container and then deploys services to it using `cargo-shuttle`. Lastly, the `user service` is not a folder in this repository, but is the user service that will be deployed by `deployer`. diff --git a/Cargo.lock b/Cargo.lock index 1a2d638aa..a6f13220f 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -7222,23 +7222,3 @@ dependencies = [ "cc", "libc", ] - -[[patch.unused]] -name = "shuttle-aws-rds" -version = "0.12.0" - -[[patch.unused]] -name = "shuttle-persist" -version = "0.12.0" - -[[patch.unused]] -name = "shuttle-poise" -version = "0.12.0" - -[[patch.unused]] -name = "shuttle-shared-db" -version = "0.12.0" - -[[patch.unused]] -name = "shuttle-static-folder" -version = "0.12.0" From 8d69225996d243ce47e23506008ca2514d0b9f5a Mon Sep 17 00:00:00 2001 From: oddgrd <29732646+oddgrd@users.noreply.github.com> Date: Wed, 22 Mar 2023 16:26:34 +0100 Subject: [PATCH 2/7] docs: add notes about services and resources --- CONTRIBUTING.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f3434ea68..99945440a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -235,8 +235,8 @@ graph BT deployer -.->|calls| provisioner service ---> common deployer --> common - cargo-shuttle --->|"features = ['loader']"| service - deployer -->|"features = ['loader']"| service + cargo-shuttle --->|"features = ['builder']"| service + deployer -->|"features = ['builder']"| service cargo-shuttle --> common service --> codegen proto ---> common @@ -256,9 +256,13 @@ The redirect through `runtime` is to make it available under the prettier name o - `runtime` contains the `alpha` runtime, which embeds a `Loader` that sets up tracing, provisions resources and starts the users service. The `runtime` starts a GRPC server to communicate with the `deployer` for deployments or `cargo-shuttle` cli for local runs. The `runtime` crate also contains the `shuttle-next` binary, which is a standalone runtime binary that is started by the `deployer` or the `cargo-shuttle` cli. This also uses a GRPC server to communicate, and takes a users shuttle-next service as a `wasm32-wasi` module. - `service` is where our special `Service` trait is defined. Anything implementing this `Service` can be loaded by the `deployer` and the local runner in `cargo-shuttle`. The `service` library also defines the `ResourceBuilder` and `Factory` trais -which are used in our codegen to provision resources. +which are used in our codegen to provision resources. The `service` library also contains the utilities we use for compiling users +crates with `cargo`. - `proto` contains the gRPC server and client definitions to allow `deployer` to communicate with `provisioner`, and to allow the `deployer` and `cargo-shuttle` cli to communicate with the `alpha` and `shuttle-next` runtimes. +- `resources` contains various implementations of `ResourceBuilder`, which are consumed in the `codegen` to provision resources. +- `services` contains implementations of `Service` for common Rust web frameworks. Anything implementing `Service` can be deployed +by shuttle. - `e2e` just contains tests which starts up the `deployer` in a container and then deploys services to it using `cargo-shuttle`. Lastly, the `user service` is not a folder in this repository, but is the user service that will be deployed by `deployer`. From 7e517fe69523abc7d16c520a692f9f8c46069407 Mon Sep 17 00:00:00 2001 From: oddgrd <29732646+oddgrd@users.noreply.github.com> Date: Thu, 23 Mar 2023 18:43:17 +0100 Subject: [PATCH 3/7] feat: briefly describe the binaries --- CONTRIBUTING.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 99945440a..b84939858 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -246,15 +246,23 @@ graph BT user -->|"features = ['codegen']"| service ``` -First, `provisioner`, `gateway`, `deployer`, `auth`, `admin` and `cargo-shuttle` are binary crates with `provisioner`, `gateway`, `auth` and `deployer` being backend services. The `cargo-shuttle` binary is the `cargo shuttle` command-line interface used by users. +### Binaries -The rest are the following libraries: +- `cargo-shuttle` is the CLI used by users to initialize, deploy and manage their projects and services on shuttle. +- `gateway` starts and manages instances of `deployer`. It proxies commands from the user sent via the CLI on port 8001 and traffic on port 8000 to the correct instance of `deployer`. +- `auth` is an authentication service that creates and manages users, in addition to converting API-keys/cookies to JWTs for authorization between internal services, like a `deployer` requesting a database from +`provisioner`. +- `deployer` is a service that runs in its own docker container, one per user project. It manages users' services and their state. +- `provisioner` is a service used for requesting databases and other resources, using a gRPC API. +- `admin` is a simple CLI used for admin tasks like reviving and stopping projects, as well as requesting +and renewing SSL certificates through the acme client in the `gateway`. + +### Libraries - `common` contains shared models and functions used by the other libraries and binaries. - `codegen` contains our proc-macro code which gets exposed to user services from `runtime`. The redirect through `runtime` is to make it available under the prettier name of `shuttle_runtime::main`. -- `runtime` contains the `alpha` runtime, which embeds a `Loader` that sets up tracing, provisions resources and starts the users -service. The `runtime` starts a GRPC server to communicate with the `deployer` for deployments or `cargo-shuttle` cli for local runs. The `runtime` crate also contains the `shuttle-next` binary, which is a standalone runtime binary that is started by the `deployer` or the `cargo-shuttle` cli. This also uses a GRPC server to communicate, and takes a users shuttle-next service as a `wasm32-wasi` module. +- `runtime` contains the `alpha` runtime, which embeds a `Loader` that sets up tracing, provisions resources and starts the users service. The `runtime` starts a GRPC server to communicate with the `deployer` for deployments or the `cargo-shuttle` CLI for local runs. The `runtime` crate also contains the `shuttle-next` binary, which is a standalone runtime binary that is started by the `deployer` or the `cargo-shuttle` CLI. This also uses a GRPC server to communicate, and takes a users shuttle-next service as a `wasm32-wasi` module. - `service` is where our special `Service` trait is defined. Anything implementing this `Service` can be loaded by the `deployer` and the local runner in `cargo-shuttle`. The `service` library also defines the `ResourceBuilder` and `Factory` trais which are used in our codegen to provision resources. The `service` library also contains the utilities we use for compiling users crates with `cargo`. From 3d1e9fa1200ca1ef5b775497328afa035f46828f Mon Sep 17 00:00:00 2001 From: oddgrd <29732646+oddgrd@users.noreply.github.com> Date: Thu, 23 Mar 2023 18:49:40 +0100 Subject: [PATCH 4/7] feat: update readme with protoc and binstall --- README.md | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index bb6a8d9cf..ebd4b761c 100644 --- a/README.md +++ b/README.md @@ -42,15 +42,30 @@ Shuttle is built for productivity, reliability and performance: ## Getting Started -Run the following command to install shuttle: +To run a shuttle project, you need to install protoc, check out the [protoc installation guide](https://docs.shuttle.rs/support/installing-protoc) in our docs. -```bash +The `cargo-shuttle` CLI can be installed with a pre-built binary or from source with cargo. + +Shuttle provides pre-built binaries of the `cargo-shuttle` CLI with every release +for most platforms, they can be found on [our GitHub](https://github.com/shuttle-hq/shuttle/releases/latest). + +Our binaries can also be installed using [cargo-binstall](https://github.com/cargo-bins/cargo-binstall), +which will automatically install the correct target for your system. +To install with `cargo-binstall`, run: + +```sh +cargo binstall cargo-shuttle +``` + +Although a bit slower, you can also install directly with cargo: + +```sh cargo install cargo-shuttle ``` -And then login: +After installing, log in with: -```bash +```sh cargo shuttle login ``` From 8992e843afd0611aa31f3aaaf4b71aa5e4a159b1 Mon Sep 17 00:00:00 2001 From: oddgrd <29732646+oddgrd@users.noreply.github.com> Date: Thu, 23 Mar 2023 19:05:22 +0100 Subject: [PATCH 5/7] feat: update mermaid graph --- CONTRIBUTING.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b84939858..06a3f5036 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -228,9 +228,12 @@ graph BT provisioner:::binary service gateway:::binary + auth:::binary user([user service]):::external gateway --> common gateway -.->|starts instances| deployer + gateway -->|key| auth + auth -->|jwt| gateway deployer --> proto deployer -.->|calls| provisioner service ---> common @@ -242,6 +245,7 @@ graph BT proto ---> common provisioner --> proto e2e -.->|starts up| gateway + e2e -.->|starts up| auth e2e -.->|calls| cargo-shuttle user -->|"features = ['codegen']"| service ``` From 67872a86c5a5d91bc0d4b9ca49a4c89b0726724e Mon Sep 17 00:00:00 2001 From: oddgrd <29732646+oddgrd@users.noreply.github.com> Date: Thu, 23 Mar 2023 19:17:54 +0100 Subject: [PATCH 6/7] refactor: auth description --- CONTRIBUTING.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 06a3f5036..720316ee9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -254,9 +254,9 @@ graph BT - `cargo-shuttle` is the CLI used by users to initialize, deploy and manage their projects and services on shuttle. - `gateway` starts and manages instances of `deployer`. It proxies commands from the user sent via the CLI on port 8001 and traffic on port 8000 to the correct instance of `deployer`. -- `auth` is an authentication service that creates and manages users, in addition to converting API-keys/cookies to JWTs for authorization between internal services, like a `deployer` requesting a database from -`provisioner`. -- `deployer` is a service that runs in its own docker container, one per user project. It manages users' services and their state. +- `auth` is an authentication service that creates and manages users. In addition to that, requests to the `gateway` that contain an api-key or cookie will be proxied to the `auth` service where it will be converted to a JWT for authorization between internal services (like a `deployer` requesting a database from +`provisioner`). +- `deployer` is a service that runs in its own docker container, one per user project. It manages a project's deployments and state. - `provisioner` is a service used for requesting databases and other resources, using a gRPC API. - `admin` is a simple CLI used for admin tasks like reviving and stopping projects, as well as requesting and renewing SSL certificates through the acme client in the `gateway`. From 4166a4bbc983fe15862d836a7a64ec12f573505d Mon Sep 17 00:00:00 2001 From: oddgrd <29732646+oddgrd@users.noreply.github.com> Date: Thu, 23 Mar 2023 19:44:12 +0100 Subject: [PATCH 7/7] refactor: runtime description --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 720316ee9..2485c49e6 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -266,7 +266,7 @@ and renewing SSL certificates through the acme client in the `gateway`. - `common` contains shared models and functions used by the other libraries and binaries. - `codegen` contains our proc-macro code which gets exposed to user services from `runtime`. The redirect through `runtime` is to make it available under the prettier name of `shuttle_runtime::main`. -- `runtime` contains the `alpha` runtime, which embeds a `Loader` that sets up tracing, provisions resources and starts the users service. The `runtime` starts a GRPC server to communicate with the `deployer` for deployments or the `cargo-shuttle` CLI for local runs. The `runtime` crate also contains the `shuttle-next` binary, which is a standalone runtime binary that is started by the `deployer` or the `cargo-shuttle` CLI. This also uses a GRPC server to communicate, and takes a users shuttle-next service as a `wasm32-wasi` module. +- `runtime` contains the `alpha` runtime, which embeds a gRPC server and a `Loader` in a service with the `shuttle_runtime::main` macro. The gRPC server receives commands from `deployer` like `start` and `stop`. The `Loader` sets up a tracing subscriber and provisions resources for the users service. The `runtime` crate also contains the `shuttle-next` binary, which is a standalone runtime binary that is started by the `deployer` or the `cargo-shuttle` CLI, responsible for loading and starting `shuttle-next` services. - `service` is where our special `Service` trait is defined. Anything implementing this `Service` can be loaded by the `deployer` and the local runner in `cargo-shuttle`. The `service` library also defines the `ResourceBuilder` and `Factory` trais which are used in our codegen to provision resources. The `service` library also contains the utilities we use for compiling users crates with `cargo`.