Skip to content

Latest commit

 

History

History
155 lines (112 loc) · 7.91 KB

README.md

File metadata and controls

155 lines (112 loc) · 7.91 KB
Raw

Infrastructure

While the application can be deployed in any server configuration that supports the application's dependencies, this project includes:

  • a Terraform project that you can use to easily and quickly provision the resources and deploy the code using Google Cloud Platform,
  • and a GH Actions workflow to deploy code updates.

Dependencies

Here is the list of technical dependencies for deploying the Rangelands app using these infrastructure resources. Note that these requirements are for this particular deployment strategy, and not dependencies of the S[project name] application itself - which can be deployed to other infrastructures.

Before proceeding, be sure you are familiar with all of these tools, as these instructions will skip over the basics, and assume you are conformable using all of them.

Terraform project

This project (in the inrastructure directory) has 2 main sections, each of which with a folder named after it. Each of these sections has a Terraform project, that logically depends on their predecessors. There is a 3rd component to this architecture, which is handled by Github Actions.

Remote state

Creates a GCP Storage Bucket, which will store the Terraform remote state.

Base

Contains multiple GCP resources needed for running the Global Rangelands application on GCP.

These resources include, but are not limited to:

  • Google Compute instance - bastion host to access the GCP infrastructure
  • Artifact Registry, for docker image storage
  • Cloud Run, to host the client application, API/CMS
  • Cloud SQL, for relational data storage
  • Cloud Functions for the Earth Engine mini tiler server
  • Networking resources
  • Uptime monitoring
  • Error reporting
  • Service accounts and permissions
  • GH Secrets

To apply this project, you will need the following GCP permissions:

  • "Editor" role
  • "Secret Manager Admin" role
  • "Cloud Run Admin" role
  • "Compute Network Admin" role
  • "Security Admin" role

The output values include access data for some of the resources above.

Please note, there are some actions that might need to be carried out manually - you'll get a promt from terraform with links to follow to complete the actions, e.g.:

  • Compute Engine API needs to be enabled

ATTENTION: the CloudRun instances will be started with a default "hello" image on the first terraform apply (since FE / BE images are not available yet); afterwards please ensure to uncomment the relevant line in cloudrun/main.tf which points to the latest available image, so that you don't overwrite with the "hello" image on subsequent terraform apply.

How to run

Deploying the included Terraform project is done in steps:

  • Terraform apply the State project. This needs to be done once, before applying the base project.
  • Terraform apply the Base project. This needs to be repeated after any changes in the base project.

For both commands, please use `-var-file=vars/terraform.tfvars`` to provide the necessary terraform variables.

For the second command, you will also need to set 2 environment variables:

  • GITHUB_TOKEN (your GH token)
  • GITHUB_OWNER (e.g. Vizzuality) to allow terraform to write to GH Secrets.

Github Actions

As part of this infrastructure, Github Actions are used to automatically apply code updates for the client application, API/CMS and the Cloud Functions EE tiler.

Building new code versions

Deployment to the CloudRun instances is accomplished by building Docker images and then pushing to Artifact Registry. When building the images, environment secrets are injected from GH Secrets as follows:

  • for the client application:
    • the following secrets set by terraform in STAGING_CLIENT_ENV_TF_MANAGED (in the format of an .env file):
      • NEXT_PUBLIC_URL
      • NEXT_PUBLIC_API_URL
      • NEXT_PUBLIC_EET_CF_URL
      • NEXT_PUBLIC_ENVIRONMENT
      • LOG_LEVEL
    • additional secrets set manually in STAGING_CLIENT_ENV (copy to be managed in LastPass)
  • for the CMS/API application
    • the following secrets set by terraform in STAGING_CMS_ENV_TF_MANAGED (in the format of an .env file):
      • HOST
      • PORT
      • APP_KEYS
      • API_TOKEN_SALT
      • ADMIN_JWT_SECRET
      • TRANSFER_TOKEN_SALT
      • JWT_SECRET
      • CMS_URL
      • DATABASE_CLIENT
      • DATABASE_HOST
      • DATABASE_NAME
      • DATABASE_USERNAME
      • DATABASE_PASSWORD
      • DATABASE_SSL

For Cloud Functions, build is executed remotely on GCP, and all variables and secrets are defined as part of the Cloud Functions (they can be checked on the GCP console).

The workflow is currently set up to deploy to the staging instance when merging to develop.

Service account permissions

Access by Github to GCP is configured through special authorization rules, automatically set up by the Terraform base project above. These permissions are necessary for the service account that runs the deployment:

  • "roles/iam.serviceAccountTokenCreator",
  • "roles/iam.serviceAccountUser",
  • "roles/run.developer",
  • "roles/artifactregistry.reader",
  • "roles/artifactregistry.writer",

Maintenance

Connecting to the Cloud SQL databases

In case you need to access the Postgres database for the app, running in Cloud SQL, you can follow these steps. This is a slimmed down version of this guide

You will need the following information from the Google Cloud console:

  • < bastion-instance-name > - name of the bastion host VM instance in Compute Engine
  • < sql instance connection name > - connection name of the Cloud SQL instance
  • < database password > - secrets manager

You will also need to ensure that the user has IAP-secured Tunnel User role.

Steps:

  • (one time per user) Run gcloud compute ssh <bastion instance name> --zone=<zone where bastion is located> to SSH into the bastion host
  • (one time per bastion host) Inside the bastion host, follow the steps to download and install the Cloud SQL Auth proxy
  • (when connecting) Run gcloud compute start-iap-tunnel <bastion instance name> 22 --local-host-port=localhost:4226 locally. This will start a tunnel, which you must keep open for the duration of your access to the SQL database
  • (when connecting) Run ssh -L 5433:localhost:5433 -i ~/.ssh/google_compute_engine -p 4226 localhost -- ./cloud-sql-proxy <sql instance connection name> --port=5433 --private-ip locally. This will start a 2nd tunnel, which you must also keep open for the duration of your access to the SQL database
  • The remote Postgres database is now reachable on a local port 5433: psql -h 127.0.0.1 -p 5433 -U db_user -W db_name

Backups

There are two main permanent data storage mechanisms in the Rangelands application that need backup.

SQL

The project's backend application relies on a PostgreSQL database, that is implemented in the infrastructure using GCP's Cloud SQL. Cloud SQL has a built-in backup functionality, which is currently configured to create a backup daily, and keep it for 30 days. Backup restoration can be done manually. Refer to the official GCP documentation on this feature for instructions and more details.

Object Storage

The application stores certain files in object storage, which is implemented in the infrastructure using GCP's Cloud Storage. Cloud Storage has built-in versioning functionality, which allows accessing old versions of a file, should it be modified, as well as accessing deleted files. Refer to the official GCP documentation on this feature for more details, and instructions on how to use it to recover lost data.