-
Notifications
You must be signed in to change notification settings - Fork 0
Production Setup
The following instructions are just as reference to show how the project setup was done. It also serves to other developers that might want to deploy it to their own clusters. If you want to contribute to the project you probably don't need this, just develop locally and submit the merge request.
This project is intended to run on a Google Cloud Kubernetes cluster: GKE (Google Kubernetes Engine). In this page you will find the instructions to setup it all up from scratch.
This setup costs around $10 a month. It uses the trick of going without Google's Load Balancer (which cuts down around $20 a month). The details about this can be found in this article:
First of all I would suggest buying a domain.
I would suggest one from Namecheap. dreamdrugs.art costed me $4.88.
Later on we will configure DNS settings to proxy our traffic through Cloudflare to avoid having to pay Google's Load Balancer! Every time a new node appears on our cluster it will be registered with Cloudflare as an A Alias.
First you need to sign up on Google Cloud and follow the registration steps.
After doing so it's time to setup the SDK that gives you gcloud the CLI to manage Google Cloud resources:
Before proceeding ensure you have:
- A working Google Cloud account.
- A project created. This guide assumes you have chosen "dreamdrugs".
- Have
gcloud(Google Cloud SDK) installed and set up with:- A default zone
- A default project
Reference: Kubernetes: The Surprisingly Affordable Platform for Personal Projects
This instructions merely follow the points describe on the article aforementioned. I suggest you to read in first.
It all boils down to create the cluster with:
gcloud container --project "dreamdrugs" clusters create "dreamdrugs" \
--zone "us-central1-a" \
--no-enable-basic-auth \
--release-channel "regular" \
--machine-type "g1-small" \
--image-type "COS_CONTAINERD" \
--disk-type "pd-standard" \
--disk-size "10" \
--metadata disable-legacy-endpoints=true \
--scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" \
--preemptible \
--num-nodes "3" \
--no-enable-cloud-logging \
--no-enable-cloud-monitoring \
--enable-ip-alias \
--network "projects/dreamdrugs/global/networks/default" \
--subnetwork "projects/dreamdrugs/regions/us-central1/subnetworks/default" \
--default-max-pods-per-node "110" \
--addons HorizontalPodAutoscaling \
--enable-autoupgrade \
--enable-autorepair
The command above might fail if you haven't enabled the Kubernetes Engine API for the project you created. Just follow the error message suggestion: "Kubernetes Engine API has not been used in project xxxxxxxxxxxx before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/container.googleapis.com/overview?project=xxxxxxxxxxxx then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.".
I got this command by doing the manual cluster creation as described in the article, near the end on the browser google was suggesting me the equivalent API command to the action I was doing through their Web Console.
You will now get this message after creating the cluster:
CRITICAL: ACTION REQUIRED: gke-gcloud-auth-plugin, which is needed for continued use of kubectl, was not found or is not executable. Install gke-gcloud-auth-plugin for use with kubectl by following https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
Simply follow their instructions:
$ gcloud components install gke-gcloud-auth-plugin
$ export USE_GKE_GCLOUD_AUTH_PLUGIN=True
$ gcloud container clusters get-credentials dreamdrugs --region us-central1-a
$ kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
arn:aws:eks:... arn:aws:eks:... arn:aws:eks:...
docker-desktop docker-desktop docker-desktop
* gke_dreamdrugs_us-central1-a_dreamdrugs gke_dreamdrugs_us-... gke_dreamdrugs_us-...
minikube minikube minikube default
rancher-desktop rancher-desktop rancher-desktop
You will need export USE_GKE_GCLOUD_AUTH_PLUGIN=True to your shell profile file ~/.bashrc or ~/.zshrc so it gets set every time you run a new shell if you want to forget about it when accessing the Google Kubernetes Cluster.
What we need to do first is to let our DNS (and many more features) to be managed, free of cost, by Cloudflare.
To do so, register to Cloudflare and click on Add site as soon as you finish registration. Then follow all the steps it describes.
The first step after adding the site is to change your domain nameservers as the Cloudflare article explains. If you are using Namecheap follow their instructions.
Afterwards you will need to get your Global API key in Cloudflare as described in here so Caleb's tool (in the next step) can update the DNS A records automatically every time a node is destroyed or created in the cluster. It's under My Profile > API Tokens > API Keys. You are going to need this when you first deploy the project with the helm install command. So keep it safe.
Build and push Caleb Doxey's image to your Google Cloud's Registry as he describes in his GitHub repository:
git clone [email protected]:calebdoxsey/kubernetes-cloudflare-sync.git /tmp/kubernetes-cloudflare-sync
cd /tmp/kubernetes-cloudflare-sync
# Build and push. In this case my project ID is dreamdrugs, update the "dreamdrugs" portion below accordingly.
docker build -t gcr.io/dreamdrugs/kubernetes-cloudflare-sync:latest .
docker push gcr.io/dreamdrugs/kubernetes-cloudflare-sync:latest
If you have authorization problems when pushing to gcr.io you might want to read about authorizing docker as explained in Google's documentation. A
gcloud auth configure-dockershould suffice.
Then assign a cluster admin role to your user in Kubernetes, which is the one kubernetes-cloudflare-sync is going to use.
# Ensure kubectl is pointing to GKE.
gcloud container clusters get-credentials dreamdrugs --region us-central1-a
# You can check you are indeed in the right context by checking the output of
kubectl config get-contexts
# Create the role binding.
kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user [email protected]
What Caleb's tool is going to do is to keep updating 'A' records that would map kubernetes.your-domain.com to the IPs of the nodes in your cluster. Then all we need is to point the root CNAME to kubernetes and proxy it through Cloudflare:
| Type | Name | Content | Proxy status |
|---|---|---|---|
| CNAME | dreamdrugs.art | kubernetes.dreamdrugs.art | ☁️ Proxied |
| A | kubernetes | 104.154.101.136 | ☁️ Proxied |
| A | kubernetes | 35.194.27.124 | ☁️ Proxied |
| A | kubernetes | 35.238.201.122 | ☁️ Proxied |
One of the perks of Cloudflare is that it offers SSL encryption from the client to Cloudflare just one click away from their dashboard, what way many features like the permissions to get the webcam from the browser will work! (They require either to be accessible in the localhost namespace or to have SSL enabled, read here).
To do this follow these two steps as described in Cloudflare's documentation:
- Go to the SSL/TLS menu under your site.
- Set your SSL/TLS encryption mode to Flexible.
- Click the Edge Certificates and enable Always Use HTTPS so it redirects all requests with scheme http to https.
Since we will be hitting directly our cluster nodes (through Cloudflare) where an NGINX service will be routing the requests. In order to let the traffic in we need to set a rule to the firewall to allow in the HTTPS traffic into the nodes of our cluster:
gcloud compute --project=dreamdrugs firewall-rules create http \
--direction=INGRESS \
--priority=1000 \
--network=default \
--action=ALLOW \
--rules=tcp:80,tcp:443,tcp:8080 \
--source-ranges=0.0.0.0/0
First we need to build all the images that are needed to gcr.io and push them.
If pushing fails because you are not authenticated you might want to do gcloud auth configure-docker.
docker build -t gcr.io/dreamdrugs/backend ./backend
docker push gcr.io/dreamdrugs/backend
docker build -t gcr.io/dreamdrugs/frontend ./frontend
docker push gcr.io/dreamdrugs/frontend
Now you are ready to deploy the project to GKE! Find the Global API Key from your Cloudflare account and let Helm handle the rollouts:
read -s CLOUDFLARE_GLOBAL_API_KEY
helm upgrade --install --dependency-update dreamdrugs deploy/dreamdrugs \
--set ingress=false \
--set backendImage=gcr.io/dreamdrugs/backend \
--set frontendImage=gcr.io/dreamdrugs/frontend \
--set redis.master.persistence.enabled=false \
--set redis.slave.persistence.enabled=false \
--set redis.password=foobar1234 \
--set [email protected] \
--set cloudflare.globalApiKey=$CLOUDFLARE_GLOBAL_API_KEY