Adds k8s files and CD workflow #11

brunocalza · 2023-10-06T15:52:32Z

Summary

This PR adds yaml files for deploying both the worker and exporter in k8s. It makes use of kustomize for dealing with multiple environments. Right now, we only have one: staging. But it's all setup for adding another one.

It also adds a CD pipeline.

Further details

Architecture explanation

This PR introduces the following architecture

Not sure how I feel about this architecture. But it was kind of forced by how we created and how we are storing the certificate ("the k8s way")

A simpler architecture would be a Load Balancer with TLS termination (we'd have to upload the certificate to the LB) proxying traffic to the K8s service directly. Not sure how easy would be to configure all this TCP proxying but would be simpler.

The benefit of this nginx architecture is: if we end up with more TCP services we don't have to keep adding load balancers, we simply open new ports, and nginx does the port routing to the correct service. But TBH, not seeing that much benefit of capnp and TCP services, so moving to HTTP would probably just make things a lot easier.

CD pipeline

We currently have only one environment: staging. We can deploy to staging by:

adding [staging] to the commit message
merging to main
creating a release

Signed-off-by: Bruno Calza <[email protected]>

brunocalza · 2023-10-06T16:02:17Z

.github/workflows/deploy.yml

+
+jobs:
+  deploy:
+    if: github.event_name == 'release' || github.ref == 'refs/heads/main' || contains(github.event.head_commit.message, '[staging]')


it will only be triggered under these conditions:

new release

merge on main

[staging] in commit message

brunocalza · 2023-10-06T16:03:24Z

.github/workflows/deploy.yml

+      - name: Prepare
+        id: prep
+        run: |
+            if [[ ${{ github.event_name }} == 'release' ]]; then


This if will later be used to select in which environment the deploy will occur. e.g. staging or production. right now, we only have staging environment

brunocalza · 2023-10-06T16:04:48Z

.github/workflows/deploy.yml

+      - name: Deploy to GKE
+        working-directory: ./k8s
+        run: |
+          DEPLOYMENT=$DEPLOYMENT IMAGE_BASIN_WORKER=$CONTAINER_REGISTRY/${{ secrets.GCP_PROJECT }}/textile/basin/basin_worker:sha-$SHA_SHORT IMAGE_BASIN_EXPORTER=$CONTAINER_REGISTRY/${{ secrets.GCP_PROJECT }}/textile/basin/basin_exporter:sha-$SHA_SHORT make deploy


calls make deploy passing the images that were just built and the DEPLOYMENT kind

brunocalza · 2023-10-06T16:05:00Z

.github/workflows/deploy.yml

+      - name: Configure Docker
+        run: gcloud auth configure-docker $CONTAINER_REGISTRY
+
+      - name: Build and push worker


build the worker

brunocalza · 2023-10-06T16:05:06Z

.github/workflows/deploy.yml

+          cache-to: type=local,dest=/tmp/.buildx-cache
+          build-args: CRATE=basin_worker
+
+      - name: Build and push exporter


build the exporter

brunocalza · 2023-10-06T16:05:23Z

.github/workflows/deploy.yml

+            cache-to: type=local,dest=/tmp/.buildx-cache
+            build-args: CRATE=basin_exporter
+
+      - name: Get GKE Credentials


connect to GKE

sanderpick

Question about where / how your deploying the ingress controller but otherwise looks great.

sanderpick · 2023-10-16T16:13:46Z

k8s/ingress-nginx-controller.yaml

in the past i've always setup the ingress controller in a different namespace using the install guide: https://kubernetes.github.io/ingress-nginx/deploy/#gce-gke. looks like your putting it in the app namespace but not sure.

It's inside ingress-nginx, if you look at the yaml, all resources have a namespace: ingress-nginx field

i'm using the same guide, but I had to add a small tweak because the basin-worker is using port 3000

Oh nice. If easier, we could just change the port number, but no strong pref

Signed-off-by: Bruno Calza <[email protected]>

brunocalza · 2023-10-18T19:00:31Z

k8s/Makefile

+
+deploy:
+	cd ${DEPLOYMENT} && $(KUSTOMIZE) edit set image textile/basin_worker=${IMAGE_BASIN_WORKER} textile/basin_exporter=${IMAGE_BASIN_EXPORTER}
+	cd ${DEPLOYMENT} && $(KUSTOMIZE) build . | sed -e 's/\x27"3000"\x27/3000/g' | kubectl apply -f -


sed -e 's/\x27"3000"\x27/3000/g' is a workaround from a limitation of kustomize build when creating a config map of configTcpServices.env. It does not understand integers as part of a key in a yaml. The problem here is that if we change the value in configTcpServices.env we'd have to change here. I'll leave this way as I think of a better approach

brunocalza · 2023-10-18T19:01:47Z

k8s/base/exporter.yaml

+            "--database-url", "$(DATABASE_URL)",
+            "--export-credentials", "$(EXPORT_CREDENTIALS)",
+            "--bind-health-address", "$(BIND_HEALTH_ADDRESS)",
+           #"--export-schedule", "$(EXPORT_SCHEDULE)",


this is commented on because i could not figure out a syntax in the env file that worked. it's just using the default

brunocalza · 2023-10-18T19:07:36Z

k8s/base/ingress-controller.yaml

@@ -0,0 +1,636 @@
+apiVersion: v1


This file comes from https://kubernetes.github.io/ingress-nginx/deploy/ . I needed to download it and do some tweaks to things that are specific to our setup. Essentially, this file creates an external Google Load Balancer and an nginx controller.

The tweaks:

we're opening port 3000

we're specifying the static IP address where DNS name points to

also a patch to this file is used to build the final nginx configuration

brunocalza · 2023-10-18T19:10:38Z

k8s/base/ingress.yaml

@@ -0,0 +1,14 @@
+apiVersion: networking.k8s.io/v1


This ingress resource is only used in the phase of issuing a certificate. If we were using HTTP, it would be used to route traffic based on hostname and paths. But right now, the nginx is doing the proxing directly to the PODs.

brunocalza · 2023-10-18T19:12:46Z

k8s/base/nginx.tmpl

@@ -0,0 +1,1609 @@
+{{ $all := . }}


I needed to tweak the default nginx configuration to allow SSL termination of a TCP service into a different port and act as a proxy. This is done by mounting this template file into the nginx POD. This solution came from kubernetes/ingress-nginx#636 (comment).

brunocalza · 2023-10-18T19:17:04Z

k8s/staging/ingress-controller-patch.json

@@ -0,0 +1,12 @@
+[


this patch specific to the staging env adds two more args to the nginx controller:

tcp-services-configmap: tells nginx how to deal with TCP connections. Take a look at the configTcpServices.env. This config means, any TCP connection to port 3000, will be proxied to basin-staging/basin-worker k8s Service at port 3000.

--default-ssl-certificate: tells nginx where to find the TLS certificate

brunocalza · 2023-10-18T19:18:48Z

k8s/staging/ingress-patch.yaml

@@ -0,0 +1,21 @@
+apiVersion: networking.k8s.io/v1


this is a patch to the ingress that is not really used but is needed to issue the certificate

avichalp · 2023-10-19T13:50:55Z

k8s/base/exporter.yaml

@@ -0,0 +1,61 @@
+apiVersion: apps/v1
+kind: Deployment


should we use cronJob for exporter instead of Deployment?

hum not sure, the exporter already has a cron-like mechanism inside it. we could move to cronJob but it would mean changing the current behavior of exporter

okay, makes sense to leave it as is then

curious that if we use kind: Deployment would it make the exporter pod to run constantly even though it wouldn't be doing anything?

it would. let's revisit the cronJob option in the next basin iteration... sounds like a nice setup but would require code changes as bruno mentioned

curious that if we use kind: Deployment would it make the exporter pod to run constantly even though it wouldn't be doing anything?

good point!

sanderpick

Great work man! Cool to see this come together.

sanderpick · 2023-10-20T05:03:19Z

k8s/base/exporter.yaml

@@ -0,0 +1,61 @@
+apiVersion: apps/v1
+kind: Deployment


it would. let's revisit the cronJob option in the next basin iteration... sounds like a nice setup but would require code changes as bruno mentioned

[staging] adds k8s files and deploy workflow

bca26bb

Signed-off-by: Bruno Calza <[email protected]>

brunocalza force-pushed the bcalza/k8s branch from a49063c to bca26bb Compare October 6, 2023 16:07

brunocalza commented Oct 6, 2023

View reviewed changes

brunocalza marked this pull request as ready for review October 6, 2023 18:14

brunocalza self-assigned this Oct 6, 2023

brunocalza force-pushed the bcalza/k8s branch 2 times, most recently from d9a6e02 to c9c8bd5 Compare October 11, 2023 15:37

sanderpick reviewed Oct 16, 2023

View reviewed changes

sanderpick approved these changes Oct 16, 2023

View reviewed changes

brunocalza marked this pull request as draft October 16, 2023 18:48

adds ingress and tls termination config [staging]

5d28d48

Signed-off-by: Bruno Calza <[email protected]>

brunocalza force-pushed the bcalza/k8s branch from c9c8bd5 to 5d28d48 Compare October 18, 2023 12:35

adds cache and cluster admin role [staging]

24acf70

Signed-off-by: Bruno Calza <[email protected]>

brunocalza force-pushed the bcalza/k8s branch from e16198c to 24acf70 Compare October 18, 2023 13:25

brunocalza commented Oct 18, 2023

View reviewed changes

brunocalza marked this pull request as ready for review October 18, 2023 19:30

brunocalza mentioned this pull request Oct 18, 2023

adds secure connection to the provider tablelandnetwork/basin-cli#11

Merged

brunocalza requested a review from avichalp October 19, 2023 13:20

avichalp reviewed Oct 19, 2023

View reviewed changes

sanderpick approved these changes Oct 20, 2023

View reviewed changes

brunocalza merged commit 87aba6d into main Oct 20, 2023

brunocalza deleted the bcalza/k8s branch October 20, 2023 14:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds k8s files and CD workflow #11

Adds k8s files and CD workflow #11

brunocalza commented Oct 6, 2023 •

edited

Loading

brunocalza Oct 6, 2023

brunocalza Oct 6, 2023

brunocalza Oct 6, 2023

brunocalza Oct 6, 2023

brunocalza Oct 6, 2023

brunocalza Oct 6, 2023

sanderpick left a comment

sanderpick Oct 16, 2023

brunocalza Oct 16, 2023

brunocalza Oct 16, 2023

sanderpick Oct 16, 2023

brunocalza Oct 18, 2023

brunocalza Oct 18, 2023

brunocalza Oct 18, 2023 •

edited

Loading

brunocalza Oct 18, 2023

brunocalza Oct 18, 2023

brunocalza Oct 18, 2023 •

edited

Loading

brunocalza Oct 18, 2023

avichalp Oct 19, 2023

brunocalza Oct 19, 2023

avichalp Oct 19, 2023

avichalp Oct 20, 2023

sanderpick Oct 20, 2023

brunocalza Oct 20, 2023

sanderpick left a comment

sanderpick Oct 20, 2023

Adds k8s files and CD workflow #11

Adds k8s files and CD workflow #11

Conversation

brunocalza commented Oct 6, 2023 • edited Loading

Summary

Further details

Architecture explanation

CD pipeline

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanderpick left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brunocalza Oct 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brunocalza Oct 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanderpick left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brunocalza commented Oct 6, 2023 •

edited

Loading

brunocalza Oct 18, 2023 •

edited

Loading

brunocalza Oct 18, 2023 •

edited

Loading