Skip to content

Local Develoment

Jump to bottom
Ramon Blanquer edited this page Mar 8, 2023 · 33 revisions

Rancher Desktop

For development I recommend using (instead of the propietary Docker Desktop) the 100% open-source Rancher Desktop which gives us both Kubernetes provisioning and container management (image building and pushing, running containers outside the cluster, and more).

An open-source desktop application for Mac, Windows and Linux. Rancher Desktop runs Kubernetes and container management on your desktop. You can choose the version of Kubernetes you want to run. You can build, push, pull, and run container images using either containerd or Moby (dockerd). The container images you build can be run by Kubernetes immediately without the need for a registry.

Follow the Rancher Desktop installation instructions. If on a macOS you can brew install rancher-desktop.

Since we will be using Skaffold for development we will have to use dockerd instead of containerd as the container engine used by Rancher Desktop:

Rancher Desktop > Preferences > Container Engine > dockerd (moby)

If you want to know why I prefer Rancher Desktop over minikube check Why not minikube? on the FAQs section.

After installing it configure your docker CLI (if you don't have it you must install it, e.g. brew install docker) to point to the Rancher Desktop docker daemon:

$ docker context ls
NAME              DESCRIPTION                               DOCKER ENDPOINT                               KUBERNETES ENDPOINT                ORCHESTRATOR
default *         Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                                                      swarm
desktop-linux                                               unix:///Users/ramon/.docker/run/docker.sock
rancher-desktop   Rancher Desktop moby context              unix:///Users/ramon/.rd/docker.sock           https://127.0.0.1:6443 (default)

$ docker context use rancher-desktop
rancher-desktop
Current context is now "rancher-desktop"

$ docker contet ls
NAME                DESCRIPTION                               DOCKER ENDPOINT                               KUBERNETES ENDPOINT                ORCHESTRATOR
default             Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                                                      swarm
desktop-linux                                                 unix:///Users/ramon/.docker/run/docker.sock
rancher-desktop *   Rancher Desktop moby context              unix:///Users/ramon/.rd/docker.sock           https://127.0.0.1:6443 (default)

Setup Requirements

If you are using Rancher Desktop you can skip reading this, but otherwise you have to be careful when choosing the Kubernetes provider (k3d, k3s, minikube, kind, ...).

The production setup of this project relies on the idea of having DNS routes on Cloudflare that point directly to the Kubernetes node IPs at ports 80 and 443. That means that if you want to mimic production your nodes should be reachable by IP and your node ports be reachable.

I stay away from ingress because having it means the cloud provisioner would place a load balancer in front and that costs considerable money for toy projects like this one. If you want to understand the architecture of this project I took it from this article where we have 1 NGINX pod running on each node (that's what DaemonSet can do) bound to the host network directly to save load balancer costs.

More on that on Production Setup - Overview.

However if you still want to use whatever ingress controller you might have, make sure it is running in the cluster, and deploy the Helm chart with the value ingress to false, see Proxying: Ingress or DaemonSet with NGINX for more.

Skaffold

Used to ease the development Kubernetes workflow. It watches for changes, and judges if those changes require a Docker rebuild of the image or simply some files can be copied from a local source to a destination container location (updating all the pods), and not only that, but updates to align with any other infrastructure change involved (e.g. Kubernetes manifest files).

Continuous Development

First make sure your minikube machine is running with minikube start and ensure kubectl is pointing at the right Kubernetes context (which is minikube's):

$ kubectl config get-contexts
CURRENT     NAME                 CLUSTER               AUTHINFO             NAMESPACE
            arn:aws:eks:...      arn:aws:eks:...       arn:aws:eks:...
            docker-desktop       docker-desktop        docker-desktop        
*           rancher-desktop      rancher-desktop       rancher-desktop       
            minikube             minikube              minikube             default

Make sure you don't run it to a production cluster! If it's not pointing to minikube you can run:

$ kubectl config use-context rancher-desktop
Switched to context "rancher-desktop".

Then change directory to project root you cloned the repo into.

$ skaffold dev

Now visit https://localhost.

Invalid SSL Certificates (Self-Signed)

Since the HTTPS in local development uses a self-signed certificate the browser displays red windows.

In the case of Google Chrome the NET::ERR_CERT_INVALID message is shown. You can continue by opening up the "Advanced options" and clicking "Continue" or by typing on that screen thisisunsafe.

Debugging

Some insight on how to do remote debugging using a Kubernetes system.

Python

Tools:

First set a breakpoint with import remote_pdb and remote_pdb.set_trace(), when that line hits on the Kubernetes pod's container, it will wait, then you can attach to it and walk through.

Check the logs to see what port it connects to:

[skaffold-helm-dreamdrugs-backend-xxx backend] RemotePdb session open at 127.0.0.1:34769, waiting for connection ...

To interactively walk through the lines of the application follow these steps:

# Find what's the pod running the backend.
[you@your-machine]$ POD_NAME=$(kubectl get pods --selector=app=skaffold-helm-dreamdrugs-backend -o jsonpath='{.items[*].metadata.name}')

[you@your-machine]$ echo $POD_NAME
skaffold-helm-dreamdrugs-backend-xxx

# Run an interactive bash (if it has it, otheriwse use `sh`) shell into that container.
[you@your-machine]$ kubectl exec $POD_NAME -t -i bash

# You can now attach to the debugger!
# Note: The Up and Down arrows work! Thanks to the described steps on python-remote-pdb's README.
[root@ skaffold-helm-dreamdrugs-backend-xxx]$ rlwrap socat - tcp:127.0.0.1:34769
[16] > /dreamdrugs/backend/app.py(55)dream()
-> data = request.get_json()
(Pdb++) l
 50         """
 51         import remote_pdb
 52         remote_pdb.set_trace()
 53
 54         # Data from POST request as JSON.
 55  ->     data = request.get_json()
 56
 57         # Get the model instance and image path.
 58         model = MODELS[data.pop('model')]()
 59         input_base64_image = data.pop('image')

NodeJS

For the frontend server side debugging, we could use a setup similar to the one described in the following article. I still have to figure it out!

FAQs

Why not minikube?

For few reasons:

  1. Rancher Desktop uses k3s, which is a production-ready distribution, and minikube is intended only for development. For instance, TrueNAS SCALE's app catalog uses k3s.
  2. It runs on VM, and that allows us to connect directly to its ports. I'm on a macOS and using minikube with using the virtualbox driver breaks. If you used minikube with the default docker driver or k3d since it uses Docker to build the Kubernetes distribution it makes it super difficult to reach directly to the node port without an ingress, proxy or service.
  3. If using minikube you would need to access the node by the minikube ip, but with Rancher Desktop I can simply skaffold dev and then go to https://localhost and http://localhost with no problems.
  4. Rancher Desktop has a UI and helps me a lot to transition away from Docker Desktop.
Clone this wiki locally