turbok8s ("turbo k8s" or "turbo kube") enhances a kubernetes cluster with bolt-on additions to make it "ready to run" your app. Like a turbo on your car, it uses the existing engine (kubernetes) to provide enhanced functionality and performance (the turbo [and other things]). This is essentially a kubernetes cluster, with a logical subset of the CNCF projects "ready to run" your app in a production-like manner. turbok8s gives you "everything you'll probably want" to run your application on kubernetes!
The input to turbok8s is a cluster. The output of turbok8s is a turbo-charged cluster with all the useful stuff ready to go! turbok8s is cluster and cloud-provider agnostic, provided the minimum hardware requirements are met.
Turbok8s is under active development. The hand deploy guide serves as a tutorial and the only way to set up turbok8s currently.
The purpose behind turbok8s is to provide an easy way run your application in kubernetes, with perfect parity across all of your environments, while also removing the chore of installing the same services over and over. If you are deploying an application to Kubernetes, the mission of turbok8s is to give you all the tools you need to do that with minimal configuration, and be up in running in minutes, while also being prod-ready and ready to scale. turbok8s in its default form gives you "everything you'll probably want" to run your applicaiton.
While cloud providers may have load balancers to use, none of them work the same. While local development tools like minikube and microk8s may have addons you can install, these don't work the same and often aren't avaliable anywhere else. Relying on even a single tool like this makes your tech stack unportable, locking you into a specific vendor or toolset.
turbok8s gives you the ability to run anywhere, on anything, on your own terms. It's worth noting that turbok8s is transparent - You can use it but you could also get to the same point yourself installing the things that turbok8s installs. turbok8s is merely an opinionated method of installing "everything you'll probably want", perhaps with some utilities thrown in. The only platform you should be locked into is Kubernetes!
Constraints and criteria for turbok8s:
- Runs the same locally or in the cloud, for parity across environments (
dev,staging,prod) - Optimized for minimal cost, reduce and/or eliminate monthly variable costs.
- Lightweight. Run on your raspberry pi
- "bolt-on" design. Add to any cluster
- Grab only the components you want, add other components at any time
- Leverage operators wherever possible for a self-managing experience
- Do not use any proprietary solitions (such as cloud load balancers). Be portable to any cluster
- Ready to scale. Work in single-master single-node environments, as well as multi-node HA environments, as well as be able to upgrade from the former to the latter
Being opinionated, we prefer installing things into Kubernetes via certain methods. The order, by precedence:
- Operators installed via the Operator Lifecycle Manager
- Operators installed ad-hoc
- Created through kustomize
- Created through raw manifest
- Helm
Assuming you run your cluster on some type of public cloud or Virtual Server Provider (VPS), turbok8s aims to give you predictiable, fixed monthly costs, which are especially crucial for startups, small teams, and solo devs. Variable monthly costs are one of the primary reasons we believe so strongly in self-hosting wherever possible on k8s! turbok8s strives to give you a fixed monthly operational cost, which only increases when you are actually ready to scale, while still remaining fixed and your desired scale.
We use vultr for the development of turbok8s. We aren't endorsed by them but we do prefer using VPS over public cloud. The current form of turbok8s runs on the smallest-possible vultr cluster: a 2vCPU 4GRAM single node, which is managed by vultr's kubernetes control plane (think GKE or AKS). In it's original form, this cost $20/mo because turbok8s also used a load balancer provider by vultr: $10/mo for the node, $10/mo for the LB.
This turned out to be a great driver and motivator for the philisophy of turbok8s: we could do better! This is what led us to purchasing a public IP for $3/mo and managing it with metalLB instead. The entire setup now only costs $13/mo! This is obviously a minimal form, being single-node, however due to using vultr's managed control plane this is an HA setup and can absolutely scale when the need arises.
The case study and the development of turbok8s itself demonstrate several things:
- Hard numbers showing that managing something yourself with kubernetes can be cheaper We saved $7/mo by managing out own LB. While this is a small number, it accounts for 35% of the cluster's cost. Now just imagine extrapolating that across an org that's spending tens of thousands of dollars in the cloud each month
- The ability to eliminate variable monthly cost. The entire setup costs exactly $13/mo. No more, no less
This has not been extensively tested yet to find out the limits, but everything runs just fine on a 2CPU 4G RAM system.
- turbok8s is tested and configured with calico for its CNI. This doesn't mean others won't work, but in the name of "picking one to get things working", Calico is assumed and guides and default configuration will reference it
We are strong believers that you should strive to self-host just about everything. Soapbox can be found here and here. That said, there are some things where it does make sense to swallow your pride and pay, for the cost of doing it yourself will likely far outweight the benefit. turbok8s aims to give you as many tools as possible to self-host, while assuming that some basic things do indeed exist outside of the cluster.
- Internet Connection
- Have to reach your services somehow!
- In all seriousness, turbok8s will probably work just fine for air-gapped environments (for the services that make sense of course) but this hasn't been attempted/tested
- DNS provider
- turbok8s assumes that you have a DNS provider somewhere that you can create A-records and CNAME's on to point to your cluster. There is a minimal amount of configuration, so click-ops is fine
- Probably doesn't pan out to become your own registrar
- The cluster itself
- BYOC (Bring Your Own Cluster)
- Git repo
- turbok8s assumes that you have a place to store your configuration somewhere external to the cluster
- Public IP address
- If you're using metalLB, turbok8s assumes you bring your own IP addresses
turbok8s is not perfect. Tools included here are not any better or worse than alternative tools. They are merely the ones we find ourselves reaching for the most often, and what we believe give you the simplest, most effective setup for success.
turbok8s would not be possible without all of the wonderful tools upon which it relies!
There are various ways to get started with turbok8s. For an in-depth, manual, and totally-configurable experience that still takes under an hour see hand deploy.
Because most personal machines aren't directly exposed to the internet, there are only a small subset of useful services that are currently spun up locally, however the ultimate goal is to be able to run everthing locally so you can mirror production.
See tekton for how to install (one-liner)
A prod-like cluster is one that is accessible in some manner over the public internet, probably running in a public cloud, VPS, or data center spomewhere.
All services can be deployed
This is the list of services turbok8s plans to support. A checkbox indicates this is working in some form. This is a living list, and "everything you'll probably want" may change in the future!
- StackGres
- Horizontally-scaling Postgresql DB
- Also considering https://operatorhub.io/operator/cloudnative-pg
- MetalLB
- Manage both public and private IP addresses
- Works with the ingress-nginx controller to allow virtual hosting of infinite domains behind a single public IP address
- ingress-nginx
- Manage routing to services
- Set up for virtual hosting
- cert-manager
- Automatic TLS cert management
- ArgoCD
- GitOps CD
- Tekton
- Creation of push-based CI/CD pipelines and task runner
- Kaniko
- Automatic image building
- Harbor
- Artifact and image repository
- Rook
- Object, block, and file storage
- Signoz
- Logs, metrics, alerting
- Kubernetes Dashboard
- See the state of the cluster in a single pane of glass
- Hashicorp Vault
- De-facto standard for secrets management
- Zero Trust Solution
- TBD, access private services in the cluster without jumping through hoops
Managed by Pulumi on Vultr - use whatever you see fit for your own clusters!
kubectl apply -f stackgres-operator.yml
Software load balancing so we can do virtual hosting of a theoritically infinite amount of sites. This load-balances a single public IP, which ingress-nginx then uses to route traffic based on the actual domain and route
Ingress controller. Examines URL's for domain and path. Works with metal-lb, and in this configuration allows us to virtual host as much as we want behind a single public IP
Also considering Traeffic but not yet explored
Installed statically with kubectl apply ... per instructions, uses cluster-issuer-acme.yaml to create a ClusterIssuer, which will reach out to Let'sEncrypt to sign Certificate Requests.
Argo is built on the gitops engine.
CI tool.
This is not an operator itself, a tool that can be used in the cluster for building images. Use tekton pipelines to run kaniko
Artifact registry. This is where we will be able to store images. Major perk of storing images in-cluster is they'll always be cached, no network latency for pushing or pulling. Also authentication is simpler. Kaniko can push here
This will deploy/manage Ceph (block storage) and MinIO (object storage).
MinIO is required for Stackgres to work so we can create an S3-compatible object store in the form of a PV for a StackGres Postgres cluster instance to consume
Deployed with manifest one-liner:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml
Also wrote a small serviceaccount with clusterrolebinding to cluster-admin for some quick n' dirty access, deployed from k8s-dashboard-sa-and-binding.yaml
Study showing that Vault is the way to go: https://github.com/derailed pretty much what everyone's using. However the awesome thing is that they've released an operator for it now, which simply interfaces with normal k8s secrets, so it's designed to be "install and use" and way less operational overhead for hosting.
CNI of choice. Calico is one of the few places where an operator does not make as much sense since a CNI operates at lower-level than app-level (core infrastructure), and most clusters will already have a CNI in-place rather than one managed in-cluster with an operator. Turbok8s assumes you already have the Calico CNI installed in your cluster (or you can start it locally with Minikube)
Organizations certified to help you with your turbok8s installation, and Kubernetes in general.
Love Kubernetes? Love turbok8s? Become a SeedStack Certified Vendor!