The CockroachDB Kubernetes Operator deploys CockroachDB on a Kubernetes cluster. You can use the Operator to manage the configuration of a running CockroachDB cluster, including:
- Authenticating certificates
- Configuring resource requests and limits
- Scaling the cluster
- Performing a rolling upgrade
- This project is in beta and is not production-ready.
- The Operator currently runs on GKE. VMware Tanzu, EKS, and AKS have not been tested.
- Currently the Kubernetes CA is required to deploy a secure cluster.
- Expansion of persistent volumes is not yet functional.
- The Operator does not yet support multiple Kubernetes clusters for multi-region deployments.
- The Operator does not yet automatically create Pod Security Policies.
- Migrating from a deployment using the Helm Chart to the Operator has not been defined or tested.
- The Operator does not yet automatically create an ingress object.
- The Operator has not been tested with Istio.
- Kubernetes 1.15 or higher (1.18 is recommended)
- kubectl
- A GKE cluster (
n1-standard-4
machines are recommended)
Apply the CustomResourceDefinition (CRD) for the Operator:
kubectl apply -f https://raw.githubusercontent.com/cockroachdb/cockroach-operator/master/config/crd/bases/crdb.cockroachlabs.com_crdbclusters.yaml
Apply the Operator manifest:
kubectl apply -f https://raw.githubusercontent.com/cockroachdb/cockroach-operator/master/manifests/operator.yaml
Validate that the Operator is running:
kubectl get pods
NAME READY STATUS RESTARTS AGE
cockroach-operator-6f7b86ffc4-9ppkv 1/1 Running 0 54s
Download example.yaml
and optionally modify the default configuration as outlined below.
apiVersion: crdb.cockroachlabs.com/v1alpha1
kind: CrdbCluster
metadata:
name: cockroachdb
spec:
dataStore:
pvc:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "60Gi"
volumeMode: Filesystem
resources:
requests:
cpu: "2"
memory: "2Gi"
limits:
cpu: "2"
memory: "2Gi"
tlsEnabled: true
image:
name: cockroachdb/cockroach:v20.2.0
nodes: 3
By default, the Operator uses the built-in Kubernetes CA to generate and approve 1 root and 1 node certificate for the cluster.
Before deploying, we recommend explicitly allocating CPU and memory to CockroachDB in the Kubernetes pods.
Enable the commented-out lines in the resources.requests
object and substitute values appropriate for your workload. Note that requests
and limits
should have identical values.
resources:
requests:
cpu: "2"
memory: "2Gi"
limits:
cpu: "2"
memory: "2Gi"
Apply example.yaml
:
kubectl create -f example.yaml
Check that the pods were created:
kubectl get pods
NAME READY STATUS RESTARTS AGE
cockroach-operator-6f7b86ffc4-9t9zb 1/1 Running 0 3m22s
cockroachdb-0 1/1 Running 0 2m31s
cockroachdb-1 1/1 Running 0 102s
cockroachdb-2 1/1 Running 0 46s
Each pod should have READY
status soon after being created.
Get a shell into one of the pods and start the CockroachDB built-in SQL client:
kubectl exec -it cockroachdb-2 -- ./cockroach sql --certs-dir=/cockroach/cockroach-certs
If you want to access the Admin UI, create a SQL user with a password while you're here:
CREATE USER roach WITH PASSWORD 'Q7gc8rEdS';
\q
To access the cluster's Admin UI, port-forward from your local machine to the cockroachdb-public
service:
kubectl port-forward service/cockroachdb-public 8080
Access the Admin UI at https://localhost:8080
. Note that certain Admin UI pages require admin
access, which you can grant to the SQL user you created earlier:
GRANT admin TO roach;
To scale the cluster up and down, edit nodes
in example.yaml
:
nodes: 4
Do not scale down to fewer than 3 nodes. This is considered an anti-pattern on CockroachDB and will cause errors.
Note that you must scale by updating the
nodes
value in the Operator configuration. Usingkubectl scale statefulset <cluster-name> --replicas=4
will result in new pods immediately being terminated.
Apply the new configuration:
kubectl apply -f example.yaml
Perform a rolling upgrade by changing the image name in example.yaml
:
image:
name: cockroachdb/cockroach:v20.2.0
Apply the new configuration:
kubectl apply -f example.yaml
Delete the StatefulSet:
kubectl delete -f example.yaml
Delete the persistent volumes and persistent volume claims:
kubectl delete pv,pvc --all
Remove the Operator:
kubectl delete -f https://raw.githubusercontent.com/cockroachdb/cockroach-operator/master/manifests/operator.yaml