This tutorial will be based on the popular Demo platform provided by Google : The Online Boutique
This tutorial will deploy 2 onlinebouque :
- one version fully instrumented with OpenTelemetry
- one version with no openTelemtry instrumentation
Home Page | Checkout Screen |
---|---|
The following tools need to be install on your machine :
- jq
- kubectl
- git
- gcloud ( if you are using GKE)
- Helm
This tutorial will generate traces and send them to Dynatrace. Therefore you will need a Dynatrace Tenant to be able to follow all the instructions of this tutorial . If you don't have any dynatrace tenant , then let's start a trial on Dynatrace
You will first need a Kubernetes cluster with 2 Nodes. You can either deploy on Minikube or K3s or follow the instructions to create GKE cluster:
PROJECT_ID="<your-project-id>"
gcloud services enable container.googleapis.com --project ${PROJECT_ID}
gcloud services enable monitoring.googleapis.com \
cloudtrace.googleapis.com \
clouddebugger.googleapis.com \
cloudprofiler.googleapis.com \
--project ${PROJECT_ID}
ZONE=us-central1-b
gcloud container clusters create onlineboutique \
--project=${PROJECT_ID} --zone=${ZONE} \
--machine-type=e2-standard-2 --num-nodes=4
git clone https://github.com/isItObservable/OpenTelemetryOperator
cd OpenTelemetryOperator
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin \
--user $(gcloud config get-value account)
kubectl apply -f nginx/deploy.yaml
Since we are using Ingress controller to route the traffic , we will need to get the public ip adress of our ingress. With the public ip , we would be able to update the deployment of the ingress for :
- hipstershop
- grafana
- K6
IP=$(kubectl get svc nginx-ingress-nginx-controller -n ingress-nginx -ojson | jq -j '.status.loadBalancer.ingress[].ip')
update the following files to update the ingress definitions :
sed -i "s,IP_TO_REPLACE,$IP," hipster-shop-otel/k8s-manifest.yaml
sed -i "s,IP_TO_REPLACE,$IP," hipster-shop-nootel/k8s-manifest.yaml
To generate traffic we are currently using K6 with the integration with Prometheus. k6 will write load testing statistics in the the Prometheus server. Therefore we will need to deploy the Prometheus Operator :
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack --set server.nodeSelector.node-type=observability --set prometheusOperator.nodeSelector.selector.node-type=observability --set prometheus.nodeSelector.selector.node-type=observability --set grafana.nodeSelector.selector.node-type=observability
To measure the impact of our experiments on use traffic , we will use the load testing tool named K6. K6 has a Prometheus integration that writes metrics to the Prometheus Server. This integration requires to enable a feature in Prometheus named: remote-writer
To enable this feature we will need to edit the CRD containing all the settings of promethes: prometehus
To get the Prometheus object named use by prometheus we need to run the following command:
kubectl get Prometheus
here is the expected output:
NAME VERSION REPLICAS AGE
prometheus-kube-prometheus-prometheus v2.32.1 1 22h
We will need to add an extra property in the configuration object :
enableFeatures:
- remote-write-receiver
so to update the object :
kubectl edit Prometheus prometheus-kube-prometheus-prometheus
After the update your Prometheus object should look like :
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
annotations:
meta.helm.sh/release-name: prometheus
meta.helm.sh/release-namespace: default
generation: 2
labels:
app: kube-prometheus-stack-prometheus
app.kubernetes.io/instance: prometheus
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/part-of: kube-prometheus-stack
app.kubernetes.io/version: 30.0.1
chart: kube-prometheus-stack-30.0.1
heritage: Helm
release: prometheus
name: prometheus-kube-prometheus-prometheus
namespace: default
spec:
alerting:
alertmanagers:
- apiVersion: v2
name: prometheus-kube-prometheus-alertmanager
namespace: default
pathPrefix: /
port: http-web
enableAdminAPI: false
enableFeatures:
- remote-write-receiver
externalUrl: http://prometheus-kube-prometheus-prometheus.default:9090
image: quay.io/prometheus/prometheus:v2.32.1
listenLocal: false
logFormat: logfmt
logLevel: info
paused: false
podMonitorNamespaceSelector: {}
podMonitorSelector:
matchLabels:
release: prometheus
portName: http-web
probeNamespaceSelector: {}
probeSelector:
matchLabels:
release: prometheus
replicas: 1
retention: 10d
routePrefix: /
ruleNamespaceSelector: {}
ruleSelector:
matchLabels:
release: prometheus
securityContext:
fsGroup: 2000
runAsGroup: 2000
runAsNonRoot: true
runAsUser: 1000
serviceAccountName: prometheus-kube-prometheus-prometheus
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector:
matchLabels:
release: prometheus
shards: 1
version: v2.32.1
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.yaml
kubectl get svc -n cert-manager
After a few minutes, you should see:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cert-manager ClusterIP 10.99.253.6 <none> 9402/TCP 42h
cert-manager-webhook ClusterIP 10.99.253.123 <none> 443/TCP 42h
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
To be able to ingest the Distributed traces generated by our Demo Application in Dynatrace, it would be requried to modify openTelemetry-manifest_daemonset.yaml
with
- your Dynatrace Tenant URL ( your dynatrace url would be
https://<TENANTID>.live.dynatrace.com
) - A dynatrace API token having the right :
Ingest OpenTelemetry traces
. You can also addIngest events
,Ingest metrics
,Ingest Logs
To generate your API token you will need to click onAccess Tokens
( in the left menu) Follow the instruction described in dynatrace's documentation Make sure that the scope Ingest OpenTelemetry traces is enabled.
export DT_TENANT_URL=<YOUR TENANT URL>
export DT_API_TOKEN=<YOUR DYNATRACE API TOKEN>
sed -i "s,TENANTURL_TOREPLACE,$DT_TENANT_URL," kubernetes-manifests/openTelemetry-manifest_daemonset.yaml
sed -i "s,DT_API_TOKEN_TO_REPLACE,$DT_API_TOKEN," kubernetes-manifests/openTelemetry-manifest_daemonset.yaml
kubectl apply -f kubernetes-manifests/openTelemetry-manifest_daemonset.yaml
This version of the hipster-shop has the various instrumentation library added in the image of the various services. All the generated Spans will be exported to the OpenTelemetryCollector deployed with the daemonset mode
kubectl create ns hipster-shop-otel
kubectl apply -f kubernetes-manifests/k8s-manifest.yaml -n hipster-shop-otel
Let's deploy the vanilla version of the hipster-shop ( no instrumentation library deployed in the docker image, except for the services built in GO and C#) All the services that would be instrumented by the operator will send their spans to a openTelemetryCollector deployed using the sidecar mode. We will also deploy the 2 instrumentation objects :
- one specific to Python ( python library is currently not supporting the b3 propagator)
- one instrumentation for the Java and nodejs services.
kubectl create ns hipster-shop-nonotel
kubectl apply -f kubernetes-manifests/openTelemetry-manifest_sidecar.yaml-n hipster-shop-nonotel
kubectl apply -f hipster-shop-nootel/instrumentation.yaml -n hipster-shop-nonotel
kubectl apply -f hipster-shop-nootel/instrumentation_python.yaml -n hipster-shop-nonotel
kubectl apply -f kubernetes-manifests/k8s-manifest.yaml -n hipster-shop-nonotel
Open Dynatrace, Click on the left menu Application & Microserivces / Distributed traces
Look for traces generated for your http://onlineboutique.>.nip.io/ and http://onlineboutique-nootel.>.nip.io/
Let's compare 2 generated traces from :
- without the instrumentation applied by the operator
- with the instrumentation applied by the operator
We can see that we have a bit more spans generated by the manual way of adding the instrumentation. This is related to the fact that we can precise the type of instrumentation library we would like to apply. So we can get more details on the various framework used by our applicaiton.