Skip to content

Commit

Permalink
Merge pull request #524 from Wihrt/prometheus-operator-docs
Browse files Browse the repository at this point in the history
docs: Add prometheus-operator tutorial
  • Loading branch information
stefanprodan authored Mar 27, 2020
2 parents 61fd505 + 42c3080 commit 9c46be1
Showing 1 changed file with 184 additions and 0 deletions.
184 changes: 184 additions & 0 deletions docs/gitbook/tutorials/prometheus-operator.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,184 @@
# Flagger with Prometheus Operator

This guide will show you how to use Flagger and Prometheus Operator.
This guide will handle only Blue/Green Deployment with podinfo application

## Prerequisites

Flagger and Prometheus Operator requires a Kubernetes cluster **v1.11** or newer

Install Prometheus-Operator with Helm v3:

```bash
helm repo add stable https://kubernetes-charts.storage.googleapis.com
helm repo update
kubectl create ns monitoring
helm upgrade -i prometheus stable/prometheus-operator \
--namespace monitoring \
--set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
--set fullnameOverride=prometheus
```

The `prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false` option allows Prometheus-Operator to watch serviceMonitor outside of his namespace.

You can also set `prometheus.service.type=nodePort` if you want to have access the Prometheus UI

Install Flagger with Helm v3:

```bash
helm repo add flagger https://flagger.app
helm repo update
kubectl create ns flagger
helm upgrade -i flagger flagger/flagger \
--namespace flagger \
--set metricsServer=http://prometheus-prometheus.monitoring:9090 \
--set meshProvider=kubernetes
```

The `meshProvider` option can be changed to your value, if you want to do something else than Blue/Green Deployment

Install Flagger Loadtester with Helm v3:

```bash
helm repo add flagger https://flagger.app
helm repo update
kubectl create ns flagger
helm upgrade -i loadtester flagger/loadtester \
--namespace flagger
```

Install podinfo with Helm v3:

```bash
helm repo add sp https://stefanprodan.github.io/podinfo
helm repo update
kubectl create ns test
helm upgrade -i podinfo sp/podinfo \
--namespace test
```

## Setting ServiceMonitor

Prometheus Operator is using mostly serviceMonitor instead of annotations.
In order to catch metrics for primary and canary service, you will need to create 2 serviceMonitors :

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: podinfo
namespace: test
spec:
endpoints:
- path: /metrics
port: http
interval: 15s
selector:
matchLabels:
app: podinfo
```
```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: podinfo-canary
namespace: test
spec:
endpoints:
- path: /metrics
port: http
interval: 15s
selector:
matchLabels:
app: podinfo-canary
```
We are setting `interval: 15s` to have a more aggressive scraping
If you do not define it, you must to use a longer interval in the Canary object

## Setting Custom metrics

Prometheus Operator is relabeling for every serviceMonitor, you can create custom metrics to you own filter.

```yaml
apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
name: request-success-rate
namespace: test
spec:
provider:
address: http://prometheus-prometheus.monitoring:9090
type: prometheus
query: |
rate(
http_requests_total{
namespace="{{ namespace }}",
job="{{ target }}-canary",
status!~"5.*"
}[{{ interval }}])
/
rate(
http_requests_total{
namespace="{{ namespace }}",
job="{{ target }}-canary"
}[{{ interval }}]
) * 100
```

You can also use `pod="{{ target }}-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"` instead of `job={{ target }}-canary`, if you want.

## Creating Canary

```yaml
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: podinfo
namespace: test
spec:
provider: kubernetes
targetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
progressDeadlineSeconds: 60
service:
port: 9898
portDiscovery: true
analysis:
interval: 30s
iterations: 10
threshold: 2
metrics:
- name: http-success-rate
templateRef:
name: request-success-rate
namespace: test
thresholdRange:
min: 99
interval: 1m
webhooks:
- name: smoke-test
type: pre-rollout
url: "http://loadtester.flagger/"
timeout: 15s
metadata:
type: bash
cmd: "curl -sd 'anon' http://podinfo-canary.test:9898/token | grep token"
- name: load-test
type: rollout
url: "http://loadtester.flagger/"
timeout: 5s
metadata:
type: cmd
cmd: "hey -z 1m -q 10 -c 2 http://podinfo-canary.test:9898"
```

## Test the canary

Execute `kubectl -n test set image deployment/podinfo podinfo=stefanprodan/podinfo:3.1.0` to see if everything works



0 comments on commit 9c46be1

Please sign in to comment.