Skip to content

Commit

Permalink
Add prometheus-gate catalog task to await prometheus range query results
Browse files Browse the repository at this point in the history
  • Loading branch information
iancoffey committed Jul 30, 2020
1 parent 3068d43 commit 4bf86c9
Show file tree
Hide file tree
Showing 5 changed files with 160 additions and 0 deletions.
27 changes: 27 additions & 0 deletions task/prometheus-gate/0.1/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Prometheus Gate

A simple gate task which will query the prometheus API in a loop and await a matching status for N length of time.

Code for the gate container is located at [github.com/iancoffey/prometheus-gate](https://github.com/iancoffey/prometheus-gate).

## Strategy

Currently, it is possible to enforce that a returned range query values all meet min, max and equals comparisons. Soon, p95 and p99 will be added as well.

- *`min`* - enforce a minimum value for the time period
- *`max`* - enforce value does not exceed target during time period
- *`equals`* - enforce a target equals value for the entire time period

## Range Query

The gate can make use of any valid [range query](https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries). This approach works best when the dataset returned is tighly scoped via the range query itself. The gate considers an empty dataset as a failure to be retried.

### Parameters

* **prometheus_endpoint:**: Prometheus API endpoint
* **range_query:**: The range query to use to define SLO
* **range_time:**: The time we want to assert fits the SLO`
* **target_value:**: The target value to assert
* **target_strategy:**: min, max or equals. p95/99 not supported yet
* **timeout:**: Maximum ticker time for gate
* **tick_time:**: How often to try to assert the desired SLO
60 changes: 60 additions & 0 deletions task/prometheus-gate/0.1/prometheus-gate.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
---
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: prometheus-gate
labels:
app.kubernetes.io/version: "0.1"
annotations:
tekton.dev/pipelines.minVersion: "0.12.1"
tekton.dev/tags: gate
tekton.dev/displayName: prometheus gate
spec:
description: >-
A gate task which will query the prometheus API in a loop and await a matching status for N length of time.
params:
- name: prometheus_endpoint
type: string
description: Prometheus API endpoint
- name: range_query
type: string
description: range query to use to define SLO
- name: range_time
type: string
description: time we want to assert fits the SLO
default: "-10m"
- name: target_value
type: string
description: value we will use as threshold
default: "1"
- name: target_strategy
type: string
description: The target_strategy is min,max or equal
default: "min"
- name: timeout
type: string
description: timeout of the gate
default: "3600"
- name: tick_time
type: string
description: how often to draw data from prometheus
default: "1m"
steps:
- name: gate
image: iancoffey/prometheus-gate-cf7c3e662ede0a4e9bc24d37d2af86d6@sha256:c3df9d8400aef7acb76f409d1e347c5ecaa6f28a69df50981c14bfe87add85d7
env:
- name: "PROMETHEUS_ENDPOINT"
value: $(params.prometheus_endpoint)
- name: "RANGE_QUERY"
value: $(params.range_query)
- name: "RANGE_TIME"
value: $(params.range_time)
- name: "TARGET_VALUE"
value: $(params.target_value)
- name: "TARGET_STRATEGY"
value: $(params.target_strategy)
- name: "TIMEOUT"
value: $(params.timeout)
- name: "TICK_TIME"
value: $(params.tick_time)
15 changes: 15 additions & 0 deletions task/prometheus-gate/0.1/samples/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Prometheus Gate Example

When provided with a valid Prometheus API endpoint and an event sink, the sample Pipeline will await there being 0 *kube_endpoint_address_not_ready* metrics for 5 minutes before proceding to the second step, which sends a CloudEvent.

## Params

**prometheus_endpoint**: The API endpoint of your local (or remote) Prometheus API. The gate will use this endpoint to submit a range query every minute.

**promotion_sink**: The URI to an event endpoint that can recieve events.

## PipelineRun

Edit the provided Pipeline run `gated-pipeline-run` to contain your prometheus API endpoint URI and an event sink promotion_sink.

Once it is accurate, run the pipleine with `kubectl apply -f ./pipelinerun.yaml`.
44 changes: 44 additions & 0 deletions task/prometheus-gate/0.1/samples/pipeline.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: gated-pipeline
spec:
params:
- name: prometheus_endpoint
type: string
description: The triggers endpoint to send out completion CloudEvent
default: ""
- name: promotion_sink
type: string
description: The triggers endpoint to send out promotion CloudEvent
default: ""
tasks:
- name: gate-slo
taskRef:
name: prometheus-gate
params:
- name: range_time
value: "-5m"
- name: target_value
value: "0"
- name: prometheus_endpoint
value: "$(params.prometheus_endpoint)"
- name: range_query
value: kube_endpoint_address_not_ready{namespace="default"}
- name: target_strategy
value: equals
- name: send-promotion-events
taskRef:
name: cloudevent
params:
- name: sink
value: $(params.promotion_sink)
- name: eventID
value: "success.1234"
- name: eventType
value: "promoted.event"
- name: source
value: "development-pipeline"
- name: data
value: '{"message": "The gate succeeded. Woo"}'
14 changes: 14 additions & 0 deletions task/prometheus-gate/0.1/samples/pipelinerun.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: gated-pipeline-run
spec:
serviceAccountName: git-rebase-service-account
pipelineRef:
name: gated-pipeline
params:
- name: prometheus_endpoint
value: your.prometheus.endpoint:9090
- name: promotion_sink
value: replace-with-your-uri

0 comments on commit 4bf86c9

Please sign in to comment.