-
Notifications
You must be signed in to change notification settings - Fork 577
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add prometheus-gate catalog task to await prometheus range query results
- Loading branch information
Showing
5 changed files
with
160 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
# Prometheus Gate | ||
|
||
A simple gate task which will query the prometheus API in a loop and await a matching status for N length of time. | ||
|
||
Code for the gate container is located at [github.com/iancoffey/prometheus-gate](https://github.com/iancoffey/prometheus-gate). | ||
|
||
## Strategy | ||
|
||
Currently, it is possible to enforce that a returned range query values all meet min, max and equals comparisons. Soon, p95 and p99 will be added as well. | ||
|
||
- *`min`* - enforce a minimum value for the time period | ||
- *`max`* - enforce value does not exceed target during time period | ||
- *`equals`* - enforce a target equals value for the entire time period | ||
|
||
## Range Query | ||
|
||
The gate can make use of any valid [range query](https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries). This approach works best when the dataset returned is tighly scoped via the range query itself. The gate considers an empty dataset as a failure to be retried. | ||
|
||
### Parameters | ||
|
||
* **prometheus_endpoint:**: Prometheus API endpoint | ||
* **range_query:**: The range query to use to define SLO | ||
* **range_time:**: The time we want to assert fits the SLO` | ||
* **target_value:**: The target value to assert | ||
* **target_strategy:**: min, max or equals. p95/99 not supported yet | ||
* **timeout:**: Maximum ticker time for gate | ||
* **tick_time:**: How often to try to assert the desired SLO |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,60 @@ | ||
--- | ||
apiVersion: tekton.dev/v1beta1 | ||
kind: Task | ||
metadata: | ||
name: prometheus-gate | ||
labels: | ||
app.kubernetes.io/version: "0.1" | ||
annotations: | ||
tekton.dev/pipelines.minVersion: "0.12.1" | ||
tekton.dev/tags: gate | ||
tekton.dev/displayName: prometheus gate | ||
spec: | ||
description: >- | ||
A gate task which will query the prometheus API in a loop and await a matching status for N length of time. | ||
params: | ||
- name: prometheus_endpoint | ||
type: string | ||
description: Prometheus API endpoint | ||
- name: range_query | ||
type: string | ||
description: range query to use to define SLO | ||
- name: range_time | ||
type: string | ||
description: time we want to assert fits the SLO | ||
default: "-10m" | ||
- name: target_value | ||
type: string | ||
description: value we will use as threshold | ||
default: "1" | ||
- name: target_strategy | ||
type: string | ||
description: The target_strategy is min,max or equal | ||
default: "min" | ||
- name: timeout | ||
type: string | ||
description: timeout of the gate | ||
default: "3600" | ||
- name: tick_time | ||
type: string | ||
description: how often to draw data from prometheus | ||
default: "1m" | ||
steps: | ||
- name: gate | ||
image: iancoffey/prometheus-gate-cf7c3e662ede0a4e9bc24d37d2af86d6@sha256:c3df9d8400aef7acb76f409d1e347c5ecaa6f28a69df50981c14bfe87add85d7 | ||
env: | ||
- name: "PROMETHEUS_ENDPOINT" | ||
value: $(params.prometheus_endpoint) | ||
- name: "RANGE_QUERY" | ||
value: $(params.range_query) | ||
- name: "RANGE_TIME" | ||
value: $(params.range_time) | ||
- name: "TARGET_VALUE" | ||
value: $(params.target_value) | ||
- name: "TARGET_STRATEGY" | ||
value: $(params.target_strategy) | ||
- name: "TIMEOUT" | ||
value: $(params.timeout) | ||
- name: "TICK_TIME" | ||
value: $(params.tick_time) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
# Prometheus Gate Example | ||
|
||
When provided with a valid Prometheus API endpoint and an event sink, the sample Pipeline will await there being 0 *kube_endpoint_address_not_ready* metrics for 5 minutes before proceding to the second step, which sends a CloudEvent. | ||
|
||
## Params | ||
|
||
**prometheus_endpoint**: The API endpoint of your local (or remote) Prometheus API. The gate will use this endpoint to submit a range query every minute. | ||
|
||
**promotion_sink**: The URI to an event endpoint that can recieve events. | ||
|
||
## PipelineRun | ||
|
||
Edit the provided Pipeline run `gated-pipeline-run` to contain your prometheus API endpoint URI and an event sink promotion_sink. | ||
|
||
Once it is accurate, run the pipleine with `kubectl apply -f ./pipelinerun.yaml`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,44 @@ | ||
--- | ||
apiVersion: tekton.dev/v1beta1 | ||
kind: Pipeline | ||
metadata: | ||
name: gated-pipeline | ||
spec: | ||
params: | ||
- name: prometheus_endpoint | ||
type: string | ||
description: The triggers endpoint to send out completion CloudEvent | ||
default: "" | ||
- name: promotion_sink | ||
type: string | ||
description: The triggers endpoint to send out promotion CloudEvent | ||
default: "" | ||
tasks: | ||
- name: gate-slo | ||
taskRef: | ||
name: prometheus-gate | ||
params: | ||
- name: range_time | ||
value: "-5m" | ||
- name: target_value | ||
value: "0" | ||
- name: prometheus_endpoint | ||
value: "$(params.prometheus_endpoint)" | ||
- name: range_query | ||
value: kube_endpoint_address_not_ready{namespace="default"} | ||
- name: target_strategy | ||
value: equals | ||
- name: send-promotion-events | ||
taskRef: | ||
name: cloudevent | ||
params: | ||
- name: sink | ||
value: $(params.promotion_sink) | ||
- name: eventID | ||
value: "success.1234" | ||
- name: eventType | ||
value: "promoted.event" | ||
- name: source | ||
value: "development-pipeline" | ||
- name: data | ||
value: '{"message": "The gate succeeded. Woo"}' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
--- | ||
apiVersion: tekton.dev/v1beta1 | ||
kind: PipelineRun | ||
metadata: | ||
name: gated-pipeline-run | ||
spec: | ||
serviceAccountName: git-rebase-service-account | ||
pipelineRef: | ||
name: gated-pipeline | ||
params: | ||
- name: prometheus_endpoint | ||
value: your.prometheus.endpoint:9090 | ||
- name: promotion_sink | ||
value: replace-with-your-uri |