[Mitigation] Breaking change in Istio 1.5 telemetry

[stefanprodan]

Istio 1.5 comes with a breaking change for Flagger uses. In Istio telemetry v2 the metric istio_request_duration_seconds_bucket has been removed and replaced with istio_request_duration_milliseconds_bucket. This change breaks Flagger's request-duration metric check that queries Prometheus for istio_request_duration_seconds_bucket.

Mitigation

With Flagger v1.0 is possible to define custom metric checks.

Create a metric template object for istio_request_duration_milliseconds_bucket:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
  name: latency
  namespace: istio-system
spec:
  provider:
    type: prometheus
    address: http://prometheus.istio-system:9090
  query: |
    histogram_quantile(
        0.99,
        sum(
            rate(
                istio_request_duration_milliseconds_bucket{
                    reporter="destination",
                    destination_workload_namespace="{{ namespace }}",
                    destination_workload=~"{{ target }}"
                }[{{ interval }}]
            )
        ) by (le)
    )

In the canary manifests, replace request-duration metric with:

  analysis:
    metrics:
    - name: latency
      templateRef:
        name: latency
        namespace: istio-system
      thresholdRange:
        max: 500
      interval: 1m

Note that you need to upgrade Flagger to v1.0 to use metric templates. Am upgrade guide is available here.