One anomaly run should not cause a perf alert.

[github-actions]

Performance change found in the
test: pytorch_image_classification_benchmarks-resnet101-mean_load_model_latency_milli_secs for the metric: mean_load_model_latency_milli_secs.

For more information on how to triage the alerts, please look at
Triage performance alert issues section of the README.

Test description: Pytorch image classification on 50k images of size 224 x 224 with resnet 101. Test link -

beam/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy

Line 34 in 42d0a6e

test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',

Test dashboard - http://104.154.241.245/d/ZpS8Uf44z/python-ml-runinference-benchmarks?orgId=1&viewPanel=7

timestamp: Thu Aug  3 18:30:50 2023, metric_value: 153167.27 <---- Anomaly
timestamp: Wed Aug  2 18:22:25 2023, metric_value: 92238.06
timestamp: Wed Aug  2 13:43:54 2023, metric_value: 72148.02
timestamp: Tue Aug  1 18:37:26 2023, metric_value: 74666.89
timestamp: Tue Aug  1 18:20:33 2023, metric_value: 86736.14
timestamp: Mon Jul 31 18:20:31 2023, metric_value: 72867.26
timestamp: Mon Jul 31 13:34:56 2023, metric_value: 71272.68
timestamp: Sun Jul 30 18:18:44 2023, metric_value: 68671.46
timestamp: Sat Jul 29 18:19:13 2023, metric_value: 70717.79
timestamp: Fri Jul 28 19:27:28 2023, metric_value: 74438.64
timestamp: Fri Jul 28 19:01:35 2023, metric_value: 70168.57

[AnandInguva]

Looks like this is a regression but this could also be a flake. I would wait for 2-3 runs to see if this is a flake or not

[tvalentyn]

this is a bug in alerting. 1 anomaly at the end of the observation should not result in an alert.

[github-actions]

Performance change found in the
test: pytorch_image_classification_benchmarks-resnet101-mean_load_model_latency_milli_secs for the metric: mean_load_model_latency_milli_secs.

For more information on how to triage the alerts, please look at
Triage performance alert issues section of the README.

Test description: Pytorch image classification on 50k images of size 224 x 224 with resnet 101. Test link -

beam/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy

Line 34 in 42d0a6e

test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',

Test dashboard - http://metrics.beam.apache.org/d/ZpS8Uf44z/python-ml-runinference-benchmarks?orgId=1&viewPanel=7

timestamp: Wed Sep 27 18:20:35 2023, metric_value: 99398.89
timestamp: Tue Sep 26 18:25:57 2023, metric_value: 103101.44 <---- Anomaly
timestamp: Mon Sep 25 18:21:32 2023, metric_value: 79205.12
timestamp: Sun Sep 24 18:20:01 2023, metric_value: 75259.23
timestamp: Sat Sep 23 18:20:44 2023, metric_value: 70242.47
timestamp: Fri Sep 22 18:21:10 2023, metric_value: 76958.97
timestamp: Thu Sep 21 18:23:52 2023, metric_value: 83634.29
timestamp: Wed Sep 20 18:27:04 2023, metric_value: 85141.63
timestamp: Tue Sep 19 18:18:56 2023, metric_value: 82870.14
timestamp: Mon Sep 18 18:32:19 2023, metric_value: 78348.73
timestamp: Sun Sep 17 18:18:21 2023, metric_value: 74249.35
timestamp: Sat Sep 16 18:18:52 2023, metric_value: 73371.55