Production - [Alerting] Autoscale: Minutes to scale-up from zero machine alert

[dotnet-eng-status]

💔 Metric state changed to alerting

Scale up issue: A queue has been waiting for a machine to scale up for more than 45 minutes, there are no machines in this queue, which could cause a lot of work to get stuck.

Wiki link for investigation and mitigation steps here

• WaitTime {Queue=ubuntu.2204.amd64.open.rt} 53
• WaitTime {Queue=windows.11.amd64.client.open} 51

Go to rule

@dotnet/dnceng, please investigate

Automation information below, do not change

Grafana-Automated-Alert-Id-54aa0d7e647e46ff9e880bf6ae532b99

[oleksandr-didyk]

For windows.11.amd64.client.open - at the time of checking (~ 13.00 UTC) the scaleset for the queue had several instances in the Creating status, so most probably the scaling up was delayed.

For ubuntu.2204.amd64.open.rt - ~18 instances are available, but none are visible in the heartbeats table. Additionally, from the metrics displayed in AzDO it seems that some of the instances were created after the alert fired.

Will monitor the alert and close if no more scaling issues will be encountered

[dotnet-eng-status]

💚 Metric state changed to ok

Scale up issue: A queue has been waiting for a machine to scale up for more than 45 minutes, there are no machines in this queue, which could cause a lot of work to get stuck.

Wiki link for investigation and mitigation steps here

Go to rule

[riarenas]

More #13774. We expect provisioning times to stay stable now.