ci.jenkins.io jobs on Windows agents are much slower than 21 days ago

[MarkEWaite]

Service(s)

ci.jenkins.io

Summary

Jobs run on ci.jenkins.io Windows agents are significantly slower today than they were 21 days ago

Reproduction steps

Compare job execution times for the Windows portions of several plugins that I maintain. Dec 23 compared to today shows a large enough increase to cause most of the plugin BOM upgrade pull requests to those repositories to fail with a timeout error. The jobs that failed with a timeout error include:

• Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/apache-httpcomponents-client-4-api-plugin#197
• Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/authorize-project-plugin#210
• Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/basic-branch-build-strategies-plugin#61
• Bump nl.jqno.equalsverifier:equalsverifier from 3.15.5 to 3.15.6 jenkinsci/elastic-axis-plugin#343
• Bump nl.jqno.equalsverifier:equalsverifier from 3.15.5 to 3.15.6 jenkinsci/git-client-plugin#1093
• Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/git-plugin#1547
• Bump nl.jqno.equalsverifier:equalsverifier from 3.15.5 to 3.15.6 jenkinsci/git-plugin#1546
• Bump org.apache.maven.plugins:maven-failsafe-plugin from 3.2.3 to 3.2.5 jenkinsci/gitlab-plugin#1610
• Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/gitlab-plugin#1611
• Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/implied-labels-plugin#180
• Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/markdown-formatter-plugin#130
• Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/nodelabelparameter-plugin#289
• Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/platformlabeler-plugin#1249
• Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/priority-sorter-plugin#328
• chore(deps): bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/schedule-build-plugin#295
• Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/testng-plugin-plugin#270
• Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/versioncolumn-plugin#212
• Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/xshell-plugin#140

There were a few pull requests at that same time that passed, but their build was dramatically longer than the typical build for their repository. The passing builds pull requests included:

• Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/elastic-axis-plugin#342 - 31 minutes when Dec 23 and Dec 20 builds took less than 7 minutes
• Bump io.jenkins.tools.bom:bom-2.401.x from 2705.vf5c48c31285b_ to 2718.v7e8a_d43b_3f0b_ jenkinsci/git-client-plugin#1094 - 62 minutes when prior builds range from 21 minutes to 44 minutes

Usually the update of the plugin bill of materials in those plugins is simple and fast. Current agents seem to be running much slower than agents were running previously.

[MarkEWaite]

Pagerduty is notifying me of performance issues that may be on the publick8s cluster.

[jonesbusy]

Same here. Most of my jobs are timeout after 1h

Example here : https://ci.jenkins.io/job/Plugins/job/hidden-parameter-plugin/job/PR-97/2/console

Takes 15mn to execute some very simple maven tasks

[MarkEWaite]

Another timeout failure at https://ci.jenkins.io/job/Core/job/jenkins-test-harness/job/PR-713/1/

[jonesbusy]

Here also: jenkinsci/coverage-model#64

Was better some hours ago but timeout came back

[dduportal]

Update:

• As per Azure Kubernetes publick8s suffers from SNAT port exhaustion: network slowness #3908 (comment), the network issue with SNAT is gone since at least 12 hours after our fixes
• The ci.jenkins.io agents (Azure VMs and Container Instances) are now using a NAT gateway and as such their egress requests are directly headed towards internet (instead of going through our internal virtual networks).

=> Can you let us know if you see a difference in performances?

Something to note: it seems that all "slow" builds were handled by Azure Container Instance (e.g. Windows builds with the containerInstance parameter set to true in the call to the pipeline library buildPlugin(). Let's have a look to the possible ACI size and performances.

[MarkEWaite]

=> Can you let us know if you see a difference in performances?

I've started the following jobs to compare performance:

• http client 4 - 31 minutes, then 5 minutes
• Basic branch build strategies - 29 minutes, then 18 minutes
• Platform labeler - 32 minutes, then 19 minutes
• PollSCM - 21 minutes, then 6 minutes
• xshell - 28 minutes, then 7 minutes

Each of them had at least one build during the last two weeks that took more than 50 minutes to complete. Builds are definitely faster. No timeouts. Windows machines are within a reasonable range of past performance.

[MarkEWaite]

As far as I can tell from my comparisons, this can be closed. They are likely more things to improve, but the most crucial items that I was monitoring have all been resolved.

@jonesbusy are the results looking good enough for you that we can close this issue?

[jonesbusy]

Last timeout I saw was 3 days ago. I didn't saw any more more recently