No results from Azure Pipelines in 4 days

[jugglinmike]

WPT is currently configured to test two releases of Edge ("dev" and "canary") and two release of Safari (the stable release and Technology Preview) in Azure Pipelines. The results from each browser are uploaded to wpt.fyi, typically at a rate of about once every 12 hours.

Currently, the most recent results from all four browsers are dated 2019-08-08, or four days ago.

This may indicate a problem with the integration of WPT and Azure Pipelines because Edge and Safari are so different and because there has been no interruption in the results collected on the Taskcluster service.

cc @Hexcles @thejohnjansen

[thejohnjansen]

Adding @mustjab as well…

It looks like our end is all working as expected, but the job scheduler is not scheduling new jobs. The last one scheduled was on the 8th.

https://dev.azure.com/web-platform-tests/wpt/_settings/agentqueues?queueId=7&view=jobs

We scheduled a manual job: https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=26919 so we can see if that works, but do you know why the jobs stopped running on the 8th? It looks like everything SHOULD be working.

[jugglinmike]

Is it possible that we've reached some limit on the number of jobs available to our account?

[thejohnjansen]

It looks like we have an issue with build matching. Though I cannot imagine how this would impact the Safari run. In the manual run, we are seeing errors that the build of Edge is supposed to be 78. This indicates to me that the MSEdgeDriver is build 78, but for some reason, the build of Edge is not. Perhaps when the Edge run fails, this also kills the Safari run?

We are debugging the install/uninstall script to see what's going on...

[jugglinmike]

78 is shaping up to be this project's unlucky number.

[foolip]

Might this be because of #17342?

[foolip]

Searching for "Scheduled build" in https://dev.azure.com/web-platform-tests/wpt/_build?definitionId=1 the most recent I see is https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=26593, which is the source of the most recent runs for Edge and Safari on wpt.fyi: https://wpt.fyi/results/?sha=9c1737440f&label=master&max-count=1

After that no scheduled builds. I've sent #18442 to maybe fix this.

[foolip]

#17342 wasn't fixed by my PR, but I've triggered a manual build to see if the configuration is working at all: https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=27211

[foolip]

https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=27211 had 2/4 configurations successfully finish and show up on wpt.fyi:
https://wpt.fyi/results/?run_id=304460009&run_id=283080010

For Safari stable, one of the jobs failed like this:

The agent: Hosted Agent lost communication with the server. Verify the machine is running and has a healthy network connection. For more information, see: https://go.microsoft.com/fwlink/?linkid=846610

I doubt that'd reliably reproduce for that job, but we'll see what happens if scheduled runs resume.

@thejohnjansen I doubt there's any connection between Edge failures and Safari failures, it's set up in a way where there should be no interference and my manual build shows that this still works. I haven't looked into what's wrong with Edge Canary, I'll leave that to you and @mustjab.

[foolip]

With #17342 resolved I believe this issue will be resolved as well, but we'll see. Possibly some of the configurations will be persistently failing, in which case we should file new issues to investigate those.

[Hexcles]

Looks like we just got the full set of four products from Azure, woohoo! https://wpt.fyi/runs

Thanks @foolip !

[jugglinmike]

This is excellent! Good job, @foolip!