Latest run upload was 2 weeks ago (Nov 14)

[lukebjerring]

See https://wpt.fyi/test-runs

[mattl]

I've been working to get the dashboard running locally and hitting some issues.

Right now, we're getting incomplete runs which leaves the dashboard empty, so we've been manually tweaking it to show older results.

I'd like to see some better documentation on this, and I need to make a start on that. @jeffcarp -- I see Chrome is lacking from https://ci.wpt.fyi/ -- where is it running?

[jeffcarp]

@mattl Chrome is still running in its own VM (instance-3) - it hasn't been migrated to Jenkins yet: #205

Documentation is definitely lacking, due to the fast pace and the project having only 1 contributor for most of its history. As things settle on Jenkins I'd love to spruce up the docs. Feel free to reach out to me on IRC with any questions, there's a lot of undocumented context I can share. Specifically I can help you log into the Chrome VM.

I've been working to get the dashboard running locally and hitting some issues.

What issues are you running into?

Not sure if this was clear but the wpt.fyi (App Engine) server isn't necessary for debugging test runs. (The test running & result serving projects are intentionally isolated from each other except for the one call the runners make at the end to create a new TestRun in the dashboard.)

[mattl]

So I did a bit more digging around yesterday, and thanks to Jeff pointing me in the right direction with Google Cloud I found the various places Jenkins is running and got runs starting again. They're failing immediately with a few errors including not being able to find kubectl and if I tell it to skip that, it fails because of being unable to find the python files.

Looking at the Dockerfiles, I see that kubectl is installed in Dockerfile.dev but not Dockerfile.jenkins -- so that seems like a reasonable place to start digging into this.

tl;dr: Things are still failing, but they're failing quickly and consistently. I can get stuck into debugging this momentarily.

[foolip]

Mike West noticed this and asked me about it today, so people who like wpt.fyi and are trying to use it are noticing. Updating the issue title to reflect that it's now 2 weeks.

[mikewest]

@mikewest has been reloading the page daily for a while now. I assumed that it wasn't going anywhere over the Thanksgiving week, but was a bit surprised to see it still broken today after a post-turkey Monday in Mountain View. :)

Not a huge deal, but I'd like to have something public to point to that shows two interoperable implementations of some things I'd like to advance through W3C process.

[mikewest]

Ping? Any update on the site status?

[foolip]

@mattl got a new Firefox run this week, and was going to upload a Chrome run today. In the meantime, it looks like we've automatically gotten a new partial Chrome run, which should be deleted. Updates should happen on this issue, but here's also an in-progress postmortem:
https://docs.google.com/document/d/19Gwbsg__DpPLGWrs2doFLQ5o8rBBVArTbevVql1mm8M/edit?usp=sharing

[mattl]

Chrome is getting close, hitting a crash in the script right now.

https://drive.google.com/file/d/1LTRK9uNx_9lJMx7cCZxWPdZSb-ZF-ruo/view

[gsnedders]

@mattl FWIW, that to me looks like a bug in wptrunner, as we should eventually set the result of those WebDriver tests to TIMEOUT if it hangs.

[mattl]

That's helpful. I'll be looking there next :)

[jgraham]

@AutomatedTester was looking at updating the pytest version, which I think he believes helps here? At the same time, the way we handle this isn't perfect since we more or less rely on killing the runner process to do the right thing, and don't attempt to stop pytest itself.

[mattl]

Where we're at today:

• Chrome and Firefox manual builds are working great and we're uploading runs from them semi-daily ("most days but not every day")

• Firefox builds are now being triggered from a new Jenkins install (manually right now, soon to be daily automated and then Chrome will come next)

• Currently running the full test suite in Edge and Safari on SauceLabs, and waiting to see how successful that is. A partial run of just 2dcontext worked great.

Next steps:

• Get successful manual runs from Edge and Safari uploaded
• Get Chrome and Firefox running nicely automated daily builds in Jenkins

[foolip]

Sweet! Given that Chrome and Firefox builds are still manual, would it be much effort to also ensure that they're testing the same commit? Or do you think that this bit will be easier once everything is in Jenkins?

[mattl]

I think it should be easy enough to get that to happen.

Here's the start of manual runs in Jenkins for Firefox:

This is Jenkins invoking a graphical browser on another VM (with a GPU for performance reasons)

[foolip]

Hah, that is so cute :) Try ./wpt manifest-download || ./wpt manifest to make that part go faster.

[jgraham]

Doesn't wpt manifest automatically download if applicable? I thought I implemented that.

[jgraham]

Also note that wpt run already does the manifest download by default, so you're doing a little bit of double work.

[foolip]

@jgraham oh, I didn't know, I just made some guesses.

[mattl]

We have a successful Safari build up on the site.

[mattl]

Very happy to report that Safari is now automatically uploading its results to the dashboard daily.

[mikewest]

I'm really happy to see the Safari results! Will automation for Chrome and Firefox return in the near future? :)

[mattl]

Chrome and Firefox will be back today. I’ve put off those while I focused on getting Edge and Safari running. Edge has run today but not to completion, but I’m pretty sure I know why. Safari has now completed three runs in a row. I’m very pleased with that result. Tomorrow is my last working day this year so I’ll be focused on getting as much automated as possible.

…

-- Matt Lee Sr. Open Web Engineer, Bocoup. https://bocoup.com/about/bocouper/mattl

[foolip]

If the system will be left running unsupervised, can you add in a rule like having run >20k tests or similar so that we don't get very partial runs pushed live?

[mattl]

Good idea. I’ll hack something up.

[foolip]

What is the total number of tests for the latest runs of each now? It's hard to tell from wpt.fyi, and intentionally so :)

[mattl]

Update: Good news -- I had two VMs running the Edge tests, and one passed. Check https://wpt.fyi -- we have a new Edge run for the first time in 36 days.

[foolip]

Oh, sweet, https://wpt.fyi/ looking better than ever!

@mdittmer, can you update https://metrics5-dot-wptdashboard.appspot.com/metrics/ with the latest data to see what that looks like?

[foolip]

@mattl, after a quick look, what still seems suspect is:

• Chrome missing mimesniff/ results
• Safari missing some directories in https://wpt.fyi/css

[mattl]

Chrome is running again right now, as is Firefox.

Re: Safari CSS...

Look at these earlier builds:

https://wpt.fyi/css?sha=95cea0bbd1
https://wpt.fyi/css?sha=13eaad17a4

It could have been a timeout issue as Sauce is presently running both Edge and Safari tests from the same account.

[foolip]

@mattl, looks like things have been running pretty well over the holidays, with the exception of #234 (comment) and Firefox not having a run since Dec 21.

Do you think that this urgent issue can be closed now? If any browser's latest run becomes older than 2 weeks again, then I'd file another emergency issue :)

[mikewest]

Firefox's last run is now more than two weeks old. Chrome's is a week and a half... Maybe not an emergency (in which case, please do close out this issue), but I don't have the impression that things are reliably automated at the moment.

[boazsender]

Your impression is unfortunately correct @mikewest, we do not yet have reliable automation yet. We're beginning work on this again next week and this is our top priority for the quarter.

[boazsender]

Hi @mikewest, we got firefox updated to our most recent available run (January 4th). We have issues with the FF and Chrome build machines that are failing to complete their last step and upload results. We are working to isolate each step in this process to prevent failures like this from happening and to make them more recoverable when they do.

[mattl]

I'm going to close this issue. We have updates working sporadically with older stable browsers, but a new approach is needed. I have identified the work needed here: #394

[foolip]

@mattl, is the old-than-it-seems Chrome run noted in #154 (comment) something that will require #394 to address, or could it be as simple as updating a directory on that VM?

While the improvements are ongoing, is there a maximum age of runs that can generally be expected, if not a strong guarantee? CC @mariestaver

[foolip]

Oh look at me, just talking about what remains to be done. First of all, thank you for all the hard work so far! People (me!) are using the dashboard every day, and every hour closer to current it is (statistically) an hour less waited, or tens of minutes not spent running tests manually.

@mikewest, have your needs been met, or does freshness continue to be a problem? There are some odd things going on in Edge (#383) and Safari (#389) that you need to be aware of if trying to compare results.

[mattl]

@foolip Chrome was running older checkouts than it should have been due to an error by me in the version of run.py on the machine (it wasn't updating the tests repo!) -- it should be back to doing current runs starting today. I will continue to look at it. We should make new issues for outstanding things, as this issue is dragging on a bit. I'll make one for Chrome now. #401

[mariestaver]

@foolip Our goal is for runs to be fresh / current always, BUT while we are still repairing the system from the recent issues outlined in this ticket & elsewhere, we can't set any expectations for a maximum. We will need to finish the work we discussed in our meeting recently in order to stabilize the system to a degree where we can set any maximum. In the meantime, thanks for your patience and help!

[foolip]

Understood, thanks!