Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to Safari Technology Preview 94 on macOS Mojave #19750

Merged
merged 2 commits into from
Oct 17, 2019
Merged

Conversation

foolip
Copy link
Member

@foolip foolip commented Oct 17, 2019

No description provided.

@foolip
Copy link
Member Author

foolip commented Oct 17, 2019

infrastructure/ expectations need to be updated, but in the meantime https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=34727 is a full run in progress.

@foolip
Copy link
Member Author

foolip commented Oct 17, 2019

Some views to check results:

From Homebrew/homebrew-cask-versions#8005 we can see that STP 94 is 14609.1.6.1, see web-platform-tests/wpt.fyi#1543 about exposing that in wpt.fyi. Importantly, it's a newer revision that Safari 13, which STP 82 was older.

The last comparison is the most relevant here. Overall green numbers are larger than the red numbers, but there seeming regressions in infrastructure/, pointerevents/, uievents/ and webdriver/ that are likely related and real. @burg you're already looking into those, right? Would WebKit bugs be helpful?

Note that some amount of differences in views like this will be due to flakiness.

@foolip foolip marked this pull request as ready for review October 17, 2019 10:35
@@ -0,0 +1,31 @@
[bless.html]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LukeZielinski as an FYI, this expectation file is probably the best example so far of something that's time-consuming to write manually. I had to copy the section the right number of times and copy all the names from log output until I got it right...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Context: #12697, and it's easier to update expectations to match actual results in Chromium, or at least I know how to do it there.

@stephenmcgruer
Copy link
Contributor

In terms of the regressions from 82 to 94, some of them seem to be MISSING results. I noticed some interpolation tests had regressed which was surprising, but on inspection they're just missing: https://wpt.fyi/results/css/css-grid?q=seq%28status%3Aok%7Cstatus%3Apass%20status%3A%21ok%26status%3A%21pass%29&run_id=313540021&run_id=342290025

Not sure what this means, can you take a look @foolip?

@foolip
Copy link
Member Author

foolip commented Oct 17, 2019

The way to start debugging that is to download the logs from https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=34727 and grep for grid-template-columns-interpolation.html. Looks like it appears in "all tests: Safari Technology Preview 3" and that job seems to end as expected.

Turns out this is just an artifact of applying the seq(status:ok|status:pass status:!ok&status:!pass) filter, because that excludes all the subtests that don't match the filter. If you remove that, you'll see that actually a bunch of subtest names have changed, overall passrate is unchanged.

Do the subtest names depend on the implementation here? If so, that should be fixed :)

@stephenmcgruer
Copy link
Contributor

Do the subtest names depend on the implementation here? If so, that should be fixed :)

Turns out they do, sent #19759

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants