RFC 99: Interop 2022

[foolip]

Rendered (easier to review than the raw Markdown)

Proposals, meeting agenda, etc., are in https://github.com/web-platform-tests/interop-2022

This RFC is ready for review as of Jan 21.

[foolip]

The State of CSS 2021 survey is now open. I hope to use the results of this survey to inform the Compat 2022 effort, in particular the question "Are there any CSS features you have difficulties using because of differences between browsers?"

[scottlow]

Thanks for kicking this off, @foolip! Microsoft remains supportive of the Compat effort and would like to propose that the Microsoft Edge team focus on shipping Subgrid in Chromium as part of Compat 2022. As called out in MDN's 2020 Browser Compatibility Report, CSS Grid usage continues to grow steadily and Subgrid came up multiple times from surveyed developers as a compatibility pain point.

We're excited to see what other areas are nominated/identified by pending surveys. Depending on the areas we collectively agree on for Compat 2022, there may be an opportunity for us to contribute further, but given our existing backlog, we'll need to see how well the two overlap.

[foolip]

Thanks @scottlow, that's great to hear! I've had a look at State of CSS 2021 early results, and Subgrid shows up there in various ways in multiple questions, so I'm fairly confident the case for Subgrid based on web developer sentiment is going to be strong.

[foolip]

After a discussion among browser vendors yesterday, the consensus was to rename this effort to Interop 2022. I've updating the RFC accordingly.

We also created https://github.com/web-platform-tests/interop-2022, and individual proposals will be issues in that repo.

@scottlow I'll go ahead and create an issue for Subgrid over there, quoting your comment above.

[foolip]

In the meeting yesterday (agenda in web-platform-tests/interop#24) we decided to have a hard stop on new proposals. Many good proposals were filed after Nov 7 which was the original timeline in this RFC, and we'll consider those, but no new proposals filed after Nov 18. In other words, web-platform-tests/interop#31 is the last proposal.

Note that https://github.com/web-platform-tests/interop-2022/labels/compat-bug-proposal aren't standalone proposals but part of web-platform-tests/interop#9.

[gsnedders]

I'd rather we actually defined a slightly more concrete scoping of what we want Interop 2022 to be.

At the moment, we have some proposals like web-platform-tests/interop#9 which stray from what I understand to be the original notion of Compat 2021:

Compatibility on the web has always been a big challenge for developers. In the last couple of years, Google and other partners, including Mozilla and Microsoft, have set out to learn more about the top pain points for web developers, to drive our work and prioritization to make the situation better. This project is connected to Google's Developer Satisfaction (DevSAT) work, and it started on a larger scale with the creation of the MDN DNA (Developer Needs Assessment) surveys in 2019 and 2020, and a deep-dive research effort presented in the MDN Browser Compatibility Report 2020. Additional research has been done in various channels, such as the State of CSS and State of JS surveys.

The goal in 2021 is to eliminate browser compatibility problems in five key focus areas so developers can confidently build on them as reliable foundations. This effort is called #Compat 2021.

See also the introduction to Google's DevSAT:

Web Developer Satisfaction, shortened DevSAT, is a Google project to gather information about the needs of web developers. Our goal is to make sure developers are being heard, know what's happening with their feedback, and see issues being addressed. DevSAT is how we ensure we are focusing on the right areas with our Engineering and Developer Relations work. The goal is a more reliable, predictable, and interoperable web platform, enabling developers to invest in and trust it, and to adopt and use new features to grow the platform and their business.

How to mash this with Compat 2021/Interop 2022 is a slightly harder question, especially when there are cases when developers largely just complain about bugs in general (and I don't think "fix all browser-specific-failures in WPT" is a reasonable goal for Interop as a project!). As such we need to be selective about what we include, and for 2021 this was done largely by choosing five specific features. I don't think it's entirely infeasible to say "there are a whole load of important bugs we should target", but then that gets into questions about how we pick bugs, especially given we probably don't want to target overly specific things.

Arguably there are two separate challenges, which I think the current process is showing:

1. In general, there's no broad triage done of developer feedback. A lot of the early parts of the process we're going through are doing fairly fundamental work trying to triage developer feedback, and it's not at all clear that WPT is the right place to be doing that. (In fact, it's almost certainly the wrong place.)
   
   This is work that arguably makes more sense to be done across SDOs, both in terms of prioritising standardisation work (which, admittedly, SDOs tend to have little direct control over) and in terms of advising those around the SDOs what the pain-points of the platform being standardised are when they are implementation concerns.
   
   A bunch of this is rooted in the fact that parts of what has come up from the State of CSS is very much things with ongoing standardisation work, and really should play into prioritises in that space as much as anything else.
2. Then, come out with a shortlist for Interop 2022 based on the output of this triage. Trying to figure out how small (or long) of a list of features we want is an open question, though questions about achievability within a year are reasonable to ask.

[annevk]

I don't think that taking 2021 as the blueprint is the way to go, as that didn't have cross-browser agreement. The issue you point to above highlights issues where Firefox has seen some of the most significant webcompat challenges due to a lack of interop. It's our position that those are worthy of inclusion. I think we should strive for some healthy mix between challenges directly relayed by web developers and issues browsers have found as part of their webcompat efforts.

I'm also not sure looking at web platform SDO is helpful as by-and-large they work on what browsers want to work on, which isn't necessarily related to the most pressing interop problems.

[jgraham]

In addition to what @annevk said, I think the process so far has revealed a set of things where we aren't yet in a position to track implementation compliance against a set of tests, because the standards work isn't done. That includes both some new features that are still in development, and some existing features which have never been given a proper basis in standards and are causing interop problems.

For those things it won't be possible to include them in a 2021-style metric. But we can still agree that those are areas which require priority at the standards level, and track progress on that work with the aim of including them as test-pass targets in a future iteration of this process.

[foolip]

I've deleted the timeline from the RFC and the description. The current "timeline" is that we'll meet every Thursday starting Jan 13 until we have finalized the test lists, and then we'll update this RFC.

[foolip]

I everyone! After many meetings and lots of work by many people, this RFC is now ready for review. Per our process, the earliest it could be accepted is in one week, but we're expecting discussion and going with a two week comment period. As always, this could be extended based on feedback.

I will also create a new issue in the repo for repo watchers who would otherwise miss this comment.

[foolip]

I consider this accepted now.

[foolip]

I've filed #106 about moving the metrics generation code into the web-platform-tests org. I think this is important to make the governance/ownership of the Interop 2022 metrics clearer.