Agenda, Aug 10th 2023

[jgraham]

Here is the proposed agenda for the Aug 10rd meeting:

• Interop 2023 - Blocked dashboard updates
• Interop 2024 Backlog
  • Proposal selection process
  • Proposed timeline Interop 2024 timeline #384
  • Updated focus area proposal template Update focus area issue template for 2024. #385
• Interop 2023 - Test change proposal backlog.

[meyerweb]

Dashboard

James: first agenda item is the dashboard
…2023 dashboard has no updates for experimental builds since 31 July and none from stable builds some time in the middle of July
…I believe the stable build one is a Chrome driver issue, so now it’s downloading the wrong driver
…the experimental problem appears to be with Safari
…so in addition to Azure problem, we’re having the Safari driver failing to start
…been furiously restarting builds, and they keep failing due to various bugs
…Sam, were you looking into the Safari issue?
Sam: I have a tentative plan by increasing the allowable number of restarts via an Azure command line argument
James: I’ll ask on Matrix why you think that’s going to work
Sam: I think it’s just randomly failing to start
James: Okay then. Let me know when you have a PR for that.
Sam: I’m also confused why the dashboard didn’t update a couple of days ago
James: That one happened because I’ve been retriggering things
…Also there’s a bug in the dashboard to due with runs happening either side of midnight
…I believe Daniel’s going to look into this
Sam: I feel like there’s a PR about this, it’s certainly something we’ve come across before
James: A link would be great

[meyerweb]

2024 process

James: Moving on to the 2024 process
…We have a list of things blocking the 2024 launch
…ETA was today, but I’m guessing that’s not going to happen due to not having representation from everyone
…First thing is, proposal selection process
…Apple was going to propose something for that, what’s the status?
Sam: There’s a PR
James: I did not see that PR
Sam: Doesn’t make sense to discuss if nobody’s looked at it
Brian: I looked at it a little bit
James: Could you give a rough overview, Sam?
Sam: For the focus area selection, we start off a very quick elimination round
…Essentially we exclude anything that doesn’t meet some very basic criteria
…If there’s disagreement about something meeting those, we leave it in
…In the second round, organizations choose their top ten proposals to include
…That’s used to filter down further, carrying forward anything anyone chose
…In the final round, choose by collective vote with veto blocking
Brian: There’s a few places you mention Khronos; I was wondering what you had in mind with that
Sam: Nothing; I think the list of groups was largely copy-pasted from the Interop 2023 proposal template
Brian: The first time I see it, it’s under requirements and the next sentence says it has to have high-quality tests in WPT or Test262
…Khronos has their own conformance tests
James: The practical problem we have with non-WPT tests, which is we can’t score them
Brian: Right, that’s what drew my attention
…I think we mean “as long as they’re testable in WPT”
Sam: I think this was a copy-paste
…We decided to allow people to propose things tested in other conformance suites, though we can’t score them
James: I think Philip was confident we could scramble structure to make things possible
…We ended up doing a one-off for a Test262 topic area where we copied over tests to WPT rather than pull data directly
…I’m okay with this being in the proposal
…If someone proposes somethiing from ECMAscript, there’s more uncertainty there
…We’d have to do more work to score directly
…I think we’d just copy tests over, which isn’t ideal
…We’re open to making a plan to score directly from other test bases
Boaz: I’ve been talking to Philip about finding a new maintainer for Test262 and bring things over to WPT, so maybe that will help one day
James: From my point of view, I’m in favor of that in general, but not sure we can do it in time to select proposals in October
James: Next question is, do we want to discuss the proposal selection proposal further, or take an action item for people to read this and properly discuss it next meeting?
Boaz: I think read it and discuss next time, also send a link to the mailing list
James: Good idea

[meyerweb]

Timeline

James: Next up is timeline
…Which Sam’s proposal includes, though it shortens the period by a week
Sam: It’s possible I messed up when doing the timeline
James: So the idea in the PR is that we’d have 3 weeks to ask for proposals, and then a week to ask for changes
Sam: Yes
James: That makes some sense to me
…Seems like a soft deadline
…The main constraint is if you hear about the proposal at the start of the open call period, you have time to write a proposal
…It seems like we have soft consensus on the launch date being 14 September, with the selection process finished by the end of November

[meyerweb]

Proposal template
James: Currently the template is only for topic areas
…Are we still accepting proposals for investigation efforts, or are people assuming there won’t be any?
Boaz: I have a strong feeling that based on the way the accessibility investigation is going, there will be appetite for it next year
James: So there’s an action item here to update the investigation area template
…For focus areas, did people have a chance to look at the PR?
Boaz: I thought it was cool there were privacy and accessibility rationales
…I also like being able to see them rendered, Github forms are cool
James: If other people don’t have immediate thoughts, it would be good if by next week we all had this reviewed
…So, action item on the group is: please review the proposal template by next week so we can be ready to merge it
…I’ll create one for an investigation template
…The question is if people are prepared to commit time to investigation, if that makes sense
…Anything else in terms of 2024 to discuss? We still have an ETA/deadline in two weeks to publish the selection process
…That’s not currently assigned to anyone, and I don’t want to volunteer for scheduling reasons
…Doing this may depend on feedback to the Apple PR

[meyerweb]

Test change proposals
James: I’m somewhat tempted to go through these one at a time, oldest to newest
…(Re: #304) Apparently the spec has been updated and there are new tests; Tim?
Tim: I just recently updated the spec to be more complete and added some tests
…Hopefully when Anne’s back, he can review this and it can be merged
James: To what extent is a change at this point in the year extending the scope of Interop?
Sam: It’s not clear to me what is actually being proposed in the test change proposal
…What is the actual change? This is saying there isn’t enough coverage
Tim: It’s either removing tests if the spec isn’t clear or making tests better
Brian: You have improved tests?
Tim: Yes
Brian: But you want to remove them from Interop?
Tim: Mason proposed that, yes
Sam: The tests we have in Interop have changed in the past few months, right?
Tim: Yes
Sam: So this is about oversight on the changes that have landed on files that are already in Interop?
Tim: If we can’t get better tests, Mason is arguing they should be pulled
Brian: This is being discussed very recently, so I think we shouldn’t decide today
James: No, but it would be useful to have a summary of this as a comment on #304

James: Moving on to #368: Add testcases for that <span contenteditable> is followed by <div>
Tim: We’re missing a Chromium person today
James: Yes, we do need someone from Chromium to comment

James: Next, #369: Add rgb() parsing tests that include non-finite inputs
…I think the last state of this is there’s ongoing discussion about what the specification says, so we probably shouldn’t change Interop
…Sam agreed with that, so I’m tempted to close this and if someone at Chromium gets unhappy, they can reopen it
Sam: Seems good to me
Tim: I agree, the quality of these tests is low and nobody will complain
Brian: I’m not confident about that
James: Oh, someone will disagree, but should they be in Interop?
Brian: I think it’s worth saying it’s not that it’s a bad test, just that there’s no value in adding it to Interop

James: On to #375: Remove WebCodecs tests for AV1 codec
Tim: Currently, Safari’s score is penalized because we don’t support the AV1 codec
…I’d like the score to not be penalized as a result of that
…We could exclude the AV1 tests, or could to ignore “precondition failed”
James: I’m more in favor of removing the AV1 tests
…There was discussion of this in the results analysis and I’m concerned creating conditionals could allow people to inflate scores beyond what they should be
Tim: We do have other tests that fail due to unsupported codecs
James: Process-wise, somebody saying “we want to exclude these tests” always wins
…I hope people don’t abuse that, but in this case it would be pretty legitimate to exclude things from Interop on the basis of not supporting codecs

James: Next is #377: Add min/max function tests
…It seems like Emilio was okay with this, but someone at Google was concerned
Tim: It’s just one test that doesn’t have consensus
…Maybe this is more a spec issue, it’s probably too late this year, I think I’ll just close this out

James: Okay, #379: Remove shadow-dom/focus-navigation/focus-with-negative-index.html from Interop 2023
…Everyone agrees we should remove this, so why haven’t we done it?
…So, action item: unlabel this test
Tim: Mason had reservations
James: He did, but said he supported removing the tests for the time being
Tim: Sounds good

James: On to #380: The expectation of pointerevent_after_target_removed.html does not match the spec
…Status appears to be there’s a discussion about what the spec actually says
…Nobody from outside Gecko has weighted in at all
Tim: I’m all for changing the tests to match the spec, whatever the spec says
James: That seems to be what we don’t know
…Do we know who to ask from Chromium about pointer events?
Tim: flackr, maybe?

James: Let’s do one more, #381: Questionable assertion in 2d-getcontext-options.any tests
…It seems like everybody is agreed on this and there’s a PR open, so we need someone to review the PR
Tim: I’ll ask someone to review it, or I might
James: It should be reviewed by someone not from Mozilla, since we wrote it