Agenda for Oct 10, 2024

[nairnandu]

Here is the proposed agenda for the Interop team meeting on Oct 10th, 2024

• Closing the call for Interop 2025 proposals
• Mobile testing and Interop 2025
• Timeline to request changes to dashboard for 2025

[nairnandu]

Attendees: @nairnandu, @chrishtr, @nt1m, @ChrisC, @jgraham, @DanielRyanSmith, @stubbornella, @dandclark

Notes

• Closing the call for Interop 2025 proposals
  • nairnandu: PR #862
• Mobile testing and Interop 2025
  • nairnandu: Context, Issues - 706, 732, 738, 788, 845, 870
  • jgraham: current state is that we have results for chrome android, exp only for firefox android. No results for Safari/WebKit. I believe getting Safari stable is relatively possible in the short term. Getting experimental releases for Safari and stable for Firefox would be a bit more work. We are not quite there to show results for all browsers. At the moment, we dont fully understand the diff between desktop and mobile results. If we do accept a proposal for mobile testing, it would act as a forcing function to get it done early next year. Not opposed to considering mobile testing.
  • nt1m: how’s the story for webdriver - especially for events?
  • jgraham: it should work. There might be implementation quality issues, which need to be investigated.
  • chrishtr: with respect to webdriver - does everything work fine for chrome on mobile?
  • DanielRyanSmith: In terms of display, it is definitely available. Not aware of any issues relating to webdriver
  • jgraham: I would say it matches my understanding. That is, I don't expect webdriver issues to be prevalent
  • chrishtr: would be happy to include mobile in Interop 2025 and work on fixing any issues
  • stubbornella: need to discuss this internally before arriving at a position
  • jgraham: another possibility - specific focus areas to be defined by mobile results
  • chrishtr: +1
  • Next step: call for consensus on inclusion of mobile testing, in the next meeting
• Timeline to request changes to dashboard for 2025
  • nairnandu: Please add any request for changes by end of October
  • nt1m: on topic to discuss is if we can stop using aligned runs and always use non-aligned runs
  • DanielRyanSmith: the risk is that some results would be based on a different set of tests
  • jgraham: the challenge, with aligned runs, is that if one browser gets stuck - it messes up the results for everyone. It makes sense to have an option to see the most recent run for each browser, even if it's not aligned. Change dashboard to support showing scores as of arbitrary wpt.fyi run (or latest run) even if not all browsers have results for that run #370
  • stubbornella: 2 sets of users for this - we are one of them who want the latest even if the results are not aligned. The casual visitor would potentially want to see an aligned run
  • jgraham: agree on the 2 use-cases. Default should be the aligned run
  • stubbornella: historical scores have been useful to debug flakiness
  • Next steps: create issues for any dashboard improvements by end of Oct
• stubbornella: URL would benefit from a restructuring of tests. When could we potentially change the tests?
  • dandclark: January would be a good time
  • jgraham: Irrespective of whether we choose URL for next year or not, the right time would be when we have a final conclusion on the Interop 2024 scores
  • chrishtr: an option would be to add new tests and mark them as tentative. Then we can switch them out at the right time so that we dont wait till 2025 launch
  • jgraham: agree with that. However, not sure if the proposal is to add/remove tests.
  • nt1m: it's more about giving added weight to existing tests. We can review during the carryover evaluation
• Interop proposal review - discussion on next steps