Are there gaps in UI test coverage downstream projects need to test specifically? #11164
-
|
Hello! I know from #11105 that there's extensive test coverage for Theia, and specifically UI testing via https://github.com/eclipse-theia/theia/tree/master/examples/playwright - That being said, as with all projects I suspect there are UI testing gaps downstream projects need to know about so they can either contribute back to Theia to help fill in the blanks (best) or test themselves in their own project, with their own visual regression testing suite (sub-optimal). My questions are as follows:
I know it's not necessarily easy to answer but thanks in advance for any input you could provide. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 8 replies
-
|
@anavarre thank you for the discussion.
The framework makes use of unit-tests, integration tests and UI tests (through playwright). That being said, I don't think downstream can trust that all or most elements in the framework are tested, we prioritized some of the more important or critical features, and accept contributions for further coverage. Note that the integration tests also help in this regard, they do not rely on UI so they are more robust and compliment the UI tests.
I do not believe it is measured today, but it is certainly useful to get a better picture of what is tested and covered. At the moment you'd have to go into the sources to see what is tested. cc @JonasHelming do you have any additional information for this item?
I'm not sure I follow. AFAIK we do not test dependencies we use, other than ensuring our monaco editor integration works well, including language features. Do you mind elaborating about this item? |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @vince-fugnitto !
Sure! In the same train of thought, think about syntax highlighting (hypothetically). It's not something that Theia provides OOTB but a dependency that is pulled via I guess what I'm trying to assess here is whether Theia is "fully" covered for what it's responsible for only and doesn't get involved into having extra test coverage for mission-critical elements that are not covered upstream otherwise? And then, is it even possible for downstream projects to know which gaps they need to consider covering in their own test suites? |
Beta Was this translation helpful? Give feedback.
-
Right, everything related to the monaco-editor and our integration is tested as best we can, sometimes at an API level since it is critical. The monaco-editor is the one that supports syntax-highlighting OOTB given that language-support is met through VS Code extensions. We have tests to ensure that for TypeScript for example the language-server works, but we can always enhance to also ensure that syntax-highlighting works as well (even though we work with Theia everyday and would notice pretty quickly that syntax-highlighting gets broken). It is very difficult to cover all UI use-cases, so we most certainly welcome enhancements to the suite, if you or any other contributor would like to help. As a community it would also help other applications based on the framework.
In the past we have either, contributed fixes to upstream issues, found alternatives that work better, or even workarounds if necessary.
Likely the best solution for downstream projects is to integrate our tests into their suite so they can get the bigger picture of what is working, and what is covered. In addition, they can also include tests that are important for their use-cases. |
Beta Was this translation helpful? Give feedback.
-
Exactly. That's very much what I'm trying to understand to see whether what we do that helps us can be reused by others in the spirit of Open Source.
Interesting idea! I'll discuss with my team whether that's something we can do. |
Beta Was this translation helpful? Give feedback.
-
|
@anavarre : The UI tests we currently have do mainly test the page object model, i.e. we test that the frame we provide for Playwright still works correctly. IMHO, a higher UI test coverage makes mostly sense in downstream projects. Due to the nature of Theia as a platform, we do not know how the UI of tools based on Theia looks like and behaves. |
Beta Was this translation helpful? Give feedback.
-
At a high level it does, though in practice it might be complicated to make everyone happy versus having each adopter have their own test suite that best cover their needs. To give you a concrete example, ATM we have a Cypress test suite that we hope is thorough enough (it tests "everything" we can think about, from menu items to panels to installing an extension, etc.) but we end up having to spend a huge amount of time fixing it every time we upgrade Theia due to the fast-pace at which it's moving (which is good! 🙏 ). E.g. when upgrading to 1.25: we updated it in half a day and have spent 2-3 days on understanding changes and fixing them in our test suite. Nobody's fault, it's just how things are. What I was wondering about is: are we reinventing the wheel and testing stuff Theia takes care of already? Shouldn't we be better off contributing back gaps we're seeing to e.g. Theia's Playwright UI test sutie and only focus our test suite on the extra UI elements we add on top of Theia? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, would be the though behind having a common test suite next to the project. Please let me mention again that we do not really have a Playwright test suite in place today, it is really only testing the page object model, it does not significantly cover Theia features. |
Beta Was this translation helpful? Give feedback.
-
|
I think we should start by understanding which test framework would be picked up for this community effort and what would be the scope and LoE down the line. Hard questions need to be answered as well IMHO:
In any case my team will never be given enough cycles to rewrite all our Cypress tests into something else but could certainly evaluate contributing new tests after this decision has been made. I'm first and foremost a Drupalist and my experience there is Drupal core contributors maintain an extensive test suite themselves (the Drupal project is way bigger so I also get this is borderline comparing oranges and apples). This is what ensures high standards of quality and prevent regressions. Maybe what I'm getting at is I understand the maintenance overhead but perhaps the best way to solve this is through extending the visual regression test suite in a way that keeps the maintenance burden bearable without giving downstream projects an incentive at building their own test suite and contribute to the project instead. |
Beta Was this translation helpful? Give feedback.
-
|
About the last paragraph: We actually aim add high test coverage, and these contributions are very welcome in the core repository, but not for end-to-end tests. This is due to their higher flakiness, longer runtime and lower robustness. A page object model like we have for Playwright partly compensates this disadvantages, but not fully. The questions you raise are all very valid, I would like to open these to discussion, I will trigger an initial discussion in the next dev meeting on May 24th. You are gladly invited if you can make it: https://github.com/eclipse-theia/theia/wiki/Dev-Meetings |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately I have a calendar clash tomorrow at 3pm UTC but I'd happily read on the outcome of this discussion, thanks! |
Beta Was this translation helpful? Give feedback.
@anavarre thank you for the discussion.
The framework makes use of unit-tests, integration tests and UI tests (through playwright). That being said, I don't think downstream can trust that all or most elements in the framework are tested, we prioritized some of the more important or critical features, and accept contributions for further coverage. Note that the integration tests also help in this regard, they do not rely on UI so they are more robust and compliment the UI tests.
I do not believe it is measured today, but it is certainly useful to get a better p…