web test pipelines are taking up to 1 hour

[phil-davis]

The API test pipelines are finishing quickly (within 15 to 20 minutes). But some web test pipelines are taking up to 1 hour to finish.

The web smokeTest pipeline is taking about 1 hour. That is too long.

• split the full web tests into more parts (that will reduce the elapsed time of each pipeline)
• look at how many web smokeTest scenarios there are. Either split that pipeline into 2 or 3 parts, or reduce the number of smokeTest scenarios.

[saw-jan]

• split tests into more parts (done in PR [full-ci] Update web to 4.0.0 #2353)

[phil-davis]

Note: the "split into parts" thing is not smart at all. It counts the number of test suites that exist and splits them between pipelines without any analysis of their content. It is supposed to be a "reasonable rough way" to make a set of pipelines that finish in a similar time.

The problem is that not every test suite has a similar number/length of scenarios. And in the case of testing with OCIS, a lot of sharing tests are skipped - all the scenarios that test with "sharing to the root folder" are not applicable to OCIS. That results in silly outcomes like:

https://drone.owncloud.com/owncloud/ocis/6227/54/6

+ ./tests/acceptance/run.sh
run.sh: running acceptance-tests-drone
waiting for backend server to start
yarn run v1.22.11
$ cucumber-js --retry 1 --require-module @babel/register --require-module @babel/polyfill --require tests/acceptance/setup.js --require tests/acceptance/stepDefinitions --format node_modules/cucumber-pretty /srv/app/web/tests/acceptance/features/webUISharingInternalUsersToRoot /srv/app/web/tests/acceptance/features/webUISharingInternalUsersToRootBlacklisted /srv/app/web/tests/acceptance/features/webUISharingInternalUsersToRootCollaborator -t 'not @skipOnOCIS and not @skip and not @notToImplementOnOCIS and not @federated-server-needed'
0 scenarios
0 steps
0m00.000s
Done in 4.46s.
Checking expected failures in /drone/src/tests/acceptance/expected-failures-webUI-on-OCIS-storage.md
runsh: There were no unexpected failures.
runsh: There were no unexpected success.

There were 3 test suites to run in that pipeline. But none of those test suites had any test scenarios relevant to OCIS. So that pipeline did nothing useful.

We could think of ways to improve the balancing of the run-time of each pipeline.

[phil-davis]

Even with my comment above, the test scenarios have got slower. We need to investigate why, and fix whatever waiting/wasted-time is happening.

[saw-jan]

In oCIS, we used to have similar drone config as web for UI tests. Maybe that might help.
But it would be better if we can equally split tests into pipelinnes

[phil-davis]

We can explicitly specify the web test suites to be run in each pipeline in oCIS, and organize those so that they almost equalize the run-time of each pipeline. The problem we had with that in the past is that when new test suites are added (or test suites are renamed...) then we have to remember to update the pipelines of web tests in oCIS. It is not such a big thing. Maybe it is worth going back to doing it that way, because the current 10 or 20 parts have big differences in run-time.

But whatever we do about spreading the test suites between pipelines, we also have to work out why the tests are now a lot slower!

[kulmann]

The issue lies in a new implementation of appSideBar.js::activatePanel() in the web acceptance test suite. We had accordions in the past and now have a new sidebar UI/UX concept implemented. I'm digging into the timing issues right now. Will open a PR in web soon.

[kulmann]

PR in web is here owncloud/web#5635, but it looks like I tried to optimize too much. Definitely found the reason for the major CI time increases.

[kulmann]

PR in web owncloud/web#5635 is merged and brought down the CI time to 35 minutes. Should help in ocis as well.

[phil-davis]

The run-time of many web pipelines in oCIS is very different. It would be good to think of a way to make this smarter, and each pipeline have a more similar number of test scenarios.

[phil-davis]

We know that there are various web test suites that have zero scenarios for the oCIS case. Those are the "sharing to root" test suites.

We could enhance the "run parts" logic run.sh in owncloud/web so that the caller can specify a list of test suites that are not to be run. The script that gets the list of available test suites can filter out the "not to run" suites and then split the remaining suites into parts, like it does now.

That will remove the useless "zero scenario" suites from the oCIS test run and might be enough to balance out the pipelines.

Maintainenance of the list of "do not run" test suites will be low. When we add new test suites in web, they will automatically get run in oCIS.