balance chunks in CI workflow

[bernt-matthias]

fixes: #2959

FOR CONTRIBUTOR:

• - I have read the CONTRIBUTING.md document and this tool is appropriate for the tools-iuc repo.
• - License permits unrestricted use (educational + commercial)
• - This PR adds a new tool or tool collection
• - This PR updates an existing tool or tool collection
• - This PR does something else (explain below)

[bernt-matthias]

• 1h 28m 4s now .. 42 test(s) out of 3055.
• 3h 11m 15s before .. 36 test(s) out of 3171.

Happy about the time diff, but I do not understand the diff in the number of tests ... ?

[mvdbeek]

Maybe the chunking logic isn't quite right? Can you compare the tool ids to see what is missing ?

[bernt-matthias]

/run-all-tool-tests

[bernt-matthias]

Lets run the workflows on the same commit... Also I had to re-run and this led to test included multiple times.

[gxydevbot]

Find all tool test results here

[mvdbeek]

3179 tests now 😌

[bernt-matthias]

Yeah, odd. I started to look in the json files. Considering the test ids I find duplicated entries in the results of the original version of the workflow. In the current run essentially all from the samtools folder:

e.g.

"functional.test_toolbox.TestForTool_samtools_view.test_tool_000021"
"functional.test_toolbox.TestForTool_samtools_view.test_tool_000021" 

also in the results from the original version of the workflow two tests are missing:

"functional.test_toolbox.TestForTool_data_manager_snpeff_databases.test_tool_000000"
"functional.test_toolbox.TestForTool_data_manager_snpeff_download.test_tool_000000"

[mvdbeek]

So we're testing more tools now (or we have more test results) ?
As long as the current run doesn't skip any tests I think this is ok (a little weird too, but ok)

[bernt-matthias]

Seems that the current, i.e. the old one, workflow skips the data_manager_snpeff_databases tool. At least I could not find it in the merged json / html.

[bernt-matthias]

And the results contain all tests of samtools twice. So there is a pattern. Maybe something with the shed.yml files?

[bernt-matthias]

ci_find_repos searches for shed.yml files. For samtools there is one in the main dir (aiming at defining the suite) and one per tool.

No idea how to fix. Suggestions?

Not sure why data_manager_snpeff_databases is missing in the old tests .. could not spot a problem in the yml file.

[mvdbeek]

No idea how to fix. Suggestions?

Merge this PR ? ;)

[bernt-matthias]

Sounds like a good idea. Was just wondering if this loop really gets all tools https://github.com/galaxyproject/planemo/blob/57e26f5ecf880a552cb956f3e3287995d4c2aead/planemo/commands/cmd_ci_find_tools.py#L31 .. or maybe skip invalid / malformed tools ... but I guess this is tested

[mvdbeek]

That could well be. Happy to merge this if you restore the schedule.

[bernt-matthias]

Squashed and restored the on statement to weekly

[mvdbeek]

Amazing, thanks for figuring this out!