tests: group tests into several link-farms (10 tests per farm) #2041
Conversation
In some edge cases, expected and result can't be directly used in string interpolation, so pass them through `lib.generators.toPretty` to be safe.
Splits up a list into many sub-lists based on the given max-size. e.g. ```nix groupListBySize 2 [ 1 2 3 4 5 ] => [ [ 1 2 ] [ 3 4 ] [ 5 ] ] ```
Attempt to find a middle ground between having a single link-farm and one test per file. This groups up to ten tests per (roughly) 28 link-farms.
Since we're using link-farms, we need to access the underlying tests as `passthru.entries.*`.
|
Initial feedback from buildbot is good: built in 4.5m, compared with the last few builds which took ~1h:
Obviously not statistically significant, since this is only one build. |
I think it was also faster at this time of day during the last few PR's that I built. Might be good to look at what the runs look like from similar time periods. These are from around this time of night a few days ago. So it would still be several times faster. |
|
Running
This isn't very scientific though, because I have several browser tabs (etc) open in the background. |
|
I ran the buildbot job again a couple of times and I've ran just the nix-eval-jobs command on the CI host a few times, it all looks good. Thank you for working on this. |
There was a problem hiding this comment.
From the bits that I evaluated, I could see marginal memory footprint increase locally but the performance in buildbot looks magnitudes better. Everything seems to function fine on my desktop and laptop (x86 linux and arm darwin). As long as it looks alright for team maintaining buildbot it seems to be a good change.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
groupListBySizeIn an attempt to strike a balance between having one very intensive test and many smaller tests at the
checks.<system>.*level, this PR groups the tests into bundles of 10, currently resulting in 28checks.<system>.test-*link farms and 46 total checks.Quoting #2015 (comment):
We can always dial the group size up/down to find the right balance, but 28 groups of 10 feels like a good starting point (112 groups of 10, when multiplied by the 4 systems).
As per #1878 (comment), we can test memory usage locally using:
nix run nixpkgs#nix-eval-jobs -- --workers 1 --check-cache-status --force-recurse --flake .#checkscc @zowoq
cc @GaetanLepage
@traxys I've had to re-work some details of your
testsshell script, supplying it with a pre-generatedtests.jsonfile, which looks like (full file: tests.json):{ "examples": "test-1.passthru.entries.examples", "extended-lib": "test-1.passthru.entries.extended-lib", "modules-clipboard": "test-1.passthru.entries.modules-clipboard", "modules-commands": "test-1.passthru.entries.modules-commands", "modules-editorconfig": "test-1.passthru.entries.modules-editorconfig", "modules-files": "test-1.passthru.entries.modules-files", "modules-filetypes": "test-2.passthru.entries.modules-filetypes", "modules-keymaps": "test-2.passthru.entries.modules-keymaps", "modules-lua-loader": "test-2.passthru.entries.modules-lua-loader", "modules-options": "test-2.passthru.entries.modules-options", "plugins-bufferlines-barbar": "test-3.passthru.entries.plugins-bufferlines-barbar", "plugins-bufferlines-bufferline": "test-3.passthru.entries.plugins-bufferlines-bufferline", "plugins-colorschemes-base16": "test-3.passthru.entries.plugins-colorschemes-base16", "plugins-colorschemes-catppuccin": "test-3.passthru.entries.plugins-colorschemes-catppuccin", }I think doing this via
passthruavoids IFD, but I'm unsure how much cost this has added to evaluating/building the devshell.Note
When using the
testscommand, you should no-longer pass a"test-"prefix.I figured the prefix was redundant in the context of
passthru.entries.