Run docker-compose based tests in multiple test workers

[MMeent]

Previously they were run sequentially, which meant a very long runtime. Now, it's reduced by a good factor when the CPU time is available.

Problem

The runtime for https://github.com/neondatabase/neon/actions/runs/12773839881/job/35613450148?pr=10394 is 17 minutes, which is a really bad developer experience.

Summary of changes

We parallelize the tests, allowing for much faster completion when and where tests are parallelizable.

[github-actions]

7304 tests run: 6933 passed, 0 failed, 371 skipped (full report)

---

Flaky tests (2)

Postgres 17

• test_physical_replication_config_mismatch_max_locks_per_transaction: release-arm64

Postgres 14

• test_wal_restore_http[False]: release-arm64

Code coverage* (full report)

• functions: 32.9% (8145 of 24776 functions)
• lines: 48.1% (68235 of 141903 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
22675f4 at 2025-01-14T22:04:22.516Z :recycle:}

[MMeent]

So, by adding -j $(nproc) we gain no profit in performance.

Yeah, I also noticed this, but only after creating the PR.
Instead, I've started upstreaming some test paralellization improvements upstream, using pg_regress' parallel schedules.

See e.g.

• HLL: Run tests on a parallel schedule citusdata/postgresql-hll#164
• pg_hint_plan: Use pg_regress -level parallelization for reduced test runtime ossc-db/pg_hint_plan#213

[a-masterov]

I agree that the extension testing process is quite long and annoying. Still, I have some doubts that just adding -j $(nproc) can significantly reduce the runtime, and that's why: the usual regression test just runs one pg_regress process and then pg_regress loads the schedule and runs the tests sequentially or parallelly. So, by adding -j $(nproc) we gain no profit in performance.
Practically, we see that the test-image test for this PR runs almost the same time as for 10394

	PR-10394	PR-10398	PR-10392
x64-pg17	5 mins	4 mins	4 mins
x64-pg16	9 mins	8 mins	8mins
arm-pg17	13 mins	12 mins	13 mins
arm-pg16	18 mins.	18 mins	17 mins

It would be more profitable to redesign the schedule but in this case, we would maintain our versions of the regression tests or try to pass our patches upstream.

But I have a better idea: we can run the tests in parallel using the different timelines. And I would like to come up with a prototype soon.