Have a way of running A/B tests against dynamic devnets

[ndr-ds]

Once we have #2933, then we could technically run A/B tests:
Spin up two small devnets, with the same exact configs, but one with binaries built from some base rev, and another one built with some changes we're trying to test. Then, if we run the same set of tests against these two devnets, serially, and in the same order, we should technically be able to catch metric changes that the changes are introducing.
Ideally we wouldn't run tests that can have non-deterministic behavior with this, as it would likely introduce too much noise in the metrics.