chore(networking): Rework internal auto concurrency tests

[bruceg]

This rebuilds the internal tests to run all at once, and to use external data files for the parameters to avoid needing a full recompile to tweak the settings.

[jszwedko]

This looks good! I like the test configuration being separate though it is a bit unfortunate that compilation time is pushing us into that pattern.

I did get this test failure locally though:

thread 'sinks::util::auto_concurrency::tests::all_tests' panicked at 'assertion failed: `(expr > high)`
expr: value = `0.126087944`,
high: `0.125`
File: "tests/data/auto-concurrency/medium-send.toml" Value: controller observed_rtt max
TestResults {
    file_path: "tests/data/auto-concurrency/medium-send.toml",
    stats: Statistics {
        in_flight: LevelTimeHistogram {
            level: 0,
            histogram: TimeHistogram {
                histogram: Histogram {
                    totals: [
                        0.002160466,
                        0.33612984500000004,
                        0.640026141,
                        0.16129062699999996,
                        9.782315846000012,
                        0.9297670430000003,
                        0.22614229900000005,
                        0.13017132,
                        0.324182766,
                        0.018357403,
                    ],
                },
                last_time: Some(
                    Instant {
                        tv_sec: 314176,
                        tv_nsec: 503624389,
                    },
                ),
            },
        },
    },
    cstats: ControllerStatistics {
        in_flight: TimeHistogram {
            histogram: Histogram {
                totals: [
                    0.002038978,
                    0.33604503299999994,
                    0.640103479,
                    0.15982594800000002,
                    9.748312863999983,
                    0.9651788440000001,
                    0.22615031700000002,
                    0.130118067,
                    0.324396845,
                    0.018414793,
                ],
            },
            last_time: Some(
                Instant {
                    tv_sec: 314176,
                    tv_nsec: 503641418,
                },
            ),
        },
        concurrency_limit: TimeHistogram {
            histogram: Histogram {
                totals: [
                    0.0,
                    0.0,
                    0.620126786,
                    0.102965506,
                    0.101770415,
                    0.102244049,
                    0.199927082,
                    0.101782596,
                    0.304869607,
                    4.321242149000001,
                    6.2903737930000005,
                ],
            },
            last_time: Some(
                Instant {
                    tv_sec: 314176,
                    tv_nsec: 405079296,
                },
            ),
        },
        observed_rtt: TimeWeightedSum {
            sum: WeightedSum {
                total: 1.2682793218495096,
                weights: 12.447222536000009,
                min: Some(
                    0.097010317,
                ),
                max: Some(
                    0.126087944,
                ),
            },
            last_observation: Some(
                Instant {
                    tv_sec: 314176,
                    tv_nsec: 503641418,
                },
            ),
        },
        averaged_rtt: TimeWeightedSum {
            sum: WeightedSum {
                total: 1.2366884388567434,
                weights: 12.145301983000003,
                min: Some(
                    0.10048516739999999,
                ),
                max: Some(
                    0.11242788340000001,
                ),
            },
            last_observation: Some(
                Instant {
                    tv_sec: 314176,
                    tv_nsec: 405079296,
                },
            ),
        },
    },
}', src/sinks/util/auto_concurrency/tests.rs:344:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    sinks::util::auto_concurrency::tests::all_tests

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 747 filtered out

error: test failed, to rerun pass '--lib'

Maybe flakey?

[bruceg]

Apparently your system can have worse than 25ms of extra delay on timings. I need to investigate using a faked clock to cause exact timing and near zero actual run time.

[jszwedko]

Apparently your system can have worse than 25ms of extra delay on timings. I need to investigate using a faked clock to cause exact timing and near zero actual run time.

I'm running them in an AWS VM so I guess I'm not terribly surprised 😄 A fake clock seems like it makes sense.

[bruceg]

I had a light bulb moment, and this will now close #3823.

[fanatid]

I do not see time::resume anywhere, advance work in this case?

[bruceg]

Yes, when the tokio timer is paused, the reference time stops. When advance is called, it steps forward by the duration specified and runs any timed events that were queued. resume ties that reference clock back to the system clock, resuming normal time flow. In this case, I want the timer to stay paused for the stability of the results, so there is no need to resume the clock. I don't think resume would be needed for most tests, but it would depend on exactly what is happening there.

[bruceg]

I think I've polished this doodoo far beyond what I should have, though it probably makes it a little better foundation for #2531.

[jszwedko]

Looks good! I reran the tests a couple times.

[bruceg]

Yeah, with the tests being able to complete so fast, I ran them about 100 times to make sure the tighter parameters were stable.