Amend load generator to allow generating more aggressive work loads

[praveen-influx]

Problem statement

It would be useful to extend the existing behaviour of load generator to generate traffic based on some command line parameters. Currently it uses a spec file (1M) to generate traffic, however this still does not stress the influxdb process. This means it's very hard to find the hot paths locally.

Solution

Allow configuring load generator through other command line params like,

    --concurrency-level 10 (this can be passed to the underlying tokio runtime for example)
    --db 5 (number of dbs it should generate the data for)
    --table 10 (number of tables it should generate the data for)
    --desired-throughput 3M (note, this is in bytes - so 3MB/s)
    --gradual-increase (flag so that it can generate traffic gradually)

The idea here is to work out the number of columns (and types) needed per table based on target throughput (in the above example 3M) and the amount of db/tables that traffic will be divided into. So as an example if we need to generate ~3MB/s between 5 dbs and 10 tables, the generator can figure out what column types to pick (to start with it can all just be u64s and f64s for example). And have a couple of tags for each that are strings. This should be sufficient to work out the division of number of columns between 10 tables (based on the knowledge of how to construct a line adhering to line protocol).

And if the --gradual-increase flag is set, the generator can gradually increase the throughput so that it's easier to spot any issues that occurs before reaching the --desired-throughput.

Alternatives considered

• Use any off the shelf tools, most off the shelf tools still require generating the traffic that is compliant for the line protocol and also we have a version of load generator already. It is probably easier to extend it and maintain than picking off the shelf solutions.

Additional context

N/A