roachtest: reduce rebalance-by-load noise #104885
Conversation
The rebalance-by-load tests assert that the normalized [0,1] CPU utilization of each node is within some threshold of the mean. The threshold was previously 10%, however it is not unexpected that despite replica load being within this threshold, that total node load is not. The current balancing implementation only concerns itself with replica load. Bump the tolerance from 10% to 15% to reduce noise. Additionally, the test did not wait for 3x replication prior to beginning the workload. This is bound to introduce flakes eventually. Wait for 3x replication before beginning. Resolves: cockroachdb#104854 Resolves: cockroachdb#104386 Release note: None
kvoli
requested review from
herkolategan and
renatolabs
and removed request for
a team
|
This change is |
kvoli
added
backport-23.1.x
There was a problem hiding this comment.
Reviewed 1 of 1 files at r1.
Reviewable status:complete! 1 of 0 LGTMs obtained (waiting on @herkolategan, @renatolabs, and @sumeerbhola)
|
bors r+ single on |
|
TYFTR! |
|
Build failed: |
|
Flakes unrelated. Let's retry. bors r+ single on |
|
Build succeeded: |
|
Encountered an error creating backports. Some common things that can go wrong:
You might need to create your backport manually using the backport tool. error creating merge commit from 1fefe10 to blathers/backport-release-22.2-104885: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict [] you may need to manually resolve merge conflicts with the backport tool. Backport to branch 22.2.x failed. See errors above. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
The rebalance-by-load tests assert that the normalized [0,1] CPU utilization of each node is within some threshold of the mean. The threshold was previously 10%, however it is not unexpected that despite replica load being within this threshold, that total node load is not.
The current balancing implementation only concerns itself with replica load.
Bump the tolerance from 10% to 15% to reduce noise.
Additionally, the test did not wait for 3x replication prior to beginning the workload. This is bound to introduce flakes eventually. Wait for 3x replication before beginning.
Resolves: #104854
Resolves: #104386
Release note: None