Flake: ScyllaCluster should allow to build connection pool using shard aware ports #1028
Assignees
Labels
kind/flake
Categorizes issue or PR as related to a flaky test.
priority/important-soon
Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Comments
tnozicka
added
priority/important-soon
zimnx
added a commit
to zimnx/scylla-operator
that referenced
this issue
Sep 20, 2022
Previous implementation gave false positives because both on CI and local environments traffic between test binary and Scylla Pods is NATed. Because we are low on resources (we use just 2 shards) probability of hitting expected shard was high even for random assignment. It wasn't spotted because message about wrong shard assignment is printed out by the driver and no error is reported anywhere. Gocql doesn't expose any way of checking if shardaware ports are used successfully, hence I implemented simple driver within a test, able to send initial packet and reading to which shard connection was established. Fixes scylladb#1028
zimnx
added a commit
to zimnx/scylla-operator
that referenced
this issue
Sep 20, 2022
Both on CI and on local environments traffic between test binary and Scylla Pods is NATed. Previous implementation gave false positives because we are low on resources (we use just 2 shards) so probability of hitting expected shard was high even for random assignment. It wasn't spotted because message about wrong shard assignment is printed out by the driver to stdout and no error is reported anywhere. Gocql doesn't expose any way of checking if shardaware ports are used successfully, hence I implemented simple driver within a test, able to send initial packet and reading to which shard connection was established. It also check in-cluster connectivity which is how clients are connecting from, compared to from outside in previous appraoch. Fixes scylladb#1028
1 task
zimnx
added a commit
to zimnx/scylla-operator
that referenced
this issue
Sep 20, 2022
Both on CI and on local environments traffic between test binary and Scylla Pods is NATed. Previous implementation gave false positives because we are low on resources (we use just 2 shards) so probability of hitting expected shard was high even for random assignment. It wasn't spotted because message about wrong shard assignment is printed out by the driver to stdout and no error is reported anywhere. Gocql doesn't expose any way of checking if shardaware ports are used successfully, hence I implemented simple driver within a test, able to send initial packet and reading to which shard connection was established. It also check in-cluster connectivity which is how clients are connecting from, compared to from outside in previous appraoch. Fixes scylladb#1028
zimnx
added a commit
to zimnx/scylla-operator
that referenced
this issue
Sep 20, 2022
Both on CI and on local environments traffic between test binary and Scylla Pods is NATed. Previous implementation gave false positives because we are low on resources (we use just 2 shards) so probability of hitting expected shard was high even for random assignment. It wasn't spotted because message about wrong shard assignment is printed out by the driver to stdout and no error is reported anywhere. Gocql doesn't expose any way of checking if shardaware ports are used successfully, hence I implemented simple driver within a test, able to send initial packet and reading to which shard connection was established. It also check in-cluster connectivity which is how clients are connecting from, compared to from outside in previous appraoch. Fixes scylladb#1028
zimnx
added a commit
to zimnx/scylla-operator
that referenced
this issue
Sep 20, 2022
Both on CI and on local environments traffic between test binary and Scylla Pods is NATed. Previous implementation gave false positives because we are low on resources (we use just 2 shards) so probability of hitting expected shard was high even for random assignment. It wasn't spotted because message about wrong shard assignment is printed out by the driver to stdout and no error is reported anywhere. Gocql doesn't expose any way of checking if shardaware ports are used successfully, hence I implemented simple driver within a test, able to send initial packet and reading to which shard connection was established. It also check in-cluster connectivity which is how clients are connecting from, compared to from outside in previous appraoch. Fixes scylladb#1028
zimnx
added a commit
to zimnx/scylla-operator
that referenced
this issue
Sep 20, 2022
Both on CI and on local environments traffic between test binary and Scylla Pods is NATed. Previous implementation gave false positives because we are low on resources (we use just 2 shards) so probability of hitting expected shard was high even for random assignment. It wasn't spotted because message about wrong shard assignment is printed out by the driver to stdout and no error is reported anywhere. Gocql doesn't expose any way of checking if shardaware ports are used successfully, hence I implemented simple driver within a test, able to send initial packet and reading to which shard connection was established. It also check in-cluster connectivity which is how clients are connecting from, compared to from outside in previous appraoch. Fixes scylladb#1028
zimnx
added a commit
to zimnx/scylla-operator
that referenced
this issue
Sep 26, 2022
Both on CI and on local environments traffic between test binary and Scylla Pods is NATed. Previous implementation gave false positives because we are low on resources (we use just 2 shards) so probability of hitting expected shard was high even for random assignment. It wasn't spotted because message about wrong shard assignment is printed out by the driver to stdout and no error is reported anywhere. Gocql doesn't expose any way of checking if shardaware ports are used successfully, hence I implemented simple driver within a test, able to send initial packet and reading to which shard connection was established. It also check in-cluster connectivity which is how clients are connecting from, compared to from outside in previous appraoch. Fixes scylladb#1028
|
4 shards on 2 core machine might not be good idea, maybe it's better to skip it on not-big-enough nodes |
|
skips usually mean bugs slipping :( |
|
Prow CI is big enough, it fails only on GH Actions where we have limited resources. |
|
ok, then I'd defer this one to when we finish the migration |
Merged
1 task
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
https://github.com/scylladb/scylla-operator/actions/runs/3056629503/jobs/4931011388#step:12:664
The text was updated successfully, but these errors were encountered: