Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scheduledlogging: fix TestCaptureIndexUsageStats for secondary tenants #109783

Merged

Conversation

zachlite
Copy link
Contributor

This commit fixes flaky behavior when TestCaptureIndexUsageStats is run for secondary tenants. The previous synchronization mechanism between logging and the test was not consistent between system and secondary tenants.

Instead, there's no attempt to synchronize, and we rely on testutils.SucceedsSoon instead.

Resolves #109465
Epic: none
Release note: None

@zachlite zachlite requested a review from a team August 30, 2023 22:20
@cockroach-teamcity
Copy link
Member

This change is Reviewable

This commit fixes flaky behavior when TestCaptureIndexUsageStats
is run for secondary tenants. The previous synchronization mechanism
between logging and the test was not consistent between system and
secondary tenants.

Instead, there's no attempt to synchronize, and we rely on
testutils.SucceedsSoon instead.

Resolves cockroachdb#109465
Epic: none
Release note: None
@zachlite zachlite force-pushed the 230830.captured_index_usage_stats branch from c13360c to 5add509 Compare August 31, 2023 13:25
Copy link
Contributor

@j82w j82w left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained

@zachlite zachlite added backport-23.1.x Flags PRs that need to be backported to 23.1 and removed backport-23.1.x Flags PRs that need to be backported to 23.1 labels Sep 1, 2023
@zachlite
Copy link
Contributor Author

zachlite commented Sep 1, 2023

bors r+

@zachlite zachlite added the backport-23.1.x Flags PRs that need to be backported to 23.1 label Sep 1, 2023
@craig
Copy link
Contributor

craig bot commented Sep 1, 2023

Build succeeded:

@craig craig bot merged commit fe4de2b into cockroachdb:master Sep 1, 2023
@blathers-crl
Copy link

blathers-crl bot commented Sep 1, 2023

Encountered an error creating backports. Some common things that can go wrong:

  1. The backport branch might have already existed.
  2. There was a merge conflict.
  3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.


error creating merge commit from 5add509 to blathers/backport-release-23.1-109783: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 23.1.x failed. See errors above.


🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

xinhaoz added a commit to xinhaoz/cockroach that referenced this pull request Sep 7, 2023
This commit reverts cockroachdb#109783 which added a `SucceedsSoon` in place
of testing each anticipated sheduling of  index usage stats
recording. Since this test checks the number of logs emitted
after each schedule, adding `SucceedsSoon` meant that entering a
retry loop could actually mean we fall into the next scheduled
index usage stats emission, thus surpassing the expected number of
logs for the schedule number being checked. This addition was meant
to fix the test for secondary tenants by waiting for all schedules
of tenants to run. This commit instead disables index usage stats for
the system tenant, allowing us to once again synchronize a single
tenant's schedule in order to verify the logs.

Fixes: cockroachdb#109924
xinhaoz added a commit to xinhaoz/cockroach that referenced this pull request Sep 8, 2023
This commit reverts cockroachdb#109783 which added a `SucceedsSoon` in place
of testing each anticipated sheduling of  index usage stats
recording. Since this test checks the number of logs emitted
after each schedule, adding `SucceedsSoon` meant that entering a
retry loop could actually mean we fall into the next scheduled
index usage stats emission, thus surpassing the expected number of
logs for the schedule number being checked. This addition was meant
to fix the test for secondary tenants by waiting for all schedules
of tenants to run. This commit instead disables index usage stats for
the system tenant, allowing us to once again synchronize a single
tenant's schedule in order to verify the logs.

Fixes: cockroachdb#109924

Release note: None
craig bot pushed a commit that referenced this pull request Sep 8, 2023
110224: scheduledlogging: fix `TestCaptureIndexUsageStats` r=xinhaoz a=xinhaoz

This commit reverts #109783 which added a `SucceedsSoon` in place of testing each anticipated sheduling of  index usage stats recording. Since this test checks the number of logs emitted after each schedule, adding `SucceedsSoon` meant that entering a retry loop could actually mean we fall into the next scheduled index usage stats emission, thus surpassing the expected number of logs for the schedule number being checked. This addition was meant to fix the test for secondary tenants by waiting for all schedules of tenants to run. This commit instead disables index usage stats for the system tenant, allowing us to once again synchronize a single tenant's schedule in order to verify the logs.

Fixes: #109924

Release note: None

Co-authored-by: Xin Hao Zhang <[email protected]>
xinhaoz added a commit to xinhaoz/cockroach that referenced this pull request Sep 8, 2023
This commit reverts cockroachdb#109783 which added a `SucceedsSoon` in place
of testing each anticipated sheduling of  index usage stats
recording. Since this test checks the number of logs emitted
after each schedule, adding `SucceedsSoon` meant that entering a
retry loop could actually mean we fall into the next scheduled
index usage stats emission, thus surpassing the expected number of
logs for the schedule number being checked. This addition was meant
to fix the test for secondary tenants by waiting for all schedules
of tenants to run. This commit instead disables index usage stats for
the system tenant, allowing us to once again synchronize a single
tenant's schedule in order to verify the logs.

Fixes: cockroachdb#109924

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-23.1.x Flags PRs that need to be backported to 23.1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

sql/scheduledlogging: TestCaptureIndexUsageStats does not work with secondary tenants
3 participants