Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sql, sqlstats: create temporary stats container for all txns #83616

Merged
merged 2 commits into from
Jul 6, 2022

Conversation

xinhaoz
Copy link
Member

@xinhaoz xinhaoz commented Jun 29, 2022

Commit 1

sql: add txn_fingerprint_id to node_statement_statistics

This commit adds the txn_fingerprint_id column to
crdb_internal.node_statement_statistics.

Release note (sql): txn_fingerprin_id has been added to
crdb_internal.node_statement_statistics. The type of the
column is NULL or STRING.


Commit 2

sql, sqlstats: create temporary stats container for all txns

Fixes: #81470

Previously, the stats collector followed different procedures for stats
collection depending on whether or not the txn was explicit.

For explicit transactions, all the stmts in the txn must be recorded with
the same transactionFingerprintID, which is only known after all stmts
in the txn have been executed. In order to record the correct
txnFingerprintID, a temporary stats container was created for stmts in the
current transaction. The transactionFingerprintID was then populated for
all stmts in the temp container, and the temp container was merged with
the parent.

For implict transactions, the assumption was there would only be a single
stmt in the txn, and so no temporary container was created, with stmts
being written directly to the application stats.

This assumption was incorrect, as it is possible for implicit txns to have
multiple stmts, such as stmts sent in a batch.

This commit ensures that stats are properly collected for implicit txns
with multiple stmts. The stats collector now follows the same procedure
for both explicit and implicit txns, creating a temporary container for
local txn stmts and merging on txn finish.

Release note (bug fix): Statement and transaction stats are now properly
recorded for implicit txns with multiple stmts.

@cockroach-teamcity
Copy link
Member

This change is Reviewable

@xinhaoz xinhaoz force-pushed the explicit-txn-stats branch 4 times, most recently from 67bc87e to d254c84 Compare June 30, 2022 18:17
xinhaoz added 2 commits June 30, 2022 14:20
This commit adds the `txn_fingerprint_id` column to
`crdb_internal.node_statement_statistics`.

Release note (sql): `txn_fingerprin_id` has been added to
`crdb_internal.node_statement_statistics`. The type of the
column is NULL or STRING.
Fixes: cockroachdb#81470

Previously, the stats collector followed different procedures for stats
collection depending on whether or not the txn was explicit.

For explicit transactions, all the stmts in the txn must be recorded with
the same `transactionFingerprintID`, which is  only known after all stmts
in the txn have been executed. In order to record the correct
txnFingerprintID, a temporary stats container was created for stmts in the
current transaction. The `transactionFingerprintID` was then populated for
all stmts in the temp container, and the temp container was merged with
the parent.

For implict transactions, the assumption was there would only be a single
stmt in the txn, and so no temporary container was created, with stmts
being written directly to the application stats.

This assumption was incorrect, as it is possible for implicit txns to have
multiple stmts, such as stmts sent in a batch.

This commit ensures that stats are properly collected for implicit txns
with multiple stmts. The stats collector now follows the same procedure
for both explicit and implicit txns, creating a temporary container for
local txn stmts and merging on txn finish.

Release note (bug fix): Statement and transaction stats are now properly
recorded for implicit txns with multiple stmts.
@xinhaoz xinhaoz force-pushed the explicit-txn-stats branch from d254c84 to 607026a Compare June 30, 2022 18:22
@xinhaoz xinhaoz marked this pull request as ready for review June 30, 2022 18:23
@xinhaoz xinhaoz requested a review from a team June 30, 2022 18:23
@xinhaoz xinhaoz requested a review from a team as a code owner June 30, 2022 18:23
@xinhaoz xinhaoz requested review from a team and rafiss June 30, 2022 18:23
Copy link
Collaborator

@rafiss rafiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice! the connExecutor changes lgtm. the sqlstats package changes also lgtm, but i haven't read that code before

and just to confirm, the second commit can be backported to v22.1, right?

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @rafiss and @xinhaoz)


-- commits line 7 at r1:
nit: typo in the name

Copy link
Contributor

@maryliag maryliag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

And the goal is indeed backport this to 22.1

Reviewed 4 of 4 files at r1, 8 of 8 files at r2, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @xinhaoz)

@xinhaoz
Copy link
Member Author

xinhaoz commented Jun 30, 2022

One test case in the 2nd commit needs the first commit. To clarify, it's okay to backport both right? Or are we aiming for just the 2nd one with the test case removed?

Copy link
Contributor

@matthewtodd matthewtodd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thank you for making this code simpler and easier to understand!

Re: what to backport, are these 2 commits here together because they're related? FWICT, it doesn't seem that they are. Is it to avoid a second (long!) CI cycle? Given the user-visible new column in the first commit, it seems better to me to just backport the second one (modifying the test), but maybe @maryliag can weigh in on whether that hygiene's worth the extra effort.

@xinhaoz
Copy link
Member Author

xinhaoz commented Jul 6, 2022

I'll bors and check with Marylia to confirm backports. Thank you for the reviews!
bors r+

@craig
Copy link
Contributor

craig bot commented Jul 6, 2022

Build succeeded:

@craig craig bot merged commit ce1b42b into cockroachdb:master Jul 6, 2022
@blathers-crl
Copy link

blathers-crl bot commented Jul 6, 2022

Encountered an error creating backports. Some common things that can go wrong:

  1. The backport branch might have already existed.
  2. There was a merge conflict.
  3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.


error creating merge commit from 607026a to blathers/backport-release-22.1-83616: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 22.1.x failed. See errors above.


🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants