server, sql: collect additional session details

[gtr]

Partially addresses #74257.

Previously, the sql server did not provide additional session details
such as total number of transactions executed, transaction fingerprint
IDs, and total active time. This change adds the aforementioned session
details to the serverpb.Session struct.

To track transaction fingerprint IDs, a new circular buffer
TxnFingerprintIDBuffer is introduced with its corresponding cluster
setting TxnFingerprintIDBufferCapacity which controls the capacity.
The total number of transactions executed is accumulated using the txnCounter
from the extraTxnState in connExecutor. The total active time is calculated
by a timeutil.StopWatch which is started and stopped when a transaction is
started, restarted, and finished.

Release note (api change): the serverpb.Session struct now has three
new fields: number of transactions executed, transaction fingerprint
IDs, and total active time.

[cockroach-teamcity]

This change is 

[blathers-crl]

Thank you for updating your pull request.

My owl senses detect your PR is good for review. Please keep an eye out for any test failures in CI.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[maryliag]

Reviewed 8 of 11 files at r1, 2 of 2 files at r2.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @gtr and @maryliag)

---

docs/generated/swagger/spec.json line 1230 at r2 (raw file):

        },
        "num_txns_executed": {
          "description": "num_txns_executed is the number of transactions that were executed so\nfar on this session.",

nit: start directly with Number of transactions...

---

docs/generated/swagger/spec.json line 1520 at r2 (raw file):

    },
    "TransactionFingerprintID": {
      "description": "TransactionFingerprintID is the hashed string constructed using the\nindividual statement fingerprint IDs that comprise the transaction.",

nit: start with Hashed string...

[gtr]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @maryliag)

---

docs/generated/swagger/spec.json line 1230 at r2 (raw file):

Previously, maryliag (Marylia Gutierrez) wrote…

nit: start directly with Number of transactions...

Done.

---

docs/generated/swagger/spec.json line 1520 at r2 (raw file):

Previously, maryliag (Marylia Gutierrez) wrote…

nit: start with Hashed string...

I think this description comes from the comment on roachpb/app_stats.go:
Should I still change it?

Code snippet:

// TransactionFingerprintID is the hashed string constructed using the
// individual statement fingerprint IDs that comprise the transaction.
type TransactionFingerprintID uint64

[Azhng]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @Azhng, @gtr, and @maryliag)

---

pkg/sql/conn_executor.go line 3135 at r3 (raw file):

		return nil
	}
	err := ex.applicationStats.IterateTransactionStats(context.Background(), options, txnStatsVisitor)

Not sure using applicationStats is a good idea here, applicationStats can be shared by multiple connExecutors concurrently, (in the case of explicit transaction, a separate instance of the applicationStats is used exclusively for that transaction).

SQL Stats in general wasn't designed with tracking session-level information in mind, and it's quite difficult with how the fingerprinting is implement. (it doesn't quite make sense to have a "session fingerprint"). If you really want to have a session level fingerprints, you might have to implement it from scratch.

[gtr]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @Azhng and @maryliag)

---

pkg/sql/conn_executor.go line 3135 at r3 (raw file):

Previously, Azhng (Archer Zhang) wrote…

Not sure using applicationStats is a good idea here, applicationStats can be shared by multiple connExecutors concurrently, (in the case of explicit transaction, a separate instance of the applicationStats is used exclusively for that transaction).

SQL Stats in general wasn't designed with tracking session-level information in mind, and it's quite difficult with how the fingerprinting is implement. (it doesn't quite make sense to have a "session fingerprint"). If you really want to have a session level fingerprints, you might have to implement it from scratch.

Good catch, I had a feeling the applicationStats were not unique to a ConnExecutor. I ended up going with a new TxnFingerprintIDBuffer which is a circular buffer tracking the last 100 txnFingerprintIDs like we discussed.

[Azhng]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @Azhng, @gtr, and @maryliag)

---

pkg/sql/txn_fingerprint_id_buffer.go line 34 at r6 (raw file):

// TxnFingerprintIDBuffer is a thread-safe circular buffer tracking transaction
// fingerprint IDs at the session level.
type TxnFingerprintIDBuffer struct {

nit: do we need to export this struct ?

---

pkg/sql/txn_fingerprint_id_buffer.go line 70 at r6 (raw file):

	b.mu.data = make([]roachpb.TransactionFingerprintID, capacity)
	for i := range b.mu.data {
		b.mu.data[i] = roachpb.InvalidTransactionFingerprintID

nit: roachpb.InvalidTransactionFingerprintID is the same as the default initialization value of the type. This initialization loop doesn't seem to be necessary.

---

pkg/sql/txn_fingerprint_id_buffer.go line 105 at r6 (raw file):

func (b *TxnFingerprintIDBuffer) checkForCapacityLocked() {
	capacityClusterSetting := TxnFingerprintIDBufferCapacity.Get(&b.st.SV)
	if b.mu.capacity != capacityClusterSetting {

you can just use cap() builtin function to check capacity instead of track it separately

---

pkg/sql/txn_fingerprint_id_buffer.go line 118 at r6 (raw file):

	ptr := 0
	b.mu.size = 0
	b.initializeBufferLocked(newCapacity)

Doesn't initializeBufferLocked() directly overwrites the b.mu.data ? then this would make the newData allocation above unnecessary.

Later we then assign the newData (newly allocated) to the b.mu.data (also newly allocated), seems like we can just skip the allocation here.

---

pkg/sql/txn_fingerprint_id_buffer.go line 138 at r6 (raw file):

// circular buffer.
func (b *TxnFingerprintIDBuffer) GetAllTxnFingerprintIDs() []roachpb.TransactionFingerprintID {
	b.mu.Lock()

hold the read lock here instead of write lock?

---

pkg/sql/txn_fingerprint_id_buffer.go line 141 at r6 (raw file):

	defer b.mu.Unlock()

	var txnFingerprintIDs []roachpb.TransactionFingerprintID

we can initialize the entire buffer here since we already know the size of the buffer

---

pkg/sql/txn_fingerprint_id_buffer.go line 170 at r6 (raw file):

	size := txnFingerprintID.Size()
	b.mu.acc.Shrink(context.Background(), size)

Since the memory is allocated once and reused, I feel we can just update the memory account each time we need to reallocate the capacity.

---

pkg/sql/txn_fingerprint_id_buffer.go line 172 at r6 (raw file):

	b.mu.acc.Shrink(context.Background(), size)
	b.mu.size--
	b.mu.readPosition++

we should wrap the read position around if the index is out of bound no ?