cli: create new tracing span in mt-start-sql

[yuzefovich]

This commit fixes a possible misuse of the tracing infrastructure with mt start-sql command. In particular, previously it was possible for a tracing span created (and finished) in runStartInternal to be used by a couple of things in makeTenantSQLServerArgs (main one being rpc.Context) that hold on to the context after the corresponding span has been finished. This is disallowed (since we reuse spans after Finish has been called on them) and was recently discovered. This problem is now fixed by deriving a child span only in mt start-sql code path that is never finished (since those users can be active throughout the whole lifetime of the tenant server).

The bug is pretty minor (I think only serverless is affected), so there is no release note.

Fixes: #113170.

Release note: None

[blathers-crl]

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[cockroach-teamcity]

This change is 

[stevendanna]

Thanks for tracking this down.

An alternative would be to make NewSeparateProcessTenantServer closer to NewServer -- which doesn't take a context and builds its own from context.Background().

But I think what you have here makes sense. The context we are deriving from is cancelled eventually, but only as a defer in runStartInternal which should only happen after we've already returned from waitForShutdown() -- which is probably closer to what we want.

Overall, I think the context management in the server startup is rather unclear. In addition to the cases you called out in the original issue, if you read through the various startup paths, we capture contexts in various closures in multiple places, all of which seem a bit suspect to me. But, that is a task for a different day.

[yuzefovich]

I agree with your reasoning, so I added a TODO to explore the refactoring at some point. For now this seems good enough.

TFTR!

bors r+

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error setting reviewers, but backport branch blathers/backport-release-23.2-113553 is ready: POST https://api.github.com/repos/cockroachdb/cockroach/pulls/113595/requested_reviewers: 422 Reviews may only be requested from collaborators. One or more of the teams you specified is not a collaborator of the cockroachdb/cockroach repository. []

Backport to branch 23.2.x failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}