Report Lineage metrics for SpannerIO #32561
Conversation
| @@ -246,9 +249,21 @@ private static SpannerAccessor createAndConnect(SpannerConfig spannerConfig) { | |||
| BatchClient batchClient = | |||
| spanner.getBatchClient(DatabaseId.of(options.getProjectId(), instanceId, databaseId)); | |||
| DatabaseAdminClient databaseAdminClient = spanner.getDatabaseAdminClient(); | |||
| String instanceConfigId = "unknown"; | |||
| try { | |||
| instanceConfigId = | |||
There was a problem hiding this comment.
Per doc spanner FQN has format spanner:{projectId}.{instanceConfigId}.{instanceId}.{databaseId}.{assetId}
However, currently instanceConfigId is not mentioned throughout SpannerIO. Had to make a real API call to get instance config Id.
Because spannerAccessor is stored in a static concurrent Map, this call will be done per (Dataflow worker) * (spanner instance destination)
cc: @manitgupta would like to learn if there a better way to do this.
There was a problem hiding this comment.
Do you know what is the avg and upper bound on "spanner instance destination" ?
There was a problem hiding this comment.
it is upper bounded by the number of SpannerIO (source+sink) used in the pipeline. (One SpannerIO transform only support single instance and database)
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
|
assign set of reviewers |
|
Assigning reviewers. If you would like to opt out of this review, comment R: @robertwb for label java. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.