feat!: add support for CommitStats #544

olavloite · 2020-10-24T10:59:11Z

Adds support for returning CommitStats from read/write transactions.

Replaces #522

Adds support for returning CommitStats from read/write transactions.

codecov · 2020-10-24T12:11:44Z

Codecov Report

Merging #544 (65f9b89) into master (f9ac29c) will increase coverage by 0.22%.
The diff coverage is 96.03%.

@@             Coverage Diff              @@
##             master     #544      +/-   ##
============================================
+ Coverage     85.04%   85.26%   +0.22%     
- Complexity     2583     2616      +33     
============================================
  Files           143      144       +1     
  Lines         14145    14208      +63     
  Branches       1369     1372       +3     
============================================
+ Hits          12030    12115      +85     
+ Misses         1542     1522      -20     
+ Partials        573      571       -2

Impacted Files	Coverage Δ	Complexity Δ
...a/com/google/cloud/spanner/TransactionManager.java	`100.00% <ø> (ø)`	`0.00 <0.00> (ø)`
...spanner/admin/database/v1/DatabaseAdminClient.java	`83.22% <ø> (ø)`	`100.00 <0.00> (ø)`
...spanner/admin/instance/v1/InstanceAdminClient.java	`79.14% <ø> (ø)`	`56.00 <0.00> (ø)`
...ava/com/google/cloud/spanner/v1/SpannerClient.java	`82.05% <ø> (ø)`	`63.00 <0.00> (ø)`
...m/google/cloud/spanner/TransactionManagerImpl.java	`87.50% <66.66%> (-1.39%)`	`21.00 <1.00> (+1.00)`	⬇️
...cloud/spanner/connection/SingleUseTransaction.java	`91.35% <66.66%> (+2.40%)`	`32.00 <2.00> (ø)`
...ud/spanner/SessionPoolAsyncTransactionManager.java	`87.21% <85.71%> (+1.50%)`	`14.00 <1.00> (+3.00)`
...ain/java/com/google/cloud/spanner/SessionPool.java	`89.23% <92.85%> (-0.02%)`	`73.00 <0.00> (ø)`
...om/google/cloud/spanner/TransactionRunnerImpl.java	`86.68% <93.75%> (+0.96%)`	`10.00 <1.00> (+1.00)`
...java/com/google/cloud/spanner/AsyncRunnerImpl.java	`93.93% <100.00%> (+1.63%)`	`9.00 <7.00> (+4.00)`
... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f9ac29c...7d45ace. Read the comment docs.

skuruppu

I'm not quite sure if we need the breaking change here. If it's just because of the interface update, I wouldn't mark it as such. But if there is an actual breaking change where users will have to update their code in order to use the next release, then let me know.

...cloud-spanner/src/main/java/com/google/cloud/spanner/SessionPoolAsyncTransactionManager.java

google-cloud-spanner/src/main/java/com/google/cloud/spanner/TransactionManagerImpl.java

Co-authored-by: skuruppu <[email protected]>

olavloite · 2020-10-31T07:34:15Z

I'm not quite sure if we need the breaking change here. If it's just because of the interface update, I wouldn't mark it as such. But if there is an actual breaking change where users will have to update their code in order to use the next release, then let me know.

There are no changes in this PR that will break existing code that just uses these interfaces. Binary compatibility with the previous version will be broken, and code that actually implements the DatabaseClient interface would need to be updated.

generated-files-bot · 2020-11-06T12:07:00Z

Warning: This pull request is touching the following templated files:

proto-google-cloud-spanner-v1/src/main/java/com/google/spanner/v1/CommitRequest.java
proto-google-cloud-spanner-v1/src/main/java/com/google/spanner/v1/CommitRequestOrBuilder.java
proto-google-cloud-spanner-v1/src/main/java/com/google/spanner/v1/CommitResponse.java
proto-google-cloud-spanner-v1/src/main/java/com/google/spanner/v1/CommitResponseOrBuilder.java
proto-google-cloud-spanner-v1/src/main/java/com/google/spanner/v1/SpannerProto.java
proto-google-cloud-spanner-v1/src/main/proto/google/spanner/v1/spanner.proto

generated-files-bot · 2020-12-05T09:29:33Z

Warning: This pull request is touching the following templated files:

proto-google-cloud-spanner-v1/src/main/java/com/google/spanner/v1/CommitRequest.java
proto-google-cloud-spanner-v1/src/main/java/com/google/spanner/v1/CommitRequestOrBuilder.java
proto-google-cloud-spanner-v1/src/main/java/com/google/spanner/v1/CommitResponse.java
proto-google-cloud-spanner-v1/src/main/java/com/google/spanner/v1/CommitResponseOrBuilder.java
proto-google-cloud-spanner-v1/src/main/java/com/google/spanner/v1/SpannerProto.java
proto-google-cloud-spanner-v1/src/main/proto/google/spanner/v1/spanner.proto

generated-files-bot · 2020-12-10T19:12:42Z

Warning: This pull request is touching the following templated files:

proto-google-cloud-spanner-v1/src/main/java/com/google/spanner/v1/CommitRequest.java
proto-google-cloud-spanner-v1/src/main/java/com/google/spanner/v1/CommitRequestOrBuilder.java
proto-google-cloud-spanner-v1/src/main/java/com/google/spanner/v1/CommitResponse.java
proto-google-cloud-spanner-v1/src/main/java/com/google/spanner/v1/CommitResponseOrBuilder.java
proto-google-cloud-spanner-v1/src/main/java/com/google/spanner/v1/SpannerProto.java
proto-google-cloud-spanner-v1/src/main/proto/google/spanner/v1/spanner.proto

elharo · 2021-01-21T13:54:37Z

google-cloud-spanner/clirr-ignored-differences.xml

@@ -406,4 +406,62 @@
    <className>com/google/cloud/spanner/AbstractLazyInitializer</className>
    <method>java.lang.Object initialize()</method>
  </difference>
+
+  <!-- Support for CommitStats added -->


another one where a major version bump is required

skuruppu · 2021-01-23T00:57:18Z

@olavloite, I just merged in #817 so you can rebase this PR. In light of the discussions about binary incompatibility, please take a look and see if the implementation needs to be changed to avoid a breaking change or if we'll just have to do a semver updgrade.

skuruppu · 2021-01-27T22:58:48Z

@olavloite, I just merged in #817 so you can rebase this PR. In light of the discussions about binary incompatibility, please take a look and see if the implementation needs to be changed to avoid a breaking change or if we'll just have to do a semver updgrade.

@skuruppu One possible way to change this into a non-breaking change would be to return the CommitStats through the options object instead of through the TransactionRunner. In the current implementation requesting and getting the CommitStats looks like this:
    TransactionRunner runner = client.readWriteTransaction(Options.commitStats());
    runner.run(
        new TransactionCallable<Void>() {
          @Override
          public Void run(TransactionContext transaction) throws Exception {
            transaction.buffer(Mutation.delete("FOO", Key.of("foo")));
            return null;
          }
        });
    System.out.println(runner.getCommitResponse().getCommitStats());
That could be changed to the following:
    CommitStatsOption option = Options.commitStats();
    TransactionRunner runner = client.readWriteTransaction(option);
    runner.run(
        new TransactionCallable<Void>() {
          @Override
          public Void run(TransactionContext transaction) throws Exception {
            transaction.buffer(Mutation.delete("FOO", Key.of("foo")));
            return null;
          }
        });
    System.out.println(option.getCommitStats());
In the latter case we don't need to add the new method TransactionRunner#getCommitResponse which is the source of the breaking change in this case. Instead, we make the CommitStatsOption class public (it is currently package-private) and add a getCommitStats() method to it.

WDYT?

I think that's a great idea. It is an option after all and as long as it's accessible when required, then it works. @thiagotnunes FYI.

thiagotnunes · 2021-01-27T23:25:48Z

@olavloite hey Knut, after talking with @skuruppu we have decided not to change the design right now, but to do a major release instead.

elharo

It's always acceptable to bump the major version. I think you'll want to add a ! after feat in the PR description (i.e. feat! ) so the release automation notices this. It might also be useful to update the snapshot version in this PR.

After this release goes out, you can remove the ignored differences file and point it at the newly released major version.

google-cloud-spanner/src/main/java/com/google/cloud/spanner/Options.java

google-cloud-spanner/src/test/java/com/google/cloud/spanner/OptionsTest.java

google-cloud-spanner/src/test/java/com/google/cloud/spanner/it/ITAsyncAPITest.java

thiagotnunes · 2021-01-29T04:51:03Z

@elharo Thanks, updated the PR description.

elharo · 2021-02-03T14:19:01Z

google-cloud-spanner/src/test/java/com/google/cloud/spanner/it/ITTransactionTest.java

+
+  @Test
+  public void transactionRunnerReturnsCommitStats() {
+    assumeFalse("Emulator does not return commit statistics", isUsingEmulator());


The assume here is surprising. Why would this be sometimes be true and sometimes not true in this one test method?

isUsingEmulator() checks whether the environment variable SPANNER_EMULATOR_HOST has been set to a non-empty value. If so, the integration tests are running against the emulator. The emulator does not support all features of Cloud Spanner, which means we need to skip some specific tests. The CI environments runs the tests both against the emulator and Cloud Spanner.

google-cloud-spanner/src/test/java/com/google/cloud/spanner/it/ITTransactionTest.java

skuruppu · 2021-02-04T02:08:17Z

@olavloite this can now be merged when you're ready.

skuruppu · 2021-02-10T05:02:07Z

@elharo friendly ping. This feature is launching soon so we need to start merging in the PRs to make sure that the samples are ready to ingested into the Cloud Docs.

elharo · 2021-02-16T12:47:28Z

google-cloud-spanner/src/main/java/com/google/cloud/spanner/AsyncRunner.java

@@ -56,4 +56,10 @@
   * {@link ExecutionException} if the transaction did not commit.
   */
  ApiFuture<Timestamp> getCommitTimestamp();
+
+  /**
+   * Returns the {@link CommitResponse} of this transaction. {@link ApiFuture#get()} will throw an


will throw --> throws
per Google style

elharo · 2021-02-16T12:49:45Z

google-cloud-spanner/src/main/java/com/google/cloud/spanner/AsyncRunnerImpl.java

    try {
-      commitTimestamp.set(delegate.getCommitTimestamp());
+      commitResponse.set(delegate.getCommitResponse());
    } catch (Throwable t) {


letting it slide because it isn't changed in this PR, but catching Throwable is only rarely what you want. This is probably worth filing a bug on.

Added issue: #875

elharo · 2021-02-16T12:51:42Z

google-cloud-spanner/src/main/java/com/google/cloud/spanner/AsyncTransactionManagerImpl.java

@@ -132,29 +133,37 @@ public void onError(Throwable t) {
          SpannerExceptionFactory.newSpannerException(
              ErrorCode.ABORTED, "Transaction already aborted"));
    }
-    ApiFuture<Timestamp> res = txn.commitAsync();
+    ApiFuture<CommitResponse> res = txn.commitAsync();


no abbreviated variable names per google style.
Concretely I did not know what this was when I read it below and had to scroll up to find out.

Changed to commitResponseFuture

elharo · 2021-02-16T12:53:34Z

google-cloud-spanner/src/main/java/com/google/cloud/spanner/CommitStats.java

+   * row from a parent table that has the ON DELETE CASCADE annotation is also counted as one
+   * mutation regardless of the number of interleaved child rows present. The exception to this is
+   * if there are secondary indexes defined on rows being deleted, then the changes to the secondary
+   * indexes will be counted individually. For example, if a table has 2 secondary indexes, deleting


will be --> are

elharo · 2021-02-16T12:53:50Z

google-cloud-spanner/src/main/java/com/google/cloud/spanner/CommitStats.java

+   * mutation regardless of the number of interleaved child rows present. The exception to this is
+   * if there are secondary indexes defined on rows being deleted, then the changes to the secondary
+   * indexes will be counted individually. For example, if a table has 2 secondary indexes, deleting
+   * a range of rows in the table will count as 1 mutation for the table, plus 2 mutations for each


will count --> counts

elharo · 2021-02-16T12:58:48Z

google-cloud-spanner/src/test/java/com/google/cloud/spanner/OptionsTest.java

+  public void testTransactionOptions_withCommitStatsAndOtherOptionAreNotEqual() {
+    Options option1 = Options.fromTransactionOptions(Options.commitStats());
+    Options option2 = Options.fromQueryOptions(Options.prefetchChunks(10));
+    assertFalse(option1.equals(option2));


assertNotEquals

elharo · 2021-02-16T12:58:58Z

google-cloud-spanner/src/test/java/com/google/cloud/spanner/OptionsTest.java

+  public void testTransactionOptions_withCommitStatsAreEqual() {
+    Options option1 = Options.fromTransactionOptions(Options.commitStats());
+    Options option2 = Options.fromTransactionOptions(Options.commitStats());
+    assertTrue(option1.equals(option2));


assertEquals

elharo · 2021-02-16T12:59:05Z

google-cloud-spanner/src/test/java/com/google/cloud/spanner/OptionsTest.java

+  public void testTransactionOptions_noOptionsAreEqual() {
+    Options option1 = Options.fromTransactionOptions();
+    Options option2 = Options.fromTransactionOptions();
+    assertTrue(option1.equals(option2));


assertEquals

elharo · 2021-02-16T13:01:44Z

google-cloud-spanner/src/test/java/com/google/cloud/spanner/it/ITTransactionManagerTest.java

+    assumeFalse("Emulator does not return commit statistics", isUsingEmulator());
+    try (TransactionManager manager = client.transactionManager(Options.commitStats())) {
+      TransactionContext transaction = manager.begin();
+      while (true) {


This one scares me since it looks like an infinite loop Could you add a counter with a maximum number of retries before failure?

This is the recommended way to use TransactionManager according to the documentation. It is also the way it is already used in multiple other test cases. I would rather either:

Keep this and all other instances as they are.

Change this and other instances + the documentation in a separate PR.

elharo · 2021-02-16T13:02:51Z

google-cloud-spanner/src/test/java/com/google/cloud/spanner/it/ITTransactionManagerTest.java

+          assertEquals(2L, manager.getCommitResponse().getCommitStats().getMutationCount());
+          break;
+        } catch (AbortedException e) {
+          Thread.sleep(e.getRetryDelayInMillis() / 1000);


BUG! Thread.sleep takes milliseconds. No need to divide by 1000.

Good point. Changed and filed a bug for the documentation that also includes this.

major semver bump planned

This is an auto-generated regeneration of the .pb.go files by cloud.google.com/go/internal/gapicgen. Once this PR is submitted, genbot will update the corresponding PR to depend on the newer version of go-genproto, and assign reviewers. Whilst this or any regen PR is open in go-genproto, genbot will not create any more regeneration PRs. If all regen PRs are closed, gapicgen will create a new set of regeneration PRs once per night. If you have been assigned to review this PR, please: - Ensure that CI is passing. If it's failing, it requires your manual attention. - Approve and submit this PR if you believe it's ready to ship. That will prompt genbot to assign reviewers to the google-cloud-go PR. Corresponding google-cloud-go PR: googleapis/google-cloud-go#3765 Changes: build: update package name to already published name PiperOrigin-RevId: 360469636 Source-Link: googleapis/googleapis@c5435cb feat: allow to disable webhook invocation per request PiperOrigin-RevId: 360468675 Source-Link: googleapis/googleapis@a031936 build(policytroubleshooter): update BUILD file for policy troubleshooter PiperOrigin-RevId: 360450845 Source-Link: googleapis/googleapis@2f0798a fix(functions): Fix service namespace in grpc_service_config. PiperOrigin-RevId: 360443299 Source-Link: googleapis/googleapis@7cef880 chore(apigateway): Set namespaces for Ruby, PHP, and C# clients PiperOrigin-RevId: 360255957 Source-Link: googleapis/googleapis@7f8b19b

…cies to v2 (googleapis#544) * deps: update dependency com.google.cloud:google-cloud-shared-dependencies to v2 * Update pom.xml Co-authored-by: Neenu Shaji <[email protected]>

🤖 I have created a release \*beep\* \*boop\* --- ## [2.3.0](https://www.github.com/googleapis/java-spanner-jdbc/compare/v2.2.6...v2.3.0) (2021-08-11) ### Features * add support for tagging to JDBC connection ([googleapis#270](https://www.github.com/googleapis/java-spanner-jdbc/issues/270)) ([a4bd82c](https://www.github.com/googleapis/java-spanner-jdbc/commit/a4bd82c8e4ce8b7179b943ac06b049598276f1b4)) ### Dependencies * update dependency com.google.cloud:google-cloud-shared-dependencies to v2 ([googleapis#544](https://www.github.com/googleapis/java-spanner-jdbc/issues/544)) ([366430d](https://www.github.com/googleapis/java-spanner-jdbc/commit/366430dc270edd09de1a0749ba360f312897b1aa)) * update dependency com.google.cloud:google-cloud-spanner-bom to v6.10.0 ([googleapis#537](https://www.github.com/googleapis/java-spanner-jdbc/issues/537)) ([8655ae5](https://www.github.com/googleapis/java-spanner-jdbc/commit/8655ae5955f5385a9d6445e13264427d73c4d37e)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

feat!: add support for CommitStats

3872199

Adds support for returning CommitStats from read/write transactions.

olavloite added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Oct 24, 2020

olavloite requested review from a team as code owners October 24, 2020 10:59

google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Oct 24, 2020

olavloite mentioned this pull request Oct 24, 2020

feat!: add support for CommitStats #522

Closed

fix: add clirr ignored differences

def8768

product-auto-label bot added the api: spanner Issues related to the googleapis/java-spanner API. label Oct 25, 2020

olavloite requested a review from thiagotnunes October 25, 2020 09:41

skuruppu approved these changes Oct 29, 2020

View reviewed changes

...cloud-spanner/src/main/java/com/google/cloud/spanner/SessionPoolAsyncTransactionManager.java Show resolved Hide resolved

google-cloud-spanner/src/main/java/com/google/cloud/spanner/TransactionManagerImpl.java Outdated Show resolved Hide resolved

olavloite and others added 2 commits October 31, 2020 08:26

Merge branch 'master' into commit-stats2

5372dae

fix: error message should start with getCommitResponse

34fbda6

Co-authored-by: skuruppu <[email protected]>

olavloite changed the title ~~feat!: add support for CommitStats~~ feat: add support for CommitStats Oct 31, 2020

Merge branch 'master' into commit-stats2

8f61c2a

Merge branch 'master' into commit-stats2

041b34d

olavloite added 2 commits December 10, 2020 20:12

fix: remove overload delay

8299054

Merge branch 'master' into commit-stats2

9b710a7

elharo previously requested changes Jan 21, 2021

View reviewed changes

olavloite added 6 commits January 23, 2021 14:51

Merge branch 'master' into commit-stats2

8c15641

chore: cleanup after merge

919cf02

fix: update copyright years of new files

97ec917

test: fix flaky test

afeb0fd

test: skip commit stats tests on emulator

8524526

test: missed one commit stats tests against emulator

06b3e22

elharo reviewed Jan 28, 2021

View reviewed changes

thiagotnunes changed the title ~~feat: add support for CommitStats~~ feat!: add support for CommitStats Jan 29, 2021

olavloite added 2 commits January 30, 2021 17:10

fix: address review comments

6573a0f

Merge branch 'master' into commit-stats2

0b448c3

elharo reviewed Feb 3, 2021

View reviewed changes

olavloite added 2 commits February 3, 2021 18:07

Merge branch 'master' into commit-stats2

83039fb

chore: use junit assertion instead of truth

57ea714

skuruppu removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Feb 4, 2021

chore: replace truth asserts with junit asserts

66c5f88

olavloite requested a review from elharo February 4, 2021 08:06

olavloite added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Feb 4, 2021

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Feb 4, 2021

olavloite added 3 commits February 5, 2021 15:49

chore: replace truth assertions with junit

1d17973

chore: cleanup test and variable names

48ef21b

fix: rename test method and variables

1f33c42

elharo reviewed Feb 16, 2021

View reviewed changes

fix: address review comments

7d45ace

thiagotnunes merged commit 44aa384 into master Feb 17, 2021

thiagotnunes deleted the commit-stats2 branch February 17, 2021 03:20

This was referenced Feb 17, 2021

chore(master): release 4.0.0 #850

Merged

chore(master): release 4.0.1 #883

Merged

feat!: add support for CommitStats #544

feat!: add support for CommitStats #544

Conversation

olavloite commented Oct 24, 2020 • edited Loading

codecov bot commented Oct 24, 2020 • edited Loading

Codecov Report

skuruppu left a comment

Choose a reason for hiding this comment

olavloite commented Oct 31, 2020

generated-files-bot bot commented Nov 6, 2020

generated-files-bot bot commented Dec 5, 2020

generated-files-bot bot commented Dec 10, 2020

Choose a reason for hiding this comment

skuruppu commented Jan 23, 2021

skuruppu commented Jan 27, 2021

thiagotnunes commented Jan 27, 2021

elharo left a comment

Choose a reason for hiding this comment

thiagotnunes commented Jan 29, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skuruppu commented Feb 4, 2021

skuruppu commented Feb 10, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olavloite commented Oct 24, 2020 •

edited

Loading

codecov bot commented Oct 24, 2020 •

edited

Loading