Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Adding CDC sample #2263

Merged
merged 103 commits into from
Nov 28, 2023
Merged
Changes from 1 commit
Commits
Show all changes
103 commits
Select commit Hold shift + click to select a range
c9db728
feat: Adding CDC Sample
farhan0102 Oct 4, 2023
df3fc13
fix: minor tweak
farhan0102 Nov 1, 2023
22e37ff
fix: minor tweak
farhan0102 Nov 1, 2023
effcd59
fix: minor style fix
farhan0102 Nov 1, 2023
9e5a6aa
fix: testing json type
farhan0102 Nov 1, 2023
7772351
fix: minor tweak
farhan0102 Nov 2, 2023
bb9ad7c
fix: Changing to json format
farhan0102 Nov 2, 2023
5541530
fix: field names
farhan0102 Nov 2, 2023
d8394e2
fix: testing
farhan0102 Nov 3, 2023
2552ff5
fix: added json type in bq to bq schema converter
farhan0102 Nov 3, 2023
5a479d6
fix: filepath
farhan0102 Nov 3, 2023
1abf42c
fix: filepath
farhan0102 Nov 3, 2023
578d3b9
fix: checkstyle
farhan0102 Nov 3, 2023
f2806e6
fix: filepath
farhan0102 Nov 3, 2023
868bb28
fix: filepath
farhan0102 Nov 3, 2023
931f4bd
fix: filepath
farhan0102 Nov 3, 2023
dc0c554
fix: checkstyle
farhan0102 Nov 3, 2023
bf5def7
fix: testing
farhan0102 Nov 3, 2023
0912819
fix: add brace
farhan0102 Nov 3, 2023
96c4064
fix: filepath
farhan0102 Nov 3, 2023
824b276
fix: filepath
farhan0102 Nov 3, 2023
1cd97fc
fix: json format
farhan0102 Nov 3, 2023
17a9c1c
fix: testing commas
farhan0102 Nov 6, 2023
b99a7a1
fix: remove white space
farhan0102 Nov 6, 2023
000716e
fix: trying out escaping in json
farhan0102 Nov 6, 2023
719899b
fix: added quotes around json dictionary
farhan0102 Nov 6, 2023
980fa10
fix: make json array
farhan0102 Nov 6, 2023
efb2a0e
fix: added braces
farhan0102 Nov 6, 2023
407d0d5
fix: removed braces
farhan0102 Nov 6, 2023
fe2ccca
fix: minor tweak
farhan0102 Nov 6, 2023
fa1db3b
fix: tweak to jsonarray
farhan0102 Nov 7, 2023
0eb1d7a
fix: tweak to jsonarray
farhan0102 Nov 7, 2023
a85aff7
fix: testing
farhan0102 Nov 7, 2023
2986c68
fix: testing
farhan0102 Nov 7, 2023
73cd65e
fix: trying exact json
farhan0102 Nov 7, 2023
b8e17dc
fix: testing
farhan0102 Nov 7, 2023
271df01
fix: removing whitespaces
farhan0102 Nov 7, 2023
2d6dec2
fix: testing with no square brackets
farhan0102 Nov 7, 2023
11e7984
fix: tweaking the json format
farhan0102 Nov 7, 2023
7b966cd
fix: trying without json field
farhan0102 Nov 7, 2023
00ae3d9
feat: Add in-stream retry logic for retryable and quota errors (#2243)
egreco12 Oct 6, 2023
2f31416
build: do not declare javadoc plugin version (#2268)
alicejli Oct 10, 2023
7cf0e5e
chore(deps): update dependency com.google.cloud:google-cloud-bigquery…
renovate-bot Oct 10, 2023
d057127
chore(deps): update dependency com.google.cloud:libraries-bom to v26.…
renovate-bot Oct 10, 2023
95b75fe
deps: update dependency com.google.cloud:google-cloud-bigquery to v2.…
renovate-bot Oct 10, 2023
91b0c43
deps: bumping google-cloud-shared-config to v1.5.8 (#2269)
ddixit14 Oct 10, 2023
3c45f48
deps: update dependency com.google.cloud:google-cloud-shared-dependen…
renovate-bot Oct 10, 2023
3150f60
chore(main): release 2.43.1-SNAPSHOT (#2256)
release-please[bot] Oct 10, 2023
66e7b6a
chore(main): release 2.44.0 (#2270)
release-please[bot] Oct 11, 2023
e23e301
build(deps): update dependency com.google.cloud:google-cloud-shared-c…
renovate-bot Oct 17, 2023
ca13e54
deps: update dependency org.json:json to v20231013 [security] (#2278)
renovate-bot Oct 17, 2023
8b1f356
deps: update dependency org.json:json to v20231013 - abandoned (#2276)
renovate-bot Oct 17, 2023
ba075b6
deps: update dependency com.google.cloud:google-cloud-bigquery to v2.…
renovate-bot Oct 17, 2023
10036ad
chore(deps): update dependency com.google.cloud:google-cloud-bigquery…
renovate-bot Oct 17, 2023
24dcf16
chore(deps): update dependency com.google.cloud:libraries-bom to v26.…
renovate-bot Oct 17, 2023
4d0bbe2
deps: update actions/checkout digest to b4ffde6 (#2281)
renovate-bot Oct 19, 2023
e63c4d1
chore: track customization in templates (#1876) (#2279)
gcf-owl-bot[bot] Oct 25, 2023
a273e60
chore(main): release 2.44.1-SNAPSHOT (#2271)
release-please[bot] Oct 25, 2023
70397ec
deps: update dependency com.google.cloud:google-cloud-shared-dependen…
renovate-bot Oct 25, 2023
1def89d
test: Add e2e test for default value use cases for default stream and…
GaoleMeng Oct 25, 2023
246cb29
chore(main): release 2.44.1 (#2286)
release-please[bot] Oct 26, 2023
8fe613f
chore(deps): update dependency com.google.cloud:google-cloud-bigquery…
renovate-bot Oct 26, 2023
6bee258
deps: update dependency com.google.cloud:google-cloud-bigquery to v2.…
renovate-bot Oct 26, 2023
53be322
chore(deps): update dependency com.google.cloud:libraries-bom to v26.…
renovate-bot Oct 27, 2023
c77ef23
feat: add support for proto3 optional tag (#2295)
agrawal-siddharth Oct 31, 2023
b5fb8f0
test: add e2e test for default value use case for stream writer (#2288)
GaoleMeng Oct 31, 2023
7a894cb
test: add e2e test to verify proto3 optional tag works (#2298)
agrawal-siddharth Oct 31, 2023
84dacd7
fix(bigquery): Make exponential backoff retry second based (#2212)
mina-asham Nov 1, 2023
3b8d302
fix: add native image configurations for com.google.rpc classes (#2305)
mpeddada1 Nov 6, 2023
2fa8443
chore: setup 2.38.x lts branch (#2208)
burkedavison Nov 6, 2023
ef933b7
deps: update dependency com.google.cloud:google-cloud-shared-dependen…
renovate-bot Nov 6, 2023
ed1289c
chore: update graalvm version to 22.3.3 in kokoro cfg files. (#1889) …
gcf-owl-bot[bot] Nov 6, 2023
4871fad
build(deps): update dependency com.google.cloud:google-cloud-shared-c…
renovate-bot Nov 6, 2023
4bb1112
chore(main): release 2.44.2-SNAPSHOT (#2289)
release-please[bot] Nov 6, 2023
5aa3cf2
deps: update dependency com.google.cloud:google-cloud-bigquery to v2.…
renovate-bot Nov 7, 2023
a4455e4
chore(main): release 2.45.0 (#2306)
release-please[bot] Nov 7, 2023
1388610
fix: add comment/documentation for setRetrySettings (#2309)
yirutang Nov 7, 2023
a00e8f3
feat: Add integration tests with RetrySettings enabled. (#2275)
egreco12 Nov 13, 2023
4259507
feat: enable 20MB request limit, this feature is allowlist only. (#2311)
yirutang Nov 14, 2023
e86dbbe
fix: Create next attempt after first attempt to initialize exponentia…
egreco12 Nov 14, 2023
1923894
feat: add new configuration for nightly retry-related tests. (#2319)
egreco12 Nov 14, 2023
8a9d39b
deps: update actions/github-script action to v7 (#2317)
renovate-bot Nov 15, 2023
af48969
chore(deps): update dependency com.google.cloud:libraries-bom to v26.…
renovate-bot Nov 15, 2023
afb13f3
fix: enable request limit e2e test. (#2320)
yirutang Nov 15, 2023
47b20c3
chore(main): release 2.45.1-SNAPSHOT (#2310)
release-please[bot] Nov 15, 2023
30e19b4
chore(main): release 2.46.0 (#2322)
release-please[bot] Nov 15, 2023
8b99088
chore: fix retry-quota jobs (#2324)
Neenu1995 Nov 16, 2023
26e6aef
chore: fix retry-quota jobs (#2325)
Neenu1995 Nov 16, 2023
766722d
chore: fix graalvm kokoro builds (#2327)
Neenu1995 Nov 21, 2023
2f1b892
Update CDC example to use work with UPSERT
PhongChuong Nov 23, 2023
71f4118
fix import order + reading json file
PhongChuong Nov 23, 2023
8cbea70
fix lint errors
PhongChuong Nov 23, 2023
86282e5
fix more lint errors due to import order
PhongChuong Nov 23, 2023
6556c7a
fix even more lint errors due to import order. Sad.
PhongChuong Nov 23, 2023
8637985
fix json path issue
PhongChuong Nov 24, 2023
fb23c25
fix format
PhongChuong Nov 24, 2023
18c705a
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Nov 24, 2023
08af044
Add modified customer data to the sample
PhongChuong Nov 24, 2023
a5ab8d9
Fix formating.
PhongChuong Nov 24, 2023
5fb5166
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Nov 24, 2023
bf21b3c
Fix lint errors.
PhongChuong Nov 24, 2023
2415a28
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Nov 24, 2023
31f799b
Fix modifiedCustomersDataFile reading from the wrong input arg.
PhongChuong Nov 28, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
test: add e2e test to verify proto3 optional tag works (#2298)
agrawal-siddharth authored and PhongChuong committed Nov 23, 2023
commit 7a894cb2cf844b0479216c6537b44e3d7654d14c
Original file line number Diff line number Diff line change
@@ -28,6 +28,7 @@
import com.google.cloud.bigquery.Field.Mode;
import com.google.cloud.bigquery.Schema;
import com.google.cloud.bigquery.storage.test.Test.*;
import com.google.cloud.bigquery.storage.test.TestOptional.*;
import com.google.cloud.bigquery.storage.v1.*;
import com.google.cloud.bigquery.storage.v1.AppendRowsRequest.MissingValueInterpretation;
import com.google.cloud.bigquery.storage.v1.Exceptions.AppendSerializationError;
@@ -207,6 +208,15 @@ ProtoRows CreateProtoRows(String[] messages) {
return rows.build();
}

ProtoRows CreateProtoOptionalRows(String[] messages) {
ProtoRows.Builder rows = ProtoRows.newBuilder();
for (String message : messages) {
FooOptionalType foo = FooOptionalType.newBuilder().setFoo(message).build();
rows.addSerializedRows(foo.toByteString());
}
return rows.build();
}

ProtoRows CreateProtoRowsMultipleColumns(String[] messages) {
ProtoRows.Builder rows = ProtoRows.newBuilder();
for (String message : messages) {
@@ -274,6 +284,35 @@ public void testBatchWriteWithCommittedStreamEU()
assertEquals(3, response2.get().getAppendResult().getOffset().getValue());
}

@Test
public void testProto3OptionalBatchWriteWithCommittedStream()
throws IOException, InterruptedException, ExecutionException {
WriteStream writeStream =
client.createWriteStream(
CreateWriteStreamRequest.newBuilder()
.setParent(tableId)
.setWriteStream(
WriteStream.newBuilder().setType(WriteStream.Type.COMMITTED).build())
.build());
StreamWriter streamWriter =
StreamWriter.newBuilder(writeStream.getName())
.setWriterSchema(ProtoSchemaConverter.convert(FooOptionalType.getDescriptor()))
.build();
LOG.info("Sending one message");

ApiFuture<AppendRowsResponse> response =
streamWriter.append(CreateProtoOptionalRows(new String[] {"aaa"}), 0);
assertEquals(0, response.get().getAppendResult().getOffset().getValue());

LOG.info("Sending two more messages");
ApiFuture<AppendRowsResponse> response1 =
streamWriter.append(CreateProtoOptionalRows(new String[] {"bbb", "ccc"}), 1);
ApiFuture<AppendRowsResponse> response2 =
streamWriter.append(CreateProtoOptionalRows(new String[] {""}), 3);
assertEquals(1, response1.get().getAppendResult().getOffset().getValue());
assertEquals(3, response2.get().getAppendResult().getOffset().getValue());
}

@Test
public void testJsonStreamWriterCommittedStream()
throws IOException, InterruptedException, ExecutionException,
26 changes: 26 additions & 0 deletions google-cloud-bigquerystorage/src/test/proto/optionalTest.proto
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
/*
* Copyright 2023 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
syntax = "proto3";

package com.google.cloud.bigquery.storage.test;

option java_package = "com.google.cloud.bigquery.storage.test";
option java_outer_classname = "TestOptional";

message FooOptionalType {
optional string foo = 1;
}

2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
@@ -52,7 +52,7 @@
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<project.protobuf-java.version>3.14.0</project.protobuf-java.version>
<project.protobuf-java.version>3.23.0</project.protobuf-java.version>
<github.global.server>github</github.global.server>
<site.installationModule>google-cloud-bigquerystorage-parent</site.installationModule>
</properties>