Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery testing suite that runs against BQ's day 0 region #28397

Merged
merged 29 commits into from
Oct 3, 2023
Merged
Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
c1e7734
support creating dataset in specified region; do it for BigQueryToTab…
ahmedabu98 Sep 7, 2023
75615ba
BigQueryIOStorageWriteIT
ahmedabu98 Sep 7, 2023
c5de23a
add options to registrar; StorageApiDirectWriteProtosIT
ahmedabu98 Sep 7, 2023
ec12f41
StorageApiSinkFailedRowsIT
ahmedabu98 Sep 7, 2023
7643895
StorageApiSinkRowUpdateIT
ahmedabu98 Sep 7, 2023
0b4a183
StorageApiSinkSchemaUpdateIT
ahmedabu98 Sep 7, 2023
3beb96c
TableRowToStorageApiProtoIT
ahmedabu98 Sep 7, 2023
3545d65
BigQuerySchemaUpdateOptionsIT; BigQueryTimePartitioningClusteringIT
ahmedabu98 Sep 7, 2023
c219e51
BigQueryIOStorageQueryIT; BigQueryIOStorageReadIT
ahmedabu98 Sep 8, 2023
02b272f
BigQueryIOStorageReadTableRowIT
ahmedabu98 Sep 11, 2023
b6fa6c6
new gradle task
ahmedabu98 Sep 11, 2023
0956fb4
spotless
ahmedabu98 Sep 11, 2023
945e09d
Merge branch 'master' of https://github.com/ahmedabu98/beam into earl…
ahmedabu98 Sep 26, 2023
8053b7a
create yaml workflow
ahmedabu98 Sep 26, 2023
95b7cff
spotless
ahmedabu98 Sep 26, 2023
02ad45b
test locally
ahmedabu98 Sep 26, 2023
e30a972
Merge branch 'apache:master' into master
ahmedabu98 Sep 27, 2023
6e878d4
add step to authenticate GCP
ahmedabu98 Sep 28, 2023
07a2055
Merge branch 'early_rollout_bq_tests' of https://github.com/ahmedabu9…
ahmedabu98 Sep 28, 2023
e008b5f
add file loads streaming test; add option to query with location
ahmedabu98 Sep 28, 2023
2fdf651
pass bq location to query operation
ahmedabu98 Sep 28, 2023
e939ccf
debug
ahmedabu98 Sep 28, 2023
6d60be2
pass location into query operations
ahmedabu98 Sep 28, 2023
88e2ec5
forgot to add it one test
ahmedabu98 Sep 29, 2023
d87e169
checks write permission
ahmedabu98 Sep 29, 2023
86727d0
use self-hosted machine
ahmedabu98 Sep 29, 2023
14a60ca
add some direction on what to do when tests fail; make early rollout …
ahmedabu98 Oct 2, 2023
2bc98d9
one more comment
ahmedabu98 Oct 2, 2023
759ce06
fix time partitioning test
ahmedabu98 Oct 3, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
forgot to add it one test
ahmedabu98 committed Sep 29, 2023

Verified

This commit was signed with the committer’s verified signature. The key has expired.
pascalkuthe Pascal Kuthe
commit 88e2ec5f8d177174116e6599cdac9c0f60132534
Original file line number Diff line number Diff line change
@@ -121,9 +121,9 @@ public static Iterable<Object[]> data() {
// an updated schema. If that happens consistently, just increase these two numbers
// to give it more time.
// Total number of rows written to the sink
private static final int TOTAL_N = 60;
private static final int TOTAL_N = 70;
// Number of rows with the original schema
private static final int ORIGINAL_N = 50;
private static final int ORIGINAL_N = 60;

private final Random randomGenerator = new Random();

@@ -484,7 +484,7 @@ public void checkRowsWithUpdatedSchema(
throws IOException, InterruptedException {
List<TableRow> actualRows =
BQ_CLIENT.queryUnflattened(
String.format("SELECT * FROM [%s]", tableSpec), PROJECT, true, false);
String.format("SELECT * FROM [%s]", tableSpec), PROJECT, true, false, bigQueryLocation);

for (TableRow row : actualRows) {
// Rows written to the table should not have the extra field if