-
Notifications
You must be signed in to change notification settings - Fork 126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add Connection interface #1374
Merged
stephaniewang526
merged 311 commits into
googleapis:main
from
stephaniewang526:query-interface
May 6, 2022
Merged
feat: add Connection interface #1374
stephaniewang526
merged 311 commits into
googleapis:main
from
stephaniewang526:query-interface
May 6, 2022
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
shollyman
reviewed
Aug 6, 2021
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryDryRunResult.java
Outdated
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryImpl.java
Outdated
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryResultSet.java
Outdated
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQuerySQLException.java
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/Connection.java
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/Connection.java
Outdated
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/ConnectionSettings.java
Show resolved
Hide resolved
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/QueryResultRowFormat.java
Outdated
Show resolved
Hide resolved
eb824b4
to
8935c10
Compare
54f0826
to
3595799
Compare
e2dc6b9
to
4584ef2
Compare
google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/it/ITBigQueryTest.java
Outdated
Show resolved
Hide resolved
google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/it/ITBigQueryTest.java
Show resolved
Hide resolved
stephaniewang526
added a commit
to stephaniewang526/java-bigquery
that referenced
this pull request
May 6, 2022
…erystorage Java client Relates to googleapis#1374
gcf-merge-on-green bot
pushed a commit
that referenced
this pull request
May 10, 2022
🤖 I have created a release *beep* *boop* --- ## [2.11.0](v2.10.10...v2.11.0) (2022-05-10) ### Features * add Connection interface ([#1374](#1374)) ([3804275](3804275)) * next release from main branch is 2.10.9 ([#1996](#1996)) ([f716427](f716427)) ### Bug Fixes * add native image configuration for Arrow ([#2018](#2018)) ([06cbe69](06cbe69)) * fix for flaky connection close issue ([#2034](#2034)) ([db3daac](db3daac)) ### Documentation * **sample:** remove unused dependency and add setup instructions ([#2010](#2010)) ([e2e9113](e2e9113)) ### Dependencies * update dependency com.google.apis:google-api-services-bigquery to v2-rev20220422-1.32.1 ([#2017](#2017)) ([b9fa786](b9fa786)) * update dependency com.google.apis:google-api-services-bigquery to v2-rev20220429-1.32.1 ([#2020](#2020)) ([78789a5](78789a5)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Hi Read of 100_000 rows takes 23930 ms. Mono.fromCallable { bigQueryOptionsBuilder.build().service }
.flatMap { context ->
val connectionSettings = ConnectionSettings.newBuilder()
.setRequestTimeout(10L)
.setUseReadAPI(true)
.setMaxResults(1000)
.setNumBufferedRows(1000)
.setUseQueryCache(true)
.build();
val connection = context.createConnection(connectionSettings)
val bqResult = connection.executeSelect(sql)
val result = Flux.usingWhen(
Mono.just(bqResult.resultSet),
{ resultSet -> resultSet.toFlux(bqResult.schema) },
{ _ -> Mono.fromRunnable<Unit> { connection.close() } }
)
Mono.just(Data(result, bqResult.schema.toSchema()))
}
...
fun ResultSet.toFlux(schema:Schema): Flux<DataRecord> {
return Flux.generate<DataRecord> { sink ->
if (next()) {
sink.next(toDataRecord(schema))
} else {
sink.complete()
}
}
} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
api: bigquery
Issues related to the googleapis/java-bigquery API.
cla: yes
This human has signed the Contributor License Agreement.
size: xl
Pull request size is extra large.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background: go/bq-sql-client-java
This PR provides a completely new Connection interface which defines separate APIs for different types of queries. This allows us to provide an industry standard way for database applications to build against the Java client library. We provide 3 new JDBC-esque API methods:
[In-scope for this PR] executeSelect - Only supports read only SELECT queries.
[In-scope for this PR] dryRun - returns som query processing statistics including schema and query parameters.
[Not in this PR] executeUpdate - Only supports DML and DDL.
[Not in this PR] execute - Any SQL - scripts, DML, DDL, SELECT etc statements.
We also integrate with the BigQueryStorage client library and use the high throughput Read API when applicable to parse query results using Arrow format. Arrow has shown better performance over Avro as the row serialization format in this experiment.