Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge gcloud-java-bigquery into master #503

Merged
merged 61 commits into from
Dec 22, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
81744f7
Create packages for bigquery and outline spi layer
mziccard Nov 5, 2015
eeae71f
Rename Bigquery to BigQuery across bigquery module
mziccard Nov 9, 2015
13e976a
Merge pull request #326 from mziccard/bigquery
aozarov Nov 12, 2015
381d056
Add BigQueryRpc default implementation
mziccard Nov 12, 2015
e3a99c2
Add error codes, handle NOT_FOUND in BigQuery RPC, add javadoc
mziccard Nov 19, 2015
12aa555
Fix empty list bug, handle job state and error in listJobs
mziccard Nov 19, 2015
120fe12
Remove quota options, always use options.projectId()
mziccard Nov 24, 2015
762fa58
Merge pull request #367 from mziccard/bigquery
aozarov Nov 24, 2015
e4a5e55
Add DatasetInfo, DatasetId, TableId and Acl models and tests
mziccard Nov 24, 2015
3fc189b
Better javadoc, move static imports
mziccard Nov 26, 2015
dea8eff
Add SerializationTest class
mziccard Nov 26, 2015
3f0d511
Handle null role in Acl.fromPb, add test
mziccard Nov 27, 2015
4f05b98
Merge pull request #417 from mziccard/bigquery
aozarov Nov 29, 2015
8d2249e
Add TableInfo, CsvOptions, ExternalDataConfiguration and UserDefinedF…
mziccard Nov 30, 2015
b18666b
Refactor TableInfo and related classes
mziccard Dec 1, 2015
5f6357e
Minor javadoc updates, rename functionDefinition to content
mziccard Dec 2, 2015
79001cf
Replace format with FormatOptions in ExternalDataConfiguration
mziccard Dec 2, 2015
f84a3b1
Handle unset of Field's mode and description
mziccard Dec 3, 2015
bd58ee5
Make FormatOptions.csv return CsvOptions, remove getters from Externa…
mziccard Dec 3, 2015
66972df
Add TableType hierarcy to wrap different types of tables
mziccard Dec 3, 2015
ae2a484
Remove TableType, add TableInfo, ViewInfo ExternalTableInfo hierarcy
mziccard Dec 4, 2015
338ede1
Fix mode's nullability by internally storing a string
mziccard Dec 4, 2015
9df4005
Make BaseTableInfo abstract and other minor fixes
mziccard Dec 4, 2015
12185e8
Merge pull request #423 from mziccard/bigquery
aozarov Dec 4, 2015
722e4a7
Add JobInfo hierarcy and related classes
mziccard Dec 9, 2015
a88e0da
Merge pull request #442 from mziccard/bigquery
mziccard Dec 9, 2015
27f4d0d
Add InsertAllRequest and InsertAllResponse. Add model classes for tab…
mziccard Dec 9, 2015
1674c9c
Document ClassCastException in FieldValue
mziccard Dec 10, 2015
d2f0a82
Minor changes to FieldValue and InsertAllRequest
mziccard Dec 11, 2015
50d381f
Remove TableRow class
mziccard Dec 11, 2015
aacc383
Fix javadoc error, add checkNotNull to FieldValue getters
mziccard Dec 11, 2015
7dfca8e
Merge pull request #443 from mziccard/bigquery
aozarov Dec 11, 2015
20d4841
Add query request and query response classes and tests
mziccard Dec 11, 2015
9756dff
Refactor QueryRequest and QueryResponse
mziccard Dec 14, 2015
e16c027
Add QueryResult class and tests
mziccard Dec 14, 2015
c6e53c8
Rename QueryResponse.job to jobId, better javadoc, no errors as empty…
mziccard Dec 14, 2015
36c7406
Merge pull request #452 from mziccard/bigquery
aozarov Dec 14, 2015
3b70b20
Refactor BigQueryRpc and DefaultBigQueryRpc
mziccard Dec 14, 2015
0f765b7
Merge pull request #453 from mziccard/bigquery
aozarov Dec 14, 2015
e1dc3af
Add BigQuery service implementation
mziccard Dec 15, 2015
d676636
Fix minor errors
mziccard Dec 16, 2015
814264a
Merge pull request #454 from mziccard/bigquery
ajkannan Dec 17, 2015
e91f47b
Make checkstyle happy, fix minor issues, add better javadoc
mziccard Dec 17, 2015
f569bdd
Move StreamingBuffer to TableInfo, remove streamingBuffer from Extern…
mziccard Dec 17, 2015
433c286
Merge pull request #482 from mziccard/bigquery
ajkannan Dec 17, 2015
2271ab9
Merge branch 'master' into bigquery
mziccard Dec 18, 2015
a08e7bb
Change poms and code to match with master udpates
mziccard Dec 18, 2015
ec115cb
Merge pull request #488 from mziccard/bigquery
aozarov Dec 18, 2015
6bc4f2c
Add integration test
mziccard Dec 18, 2015
e46ce4f
Add BigQuery module to main pom
mziccard Dec 18, 2015
ba8acf1
Few minor fixes and make java8 happy
mziccard Dec 18, 2015
7eee528
Minor fixes to RemoteBigQueryHelper
mziccard Dec 21, 2015
3781aa7
Merge pull request #492 from mziccard/bigquery
aozarov Dec 21, 2015
bf6a3fe
Add BigQuery module README
mziccard Dec 21, 2015
6ee866a
Add troubleshooting section, fix line length, sort imports, remove un…
mziccard Dec 22, 2015
75a0ece
Merge pull request #496 from mziccard/bigquery
ajkannan Dec 22, 2015
862905f
Make BigQuery get/create/updated Table and Job methods generic
mziccard Dec 22, 2015
a0f5f1b
Merge pull request #500 from mziccard/bigquery
aozarov Dec 22, 2015
d299e06
Merge branch 'master' into bigquery
mziccard Dec 22, 2015
7b1b0dc
Add BigQuery to common README.md and TESTING.md
mziccard Dec 22, 2015
a1299c7
Merge pull request #502 from mziccard/bigquery
mziccard Dec 22, 2015
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ This client supports the following Google Cloud Platform services:
- [Google Cloud Datastore] (#google-cloud-datastore)
- [Google Cloud Storage] (#google-cloud-storage)
- [Google Cloud Resource Manager] (#google-cloud-resource-manager)
- [Google Cloud BigQuery] (#google-cloud-bigquery)

> Note: This client is a work-in-progress, and may occasionally
> make backwards-incompatible changes.
Expand Down Expand Up @@ -214,6 +215,51 @@ while (projectIterator.hasNext()) {
}
```

Google Cloud BigQuery
----------------------

- [API Documentation][bigquery-api]
- [Official Documentation][cloud-bigquery-docs]

#### Preview

Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you
must [supply credentials](#authentication) and a project ID if running this snippet elsewhere.

```java
import com.google.gcloud.bigquery.BaseTableInfo;
import com.google.gcloud.bigquery.BigQuery;
import com.google.gcloud.bigquery.BigQueryOptions;
import com.google.gcloud.bigquery.Field;
import com.google.gcloud.bigquery.JobStatus;
import com.google.gcloud.bigquery.LoadJobInfo;
import com.google.gcloud.bigquery.Schema;
import com.google.gcloud.bigquery.TableId;
import com.google.gcloud.bigquery.TableInfo;

BigQuery bigquery = BigQueryOptions.defaultInstance().service();
TableId tableId = TableId.of("dataset", "table");
BaseTableInfo info = bigquery.getTable(tableId);
if (info == null) {
System.out.println("Creating table " + tableId);
Field integerField = Field.of("fieldName", Field.Type.integer());
bigquery.create(TableInfo.of(tableId, Schema.of(integerField)));
} else {
System.out.println("Loading data into table " + tableId);
LoadJobInfo loadJob = LoadJobInfo.of(tableId, "gs://bucket/path");
loadJob = bigquery.create(loadJob);
while (loadJob.status().state() != JobStatus.State.DONE) {
Thread.sleep(1000L);
loadJob = bigquery.getJob(loadJob.jobId());
}
if (loadJob.status().error() != null) {
System.out.println("Job completed with errors");
} else {
System.out.println("Job succeeded");
}
}
```

Troubleshooting
---------------

Expand Down Expand Up @@ -276,3 +322,7 @@ Apache 2.0 - See [LICENSE] for more information.

[resourcemanager-api]:http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/resourcemanager/package-summary.html
[cloud-resourcemanager-docs]:https://cloud.google.com/resource-manager/

[cloud-bigquery]: https://cloud.google.com/bigquery/
[cloud-bigquery-docs]: https://cloud.google.com/bigquery/docs/overview
[bigquery-api]: http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/bigquery/package-summary.html
30 changes: 30 additions & 0 deletions TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ This library provides tools to help write tests for code that uses the following
- [Datastore] (#testing-code-that-uses-datastore)
- [Storage] (#testing-code-that-uses-storage)
- [Resource Manager] (#testing-code-that-uses-resource-manager)
- [BigQuery] (#testing-code-that-uses-bigquery)

### Testing code that uses Datastore

Expand Down Expand Up @@ -103,5 +104,34 @@ You can test against a temporary local Resource Manager by following these steps

This method will block until the server thread has been terminated.

### Testing code that uses BigQuery

Currently, there isn't an emulator for Google BigQuery, so an alternative is to create a test
project. `RemoteBigQueryHelper` contains convenience methods to make setting up and cleaning up the
test project easier. To use this class, follow the steps below:

1. Create a test Google Cloud project.

2. Download a [JSON service account credentials file][create-service-account] from the Google
Developer's Console.

3. Create a `RemoteBigQueryHelper` object using your project ID and JSON key.
Here is an example that uses the `RemoteBigQueryHelper` to create a dataset.
```java
RemoteBigQueryHelper bigqueryHelper =
RemoteBigQueryHelper.create(PROJECT_ID, new FileInputStream("/path/to/my/JSON/key.json"));
BigQuery bigquery = bigqueryHelper.options().service();
String dataset = RemoteBigQueryHelper.generateDatasetName();
bigquery.create(DatasetInfo.builder(dataset).build());
```

4. Run your tests.

5. Clean up the test project by using `forceDelete` to clear any datasets used.
Here is an example that clears the dataset created in Step 3.
```java
RemoteBigQueryHelper.forceDelete(bigquery, dataset);
```

[cloud-platform-storage-authentication]:https://cloud.google.com/storage/docs/authentication?hl=en#service_accounts
[create-service-account]:https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount
Loading