Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce BigQuery Regionalization Support #971

Merged
merged 4 commits into from
Apr 3, 2018

Conversation

dwsupplee
Copy link
Contributor

This change set introduces the ability to supply a location when managing jobs.

Some usage examples:

use Google\Cloud\BigQuery\BigQueryClient;

// This default location will be used for creating datasets as well as any job activity
// where we can't infer the location from a network request.
$client = new BigQueryClient([
    'location' => 'asia-northeast1'
]);

// All calls through this job will be routed to the `asia-northeast1` region, using
// the default supplied in the client config.
$job = $client->job('an-existing-job');

// Users can explicitly specify the location if they did not include one in the client
// config or want to override it. With an existing job...
$job = $client->job('an-existing-job', [
    'location' => 'asia-northeast1'
]);

// And when creating a job configuration...
$jobConfig = $client->queryConfig('SELECT * FROM `dataset.table`')
    ->location('asia-northeast1');

foreach ($client->datasets() as $dataset) {
    // `asia-northeast1` is not defaulted here, as network requests have given us
    // the necessary location data and we may be pulling in datasets from other regions. 
    $jobConfig = $dataset->table('my_table')
        ->load('somedata');
}

// Defaults to `asia-northeast1`.
$dataset = $client->createDataset('my_new_dataset');

@dwsupplee dwsupplee added the api: bigquery Issues related to the BigQuery API. label Mar 30, 2018
@dwsupplee dwsupplee requested review from jdpedrie and tswast March 30, 2018 14:30
@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Mar 30, 2018
@dwsupplee
Copy link
Contributor Author

/cc @jjolma If you would like to take a look as well :)

Copy link
Contributor

@jdpedrie jdpedrie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a couple little doc things

* @type string $location If provided, determines the default geographic
* location used when creating datasets and managing jobs. Please
* note: This is only required for jobs started outside of the US
* and EU regions. Also, if location metadata has already been

This comment was marked as spam.

This comment was marked as spam.

@@ -537,6 +577,10 @@ public function createDataset($id, array $options = [])
unset($options['metadata']);
}

if (!isset($options['location']) && $this->location) {

This comment was marked as spam.

This comment was marked as spam.

class RegionalizationTest extends BigQueryTestCase
{
const REGION_ASIA = 'asia-northeast1';
const REGION_US = 'US';

This comment was marked as spam.

self::$datasetAsia->id(),
self::$tableAsia->id()
)
)->location($location);

This comment was marked as spam.

This comment was marked as spam.

$caught = false;
$copyTable = self::$dataset->table(self::TESTING_PREFIX);
$copy = self::$tableAsia->copy($copyTable)
->location($location);

This comment was marked as spam.

{
$caught = false;
$extract = self::$tableAsia->extract(
self::$bucket->object(self::TESTING_PREFIX)

This comment was marked as spam.

This comment was marked as spam.

file_get_contents(__DIR__ . '/data/table-data.json')
)
->sourceFormat('NEWLINE_DELIMITED_JSON')
->location($location);

This comment was marked as spam.

];
}

private function assertExceptionShouldBeThrown($location, $caught)

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

@dwsupplee
Copy link
Contributor Author

@tswast Thanks for the solid feedback. PTAL and let me know what you think of the updates.

Copy link

@tswast tswast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Much clearer. Thanks!

@jjolma
Copy link

jjolma commented Mar 30, 2018

LGTM. Thanks!

@dwsupplee dwsupplee merged commit d4290f0 into googleapis:master Apr 3, 2018
@dwsupplee dwsupplee mentioned this pull request Apr 3, 2018
gcf-owl-bot bot added a commit that referenced this pull request Dec 12, 2024
Source-Link: googleapis/googleapis@7d0c6be

Source-Link: googleapis/googleapis-gen@2ddc476
Copy-Tag: eyJwIjoiQ29tcHV0ZS8uT3dsQm90LnlhbWwiLCJoIjoiMmRkYzQ3NmQ0Y2NlNmYwN2I1MWNjNjk2ZmZjM2UwYzRjNzYxYmIxOCJ9
bshaffer pushed a commit that referenced this pull request Dec 12, 2024
Source-Link: googleapis/googleapis@7d0c6be
Source-Link: googleapis/googleapis-gen@2ddc476
Copy-Tag: eyJwIjoiQ29tcHV0ZS8uT3dsQm90LnlhbWwiLCJoIjoiMmRkYzQ3NmQ0Y2NlNmYwN2I1MWNjNjk2ZmZjM2UwYzRjNzYxYmIxOCJ9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants