BigQuery import from GCS buckets #113
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It's not needed anymore, presumably due to changes in the move() algorithm.
…dshift_load_when_bucket_load_not_available
| @@ -152,7 +158,8 @@ def test_select_lastpass_session_by_config(self, | |||
| default_boto3_session=PleaseInfer.token, | |||
| default_gcp_creds=PleaseInfer.token, | |||
| default_gcs_client=PleaseInfer.token, | |||
| scratch_s3_url='s3://foo/') | |||
| scratch_s3_url='s3://foo/', | |||
| scratch_gcs_url='gs://bar/') | |||
There was a problem hiding this comment.
fiddler on the roof style singing TRADITION
crvena-sonja
approved these changes
Nov 11, 2020
Co-authored-by: Sonja Duric <[email protected]>
Co-authored-by: Sonja Duric <[email protected]>
Co-authored-by: Sonja Duric <[email protected]>
Co-authored-by: Sonja Duric <[email protected]>
Co-authored-by: Sonja Duric <[email protected]>
Co-authored-by: Sonja Duric <[email protected]>
Co-authored-by: Sonja Duric <[email protected]>
| @@ -135,6 +135,34 @@ def test_s3_scratch_bucket_via_prefix_assumed_role(self, | |||
| self.assertIsNone(out) | |||
| mock_get_config.assert_called_with('records_mover', 'bluelabs') | |||
|
|
|||
| @patch('records_mover.creds.base_creds.get_config') | |||
| @patch('records_mover.creds.base_creds.os') | |||
There was a problem hiding this comment.
E128 continuation line under-indented for visual indent
| @@ -135,6 +135,34 @@ def test_s3_scratch_bucket_via_prefix_assumed_role(self, | |||
| self.assertIsNone(out) | |||
| mock_get_config.assert_called_with('records_mover', 'bluelabs') | |||
|
|
|||
| @patch('records_mover.creds.base_creds.get_config') | |||
| @patch('records_mover.creds.base_creds.os') | |||
| def test_gcs_scratch_bucket_configured_true(self, | |||
There was a problem hiding this comment.
E128 continuation line under-indented for visual indent
| mock_get_config.assert_called_with('records_mover', 'bluelabs') | ||
|
|
||
| @patch('records_mover.creds.base_creds.get_config') | ||
| @patch('records_mover.creds.base_creds.os') |
There was a problem hiding this comment.
E128 continuation line under-indented for visual indent
|
|
||
| @patch('records_mover.creds.base_creds.get_config') | ||
| @patch('records_mover.creds.base_creds.os') | ||
| def test_gcs_scratch_bucket_not_configured_true(self, |
There was a problem hiding this comment.
E128 continuation line under-indented for visual indent
| @@ -163,6 +191,36 @@ def test_s3_scratch_bucket_no_config_file(self, | |||
| self.assertIsNone(out) | |||
| mock_get_config.assert_called_with('records_mover', 'bluelabs') | |||
|
|
|||
| @patch('records_mover.creds.base_creds.get_config') | |||
| @patch('records_mover.creds.base_creds.os') | |||
There was a problem hiding this comment.
E128 continuation line under-indented for visual indent
| @@ -163,6 +191,36 @@ def test_s3_scratch_bucket_no_config_file(self, | |||
| self.assertIsNone(out) | |||
| mock_get_config.assert_called_with('records_mover', 'bluelabs') | |||
|
|
|||
| @patch('records_mover.creds.base_creds.get_config') | |||
| @patch('records_mover.creds.base_creds.os') | |||
| def test_gcs_scratch_bucket_no_config_file_true(self, | |||
There was a problem hiding this comment.
E128 continuation line under-indented for visual indent
| self.assertIsNone(out) | ||
| mock_get_config.assert_called_with('records_mover', 'bluelabs') | ||
|
|
||
| @patch('records_mover.creds.base_creds.os') |
There was a problem hiding this comment.
E128 continuation line under-indented for visual indent
| out = creds.default_scratch_gcs_url() | ||
| self.assertIsNone(out) | ||
|
|
||
| @patch('records_mover.creds.base_creds.os') |
There was a problem hiding this comment.
E128 continuation line under-indented for visual indent
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Right now we only have code to load data into BigQuery by streaming data over the network to it. This PR teaches it as well to be able to point to existing data in a GCS bucket.
This is important when data gets large - there are ways to move mass amount of data into a GCS bucket (e.g., GCP Cloud Data Transfer) that accomplish the goal much faster than trying to stream data down to your laptop and back up again. BigQuery is able to read from a bucket in parallel and over a much faster network in order to load data.
One major limitation remaining: all of the methods that Records Mover knows about to get data into that GCS bucket are inherently slow as they rely on streaming data down to Records Mover and then back up to the GCS bucket. In practice for large datasets (>100MB) folks will want to first get data into a GCS bucket and then use Records Mover.
Watch this space.
Depends on #120 and set to merge into it for clarity of diff.