Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove obsolete and deprecated bigquery native write. #23557 #23558

Merged
merged 9 commits into from
Oct 12, 2022
Merged

Remove obsolete and deprecated bigquery native write. #23557 #23558

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
ff29677
Remove obsolete and deprecated bigquery native write.
robertwb Oct 10, 2022
c318966
changes
robertwb Oct 10, 2022
b9bd7b7
lint
robertwb Oct 10, 2022
1e838db
fix options check
robertwb Oct 10, 2022
482229d
fix no experiments case
robertwb Oct 11, 2022
989b682
changes
robertwb Oct 11, 2022
31d4998
Merge branch 'master' into no-native-bq-write
robertwb Oct 11, 2022
7c81c44
adapt tests
robertwb Oct 11, 2022
0c48d23
lint
robertwb Oct 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@
* Decreased TextSource CPU utilization by 2.3x (Java) ([#23193](https://github.com/apache/beam/issues/23193)).
* Fixed bug when using SpannerIO with RuntimeValueProvider options (Java) ([#22146](https://github.com/apache/beam/issues/22146)).
* Fixed issue for unicode rendering on WriteToBigQuery ([#10785](https://github.com/apache/beam/issues/10785))
* Remove obsolete variant of BigQuery Write ([#23564](https://github.com/apache/beam/issues/23564)).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe comment that old code using this variant will redirect to the new variant with a warning. This isn't a breaking change right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same line added in #23557 introduces a conflict. Could just change this line to

Remove obsolete variant of BigQuery Read and Write ([#23559](https://github.com/apache/beam/issues/23559)).

during rebase.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I merged these lines in the changelog.

* Bumped google-cloud-spanner dependency version to 3.x for Python SDK ([#21198](https://github.com/apache/beam/issues/21198)).

## New Features / Improvements
Expand Down
1 change: 0 additions & 1 deletion sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@
from apache_beam.io.gcp.bigquery_tools import BigQueryWrapper
from apache_beam.io.gcp.internal.clients import bigquery
from apache_beam.io.gcp.tests.bigquery_matcher import BigqueryMatcher
from apache_beam.runners.direct.test_direct_runner import TestDirectRunner
from apache_beam.testing import test_utils
from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
from apache_beam.testing.test_pipeline import TestPipeline
Expand Down
2 changes: 1 addition & 1 deletion sdks/python/apache_beam/io/gcp/bigquery.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1410,7 +1410,7 @@ def __next__(self):


@deprecated(since='2.11.0', current="WriteToBigQuery")
class BigQuerySink(*args, **kwargs):
def BigQuerySink(*args, **kwargs):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why change this from class to def?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This way one can still do io.Write(BigQuerySink(...)) even though the old class is gone.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it makes sense, add a test to make sure this fallback works.

Since any code calling the fallback will expect the same interface as the deprecated BigQuerySink right? Is the interface exactly the same, or could users get runtime errors in unexpected places?

If there are unexpected runtime errors consider fully sunsetting instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is intended to be a drop-in replacement, with the old variant deprecated over two years ago.

"""A deprecated alias for WriteToBigQuery."""
warnings.warn(
"Native sinks no longer implemented; "
Expand Down
1 change: 0 additions & 1 deletion sdks/python/apache_beam/runners/dataflow/ptransform_overrides.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@

# pytype: skip-file

from apache_beam.options.pipeline_options import DebugOptions
from apache_beam.options.pipeline_options import GoogleCloudOptions
from apache_beam.options.pipeline_options import StandardOptions
from apache_beam.pipeline import PTransformOverride
Expand Down