Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert three commits related to supporting custom coder in reshuffle #33414

Merged
merged 1 commit into from
Dec 18, 2024

Conversation

shunping
Copy link
Contributor

It is causing some internal test failure so we revert it for now.

- Fix custom coder not being used in Reshuffle (global window) (apache#33339)
- Fix custom coders not being used in Reshuffle (non global window) apache#33363
- Add missing to_type_hint to WindowedValueCoder apache#33403
@chamikaramj
Copy link
Contributor

LGTM. Thanks.

Copy link
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@shunping
Copy link
Contributor Author

Failed tests are unrelated to the changes.

@chamikaramj chamikaramj merged commit e9424b9 into apache:master Dec 18, 2024
85 of 91 checks passed
@robertwb
Copy link
Contributor

Just a thought, as this changes coders in some cases, should this be guarded by the update compatibility flag? https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/pipeline_options.py#L592

@shunping
Copy link
Contributor Author

shunping commented Dec 19, 2024

As the flag is defined in "StreamingOptions", is it previously designed for using in streaming case?

@kennknowles
Copy link
Member

As the flag is defined in "StreamingOptions", is it previously designed for using in streaming case?

Yes, it is designed for "streaming update" where you may have an in-progress aggregation in a shuffle when you do a pipeline update. Then you need the state to be compatible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants