[Spark] Restore memory sensitive GBK translation (#33520) #33521
Conversation
JozoVilcek
changed the title
JozoVilcek
changed the title
|
Assigning reviewers. If you would like to opt out of this review, comment R: @damccorm added as fallback since no labels match configuration Available commands:
The PR bot will only process comments in the main thread (not review comments). |
There was a problem hiding this comment.
We definitely should add some sort of test for the behavior, because otherwise we risk precisely the same situation - some future optimization will break this one. I'd also like to solve the conditions under which the optimization is correct, I'll be happy to assist with that. :)
| // we can drop the windows and recover them later | ||
| groupedByKey = | ||
| GroupNonMergingWindowsFunctions.groupByKeyInGlobalWindow( | ||
| inRDD, keyCoder, coder.getValueCoder(), partitioner); |
There was a problem hiding this comment.
Actually, seems that these optimizations are not mutually exclusive. We might want to apply both.
| GroupNonMergingWindowsFunctions.groupByKeyInGlobalWindow( | ||
| inRDD, keyCoder, coder.getValueCoder(), partitioner); | ||
| } else if (GroupNonMergingWindowsFunctions.isEligibleForGroupByWindow(windowingStrategy)) { | ||
| if (GroupNonMergingWindowsFunctions.isEligibleForGroupByWindow(windowingStrategy)) { | ||
| // we can have a memory sensitive translation for non-merging windows | ||
| groupedByKey = |
There was a problem hiding this comment.
I know this is not part of this issue, but there is unresolved problem with this - under certain circumstances there is a need for GBK result to be Reiterable (e.g. CoGroupByKey). Could we figure out a way to automatically figure out if this expansion is correct? From the top of my head - if a Pipeline does not use CoGBK at all (better if we can look is this transform is part of CoGBK expansion), then this might be OK.
Restore priority of memory sensitive GBK translation in spark-runner to avoid OOM #33520
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.