Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement memory management / limiting in SymmetricHashJoinExec #5636

Closed
alamb opened this issue Mar 18, 2023 · 1 comment · Fixed by #5937
Closed

Implement memory management / limiting in SymmetricHashJoinExec #5636

alamb opened this issue Mar 18, 2023 · 1 comment · Fixed by #5937

Comments

@alamb
Copy link
Contributor

alamb commented Mar 18, 2023

          Actually there is also `SymmetricHashJoinExec` -- but, I suppose, its memory management is a bit more complicated and may be behind the scope of this issue:
  • It's, sort of, isolated -- at this moment there are no options for planning this operator with DataFusion planner
  • I don't think that simply throwing error and aborting execution is acceptable for SymmetricHashJoinExec -- if my understanding is correct -- main use case for this operator is joining two unbounded sources (streaming jobs), and from this point of view it doesn't make much sense to limit memory without any spilling fallbacks (subjectively, it doesn't seem correct to fail data streaming job in case of memory overallocation attempt)

My proposal here would be to file separate issue for SymmetricHashJoinExec memory management, and (as I see it) implement memory limitation along with data spilling. Maybe, we can go for it when we have reliable spilling for HashJoinExec, however, prior to it is also an option.

@ozankabak, @metesynnada it would be great to hear your thoughts on it.

Originally posted by @korowa in #5220 (comment)

@metesynnada
Copy link
Contributor

I will be looking into this in a week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants