Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix 'Too many open files' on fuzz test. #12961

Merged
merged 1 commit into from
Oct 16, 2024

Conversation

dhegberg
Copy link
Contributor

Which issue does this PR close?

Closes #10674.

Rationale for this change

  • On MacOS a batch size of 1,000,000 with a 10KB pool size results in a panic Too many open files.
  • The spill threshold for 10KB pool size already spills at the 20,000 batch size, so going to 1,000,000 for this test seem unnecessary.

What changes are included in this PR?

  • change upper spill test threshold batch size to 500,000 from 1,000,000.
  • rename test to test_sort_10k_mem to reflect the 10KB pool size to be consistent with the test test_sort_100k_mem which uses a 100KB pool size.

Are these changes tested?

Changes are to tests.

Are there any user-facing changes?

No

@github-actions github-actions bot added the core Core DataFusion crate label Oct 16, 2024
@alamb
Copy link
Contributor

alamb commented Oct 16, 2024

THANK YOU ! I found this too many open files really annoying as well

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @dhegberg

I ran this PR and verified that it no longer fails locally for me

cargo test --test fuzz

Thank you 🙏

@dhegberg
Copy link
Contributor Author

THANK YOU ! I found this too many open files really annoying as well

Happy to help. Let me know if anything else is needed before merging.

@alamb alamb merged commit 589c830 into apache:main Oct 16, 2024
24 checks passed
@alamb
Copy link
Contributor

alamb commented Oct 16, 2024

I think it is all good -- thanks agian

@dhegberg dhegberg deleted the fix_fuzz_test_macos branch October 17, 2024 00:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Core DataFusion crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

test_sort_1k_mem failed with too many open file error when I run locally
2 participants