GH-15054: [Python] Finalize S3 on pytest exit to prevent shutdown crashes #33793
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
In a wheel though, we statically link everything, and then destruction order is undefined as usual? |
raulcd
reviewed
Jan 23, 2023
| @@ -265,3 +265,13 @@ def add_fs(doctest_namespace, request, tmp_path): | |||
| doctest_namespace["local_path"] = str(tmp_path) | |||
| doctest_namespace["path"] = str(path) | |||
| yield | |||
|
|
|||
There was a problem hiding this comment.
I think the linter error is only because it expects two empty lines here.
Suggested change
Yes, I think that could explain it. Still, this fix would require pyarrow users to know about and remember calling the finalize S3 method. I'm going to pause this idea and make an attempt at an arrow::Shutdown. |
|
Closing in favor of #33858 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For some reason destroying an S3Client at shutdown can cause a crash in our tests. It appears to be because S3's logging is already shut down. I wouldn't think this would happen because I would think the linker would tear down S3 shared libraries AFTER it tears down Arrow shared libraries.
However, we can also finalize S3 in conftest.py to prevent this crash.
It would still be good to understand the root cause. Otherwise we need to advise our users to call finalize_s3.