Crash on exist with pyarrow >= 12.0.0

[mattw-nws]

We're getting this at the end of a run of ngen with routing in some environments, with pyarrow >= 12.0.0 (up to 12.0.1 at time of writing):

/arrow/cpp/src/arrow/filesystem/s3fs.cc:2598:  arrow::fs::FinalizeS3 was not called even though S3 was initialized.  This could lead to a segmentation fault at exit
/home/runner/work/_temp/b7ac2c59-df3f-4a2c-97fc-aac3460ed081.sh: line 2:  4242 Segmentation fault      (core dumped) ./cmake_build/test/test_routing_pybind

e.g. here: https://github.com/NOAA-OWP/ngen/actions/runs/5348001534/jobs/9697297839?pr=534#step:7:121

This seems related to apache/arrow#15054 and apache/arrow#33858 ... but it seems like those are closed and should be incorporated into pyarrow 12.0.0 ... it may be that there needs to be an explicit initialize or finalize call for the S3 API to make this go away, but it's unclear.

Current behavior

t-route+ngen crashes at exit with the above message ... mainly this is causing us to have failing tests.

Expected behavior

No crash

Steps to replicate behavior (include URLs)

1. Re-running the ngen+t-route tests with pyarrow >= 12.0.0 seems to be the most reliable way so far observed, though it's not 100% reliable. But this may be happening more often and people not paying it any attention (since it happens at the end of the process).

Screenshots

[pitrou]

We hope to have a reliable fix for this in apache/arrow#36442
If you are able to easily reproduce the issue, could you perhaps try one of the wheels produced in apache/arrow#36442 (comment)
(click on the "crossbow" icon corresponding to your configuration, then click on "summary" in the Github Actions UI, and you should see a wheel artifact)