You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug, including details regarding any error messages, version, and platform.
On Windows via pyarrow reading CSV file with a very, very, very long field (several million characters).
Since we don't know ahead of time length of the column - keep trying to read doubling buffer size.
Issue is that upon exception file seems to be not properly closed leaving an open handle and preventing temporary file(s) from being removed.
Python code:
while True:
try:
table = csv.read_csv(csv_file_path, read_options=read_options, parse_options=parse_options,
convert_options=convert_options)
break
except pa.lib.ArrowInvalid:
print(f'Doubling CSV block_size from {read_options.block_size}')
read_options.block_size *= 2
This issue is possibly related to #31796 since it seems to be about properly closing file handles on Windows.
Component(s)
C++, Python
The text was updated successfully, but these errors were encountered:
Describe the bug, including details regarding any error messages, version, and platform.
On Windows via pyarrow reading CSV file with a very, very, very long field (several million characters).
Since we don't know ahead of time length of the column - keep trying to read doubling buffer size.
Issue is that upon exception file seems to be not properly closed leaving an open handle and preventing temporary file(s) from being removed.
Python code:
This issue is possibly related to #31796 since it seems to be about properly closing file handles on Windows.
Component(s)
C++, Python
The text was updated successfully, but these errors were encountered: