Make the splitter use less memory

[rom1504]

If doing that then the memory usage can be capped to very low values

https://github.com/criteo/autofaiss/blob/master/autofaiss/readers/embeddings_iterators.py#L139 see this as example

[rom1504]

https://arrow.apache.org/docs/python/ipc.html + read parquet in batch

[rom1504]

read the files in batches, no need to prepare the whole first file of feather batches

[rom1504]

at least write all the shards in parallel so it's faster to write to high latency fs (s3, hdfs), eg by using https://filesystem-spec.readthedocs.io/en/latest/async.html

[rom1504]

done now