cat uses excessive memory when using large skip number

[RamilNurtdinov]

Hi.
I am trying to split a big (5Gb) parquet file into small ones using "--skip" and "--limit"
Skip 5 million and write 6-th, skip 6 million and write 7-th.

However during skipping "parquet-tools cat" start consuming all my memory, reaching 128Gb it goes to swap and eventually killed.
Can you check the memory usage during "--skip"

Ramil
.

[hangxie]

yeah it seems an issue from parquet-go that use excessive memory with skip, I need some time to troubleshoot the issue.

FYI -limit is not an issue, it's problem from --skip

[hangxie]

Filed ticket to parquet-go xitongsys/parquet-go#545

[hangxie]

try v1.19.5, it has a hardcoded page size for SkipRows, I will come up with a cli parameter to make this configurable (#221)