Skip to content

Commit

Permalink
feat(arrow/ipc): implement lazy loading/zero-copy for IPC files (#216)
Browse files Browse the repository at this point in the history
### Rationale for this change
closes #207 

### What changes are included in this PR?
Adding new method `NewMappedFileReader` to ipc package which accepts a
byte slice instead of a `ReaderAtSeeker`. Updates `ipcSource` to
reference the raw byte slices from the input directly instead of
wrapping with `bytes.NewReader` which forces copies via `Read`,
`ReadFull`, etc.

### Are these changes tested?
Unit tests added to confirm that the pointers match and that we aren't
allocating unnecessarily.

### Are there any user-facing changes?
Shouldn't be any user-facing changes other than a reduction in memory
usage when reading non-compressed IPC data.
  • Loading branch information
zeroshade authored Dec 18, 2024
1 parent 1ad41e1 commit 95c4e4d
Show file tree
Hide file tree
Showing 6 changed files with 369 additions and 160 deletions.
Loading

0 comments on commit 95c4e4d

Please sign in to comment.