Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Remote Store] Support to emit multiple streams for a file content each responsible for processing a specific part of the file #6980

Closed
Tracked by #6703
raghuvanshraj opened this issue Apr 4, 2023 · 0 comments
Assignees
Labels
enhancement Enhancement or improvement to existing feature or request

Comments

@raghuvanshraj
Copy link
Contributor

Reference: #6703

  • Offset based InputStream extensions to emit stream from a specific part of a file, which starts reading from a specific position and ensures that maximum length of content read doesn't exceed a specified limit. OffsetRangeFileInputStream achieves this for File objects, while OffsetRangeIndexInputStream achieves this for lucene's IndexInput construct
  • RemoteTransferContainer has utilities to open streams to specific parts of the file based on the type as mentioned in the previous point. It also manages post upload tasks by implementing an UploadFinalizer.
  • ResettableCheckedInputStream allows for individual parts to be reset through mark and reset in the event of upload failures
@raghuvanshraj raghuvanshraj added enhancement Enhancement or improvement to existing feature or request untriaged labels Apr 4, 2023
@raghuvanshraj raghuvanshraj changed the title Support to emit multiple streams for a file content each responsible for processing a specific part of the file [Remote Store] Support to emit multiple streams for a file content each responsible for processing a specific part of the file Apr 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancement or improvement to existing feature or request
Projects
None yet
Development

No branches or pull requests

2 participants