Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Buffer Improvements: write new S3 buffer implementation #9923

Open
tobz opened this issue Nov 4, 2021 · 0 comments
Open

Buffer Improvements: write new S3 buffer implementation #9923

tobz opened this issue Nov 4, 2021 · 0 comments
Labels
domain: buffers Anything related to Vector's memory/disk buffers type: task Generic non-code related tasks

Comments

@tobz
Copy link
Contributor

tobz commented Nov 4, 2021

Part of the Buffer Improvements RFC (RFC, #9476)

We need to write a new buffer implementation that can use S3 as the backing store.

The RFC itself covers a lot more about the goals, overall, with the notion of external buffers and why users want them. We'll cover the high-level items here:

  • batching will be required (this is an analogue to the disk buffer flush interval, but also we probably need to support size-based, because high event throughput could lead to memory usage spikes otherwise)
  • we have no ordering requirements going in or coming out
  • we will follow the S3 sink in terms of design: users will have to set up an SQS queue for S3 notifications in order to let the receiver side of the buffer do its thing

Things without concrete answers at the moment:

  • what does the buffer configuration look like? now we have to consider things like auth, batching, etc
@tobz tobz added type: task Generic non-code related tasks domain: buffers Anything related to Vector's memory/disk buffers labels Nov 4, 2021
@tobz tobz mentioned this issue Nov 4, 2021
18 tasks
@tobz tobz self-assigned this Dec 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: buffers Anything related to Vector's memory/disk buffers type: task Generic non-code related tasks
Projects
None yet
Development

No branches or pull requests

1 participant