Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

External buffer support (kafka, kinesis, etc) #5463

Closed
binarylogic opened this issue Dec 9, 2020 · 1 comment
Closed

External buffer support (kafka, kinesis, etc) #5463

binarylogic opened this issue Dec 9, 2020 · 1 comment
Labels
domain: buffers Anything related to Vector's memory/disk buffers meta: idea Anything in the idea phase. Needs further discussion and consensus before work can begin. needs: approval Needs review & approval before work can begin. needs: rfc Needs an RFC before work can begin. type: feature A value-adding code addition that introduce new functionality.

Comments

@binarylogic
Copy link
Contributor

binarylogic commented Dec 9, 2020 •
edited
Loading

Please see #3485 for additional details (this issue was opened unaware of that issue).

We currently offer memory and disk buffers, but we do not allow the use of external systems for buffers. It is interesting to think about configuring Kafka, Kinesis, or even Redis as an external buffer for Vector sinks. This would pair well with #5462.

This is different than using any of these systems as sources/sinks since Vector would transparently handle the partitioning and replaying of the data, shifting the management of these intricacies into Vector.
@binarylogic binarylogic added meta: idea Anything in the idea phase. Needs further discussion and consensus before work can begin. domain: buffers Anything related to Vector's memory/disk buffers needs: approval Needs review & approval before work can begin. needs: rfc Needs an RFC before work can begin. type: feature A value-adding code addition that introduce new functionality. labels Dec 9, 2020
@binarylogic binarylogic mentioned this issue Dec 9, 2020
Closed
This was referenced Jan 3, 2021
Closed
Closed
@binarylogic binarylogic mentioned this issue Oct 5, 2021
Closed
18 tasks
@jszwedko
Copy link
Member

Closing in-favor of #9923 which was is the first external buffer we think we'll target.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
domain: buffers Anything related to Vector's memory/disk buffers meta: idea Anything in the idea phase. Needs further discussion and consensus before work can begin. needs: approval Needs review & approval before work can begin. needs: rfc Needs an RFC before work can begin. type: feature A value-adding code addition that introduce new functionality.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jszwedko @lukesteensen @binarylogic