Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support commpression and partitioner parameters for kafka logger #481

Open
uralmetal opened this issue Nov 23, 2023 · 5 comments
Open

Support commpression and partitioner parameters for kafka logger #481

uralmetal opened this issue Nov 23, 2023 · 5 comments
Labels
enhancement New feature or request waiting feedback
Milestone

Comments

@uralmetal
Copy link

uralmetal commented Nov 23, 2023

Is your feature request related to a problem? Please describe.
I have a problem with writing to kafka:

  1. I want to send to all partitions, because traffic is read evenly by all consumers
  2. I want to compress traffic, because it to store longer in kafka partition

Describe the solution you'd like

  1. For first case i suggest following solution:
  • If partition parameter is not null, then behavior remains as is
  • If partition parameter is null, then go-dnscollector use round-robin partitioner for kafka. Like vector

Example:

partition: 0 # go-dnscollector write to partition 0
partition: null # go-dnscollector write to all partition
  1. For second case i suggest to add compression field like vector or logstash

For solution any problem with kafka parameters - field kafka_parameters like vector. It's not required condition for this issue :)

@dmachard dmachard added the enhancement New feature or request label Nov 24, 2023
@dmachard dmachard added this to the v0.39.0 milestone Nov 24, 2023
@dmachard
Copy link
Owner

compression support will be available for next release

@dmachard
Copy link
Owner

The release v0.39.0 is now available with compression support for kafka logger.

@dmachard dmachard removed this from the v0.39.0 milestone Dec 22, 2023
@uralmetal
Copy link
Author

Nice! Tell me, when will the task with the partitioner be taken on?

@dmachard dmachard added the help wanted Extra attention is needed label Jun 16, 2024
@dmachard dmachard added this to the v1.0.0 milestone Jun 30, 2024
@dmachard
Copy link
Owner

Writes in round-robin partitioner is planned for the first next stable release.

@dmachard dmachard added waiting feedback and removed help wanted Extra attention is needed labels Jul 1, 2024
@uralmetal
Copy link
Author

@dmachard hi! We were tested round robin with other team. You see report in another issue #800

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request waiting feedback
Projects
None yet
Development

No branches or pull requests

2 participants