Add Burrow input plugin. #2485
Add Burrow input plugin. #2485
Conversation
Burrow(https://github.com/linkedin/Burrow/) is a monitoring companion for Apache Kafka that provides consumer lag and topic offset checking.
|
just wanted to check in and see if you have any updates on this guy? we've been watching it for a bit, would be helpful for our team to have this instead of using yet another tool to pull these in. |
|
@zeeto-slytherin I haven't had a chance to review yet. If possible it would be helpful if you, or anyone else in the community reviewed the plugin design, code, or tested out the code. There is an amd64 linux build linked in the circleci build. |
|
can you drop a link to where the linked artifact is? cannot seem to find it anywhere |
danielnelson
added
feat
|
this could be useful, we're parsing burrow with http now |
|
@burdandrei @zeeto-slytherin Hey! Since no progress on this PR, I made a rewrite of this plugin #3489. About 90% test coverage, more configuration options, stable (in production for a while). If you are still interested, maybe you will have a look? |
|
@arkady-emelyanov wow! i'll take a look! great job! |
There was a problem hiding this comment.
@codeb2cc Here is a high level review, I think there may be some issues with the output schema that we would need to discuss further but nothing major.
@arkady-emelyanov @codeb2cc We should first decide which burrow plugin to move forward with. Can you join forces on one of the pull requests to address the review comments?
| } | ||
|
|
||
| var wg sync.WaitGroup | ||
| errChan := errchan.New(len(b.Urls)) |
There was a problem hiding this comment.
Instead of using an error channel you can just AddError to the accumulator (this was probably added after you wrote this pull request).
| return errChan.Error() | ||
| } | ||
|
|
||
| var tr = &http.Transport{ |
There was a problem hiding this comment.
Create the transport at the same time you instantiate the client, this allows differing settings across multiple plugin instances.
| } | ||
|
|
||
| var tr = &http.Transport{ | ||
| ResponseHeaderTimeout: time.Duration(3 * time.Second), |
There was a problem hiding this comment.
http.Client.Timeout is sufficient, so you can remove this timeout
| } | ||
| acc.AddFields("burrow_consumer", startFields, tags, time.Unix(0, partition.Start.Timestamp*int64(time.Millisecond))) | ||
|
|
||
| endFields := map[string]interface{}{ |
There was a problem hiding this comment.
This seems problematic to me, since we are using the same tagset and fields as the start offset/lag, and there is potential for overwrite since we are storing the values at a timestamp based on the partition start/end.
| Error bool `json:"error"` | ||
| Message string `json:"message"` | ||
| Status ConsumerGroupStatus `json:"status"` | ||
| Request BurrowResponseRequestInfo `json:"request"` |
There was a problem hiding this comment.
Probably would make sense to have a type:
type BurrowResponse struct {
Error bool `json:"error"`
Message string `json:"message"`
Request BurrowResponseRequestInfo `json:"request"`
}
Then you should be able to embed it like:
type BurrowResponseConsumerStatus struct {
Status ConsumerGroupStatus `json:"status"`
BurrowResponse
}
| - topic | ||
| - partition | ||
| - status | ||
|
|
|
|
||
| - Measurement | ||
| - burrow_topic | ||
| - burrow_consumer |
| ## Topics to monitor. Default to monitor all. | ||
| #topics = [] | ||
| ## Groups to monitor. Default to monitor all. | ||
| #groups = [] |
There was a problem hiding this comment.
Need SSL/TLS support options
|
It's been more than 1 year... |
Burrow(https://github.com/linkedin/Burrow/) is a monitoring
companion for Apache Kafka that provides consumer lag and topic
offset checking.
Required for all PRs: