-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Setting Prometheus Remote_write Period default value to 1m #38553
Conversation
This pull request does not have a backport label.
To fixup this pull request, you need to add the backport labels for the needed
|
💚 Build Succeeded
History
cc @gizas |
💚 Build Succeeded
History
cc @gizas |
💚 Build Succeeded
History
cc @gizas |
💚 Build Succeeded
History
cc @gizas |
💚 Build Succeeded
History
cc @gizas |
💚 Build Succeeded
History
cc @gizas |
💚 Build Succeeded
History
cc @gizas |
💚 Build Succeeded
cc @gizas |
💚 Build Succeeded
cc @gizas |
💚 Build Succeeded
cc @gizas |
Co-authored-by: Tetiana Kravchenko <[email protected]>
Co-authored-by: Tetiana Kravchenko <[email protected]>
Co-authored-by: Tetiana Kravchenko <[email protected]>
if duration < 60*time.Second || err != nil { | ||
duration = time.Second * 60 | ||
} | ||
logp.Debug("Period for counter cache for remote_write", duration.String()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
now the log is:
{"log.level":"debug","@timestamp":"2024-03-28T14:49:07.238Z","log.logger":"Period for counter cache for remote_write","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/metricbeat/module/prometheus/remote_write.remoteWriteEventsGeneratorFactory","file.name":"remote_write/data.go","file.line":52},"message":"1m0s","service.name":"metricbeat","ecs.version":"1.6.0"}
logp.Debug("Period for counter cache for remote_write", duration.String()) | |
logp.Debug("prometheus.remote_write.cache", "Period for counter cache for remote_write: %v", duration.String()) |
I used the same logger as in https://github.com/elastic/beats/blob/main/x-pack/metricbeat/module/prometheus/remote_write/data.go#L89
this code is executed twice, we need to check if it is expected
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed the logger !
For the number of calls now: I see that our function is called here in the init.
I dont see a second place for call. So seems that init() is called twice? Still looking
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I found it:
First call happens here and second call happens here . So we read config initially and then after reload period we check again.
So it is something that happens only on config load time.
Logs to prove above:
{"log.level":"info","@timestamp":"2024-03-29T09:11:56.648Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch","file.name":"instance/beat.go","file.line":520},"message":"metricbeat start running.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-29T09:11:56.648Z","log.logger":"cfgfile","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Check","file.name":"cfgfile/reload.go","file.line":131},"message":"Checking module configs from: /usr/share/metricbeat/modules.d/*.yml","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-29T09:11:56.649Z","log.logger":"cfgfile","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.LoadList","file.name":"cfgfile/cfgfile.go","file.line":204},"message":"Load config from file: /usr/share/metricbeat/modules.d/prometheus.yml","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-29T09:11:56.649Z","log.logger":"cfgfile","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Check","file.name":"cfgfile/reload.go","file.line":145},"message":"Number of module configs found: 1","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-29T09:11:56.649Z","log.logger":"prometheus.remote_write.cache","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/metricbeat/module/prometheus/remote_write.remoteWriteEventsGeneratorFactory","file.name":"remote_write/data.go","file.line":47},"message":"Period for counter cache for remote_write: 3m0s","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-29T09:11:56.649Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Run","file.name":"cfgfile/reload.go","file.line":163},"message":"Config reloader started","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-29T09:11:56.649Z","log.logger":"cfgfile","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Run","file.name":"cfgfile/reload.go","file.line":193},"message":"Scan for new config files","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-29T09:11:56.649Z","log.logger":"cfgfile","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.LoadList","file.name":"cfgfile/cfgfile.go","file.line":204},"message":"Load config from file: /usr/share/metricbeat/modules.d/prometheus.yml","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-29T09:11:56.650Z","log.logger":"cfgfile","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Run","file.name":"cfgfile/reload.go","file.line":212},"message":"Number of module configs found: 1","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-29T09:11:56.650Z","log.logger":"reload","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Reload","file.name":"cfgfile/list.go","file.line":92},"message":"Starting reload procedure, current runners: 0","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-29T09:11:56.650Z","log.logger":"reload","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Reload","file.name":"cfgfile/list.go","file.line":110},"message":"Start list: 1, Stop list: 0","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-29T09:11:56.650Z","log.logger":"prometheus.remote_write.cache","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/metricbeat/module/prometheus/remote_write.remoteWriteEventsGeneratorFactory","file.name":"remote_write/data.go","file.line":47},"message":"Period for counter cache for remote_write: 3m0s","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-29T09:11:56.650Z","log.logger":"reload","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Reload","file.name":"cfgfile/list.go","file.line":154},"message":"Starting runner: RunnerGroup{prometheus [metricsets=1]}","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-03-29T09:11:56.650Z","log.logger":"module","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start","file.name":"module/wrapper.go","file.line":129},"message":"Starting Wrapper[name=prometheus, len(metricSetWrappers)=1]","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-29T09:11:56.650Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Run","file.name":"cfgfile/reload.go","file.line":223},"message":"Loading of config files completed.","service.name":"metricbeat","ecs.version":"1.6.0"}
(I will leave this comment as unresolved just for reference)
@@ -21,6 +25,7 @@ var defaultConfig = config{ | |||
TypesPatterns: TypesPatterns{ | |||
CounterPatterns: nil, | |||
HistogramPatterns: nil}, | |||
Period: time.Second * 60, | |||
} | |||
|
|||
func (c *config) Validate() error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could this function be a better fit for the period
validation? check can be moved here with the info log message regarding the min value
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I moved this to the Validate() 4cfcf01
return err | ||
} else if duration < 60*time.Second { | ||
// by default prometheus push data with the interval 60s, in order to calculate counter rate we are setting Period to 60secs accordingly | ||
c.Period = time.Second * 60 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we log a warning saying the period needs to be at least 60s?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@rdner , @fearful-symmetry can you please have a review here please? |
@gizas looks like you still need to run |
…remote_write_ratefix
* Correcting period for Prometheus remote_write * Update CHANGELOG.next.asciidoc (cherry picked from commit e798bb1)
…38811) * Correcting period for Prometheus remote_write * Update CHANGELOG.next.asciidoc (cherry picked from commit e798bb1) Co-authored-by: Andrew Gizas <[email protected]>
Proposed commit message
Checklist
CHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.How to test this PR locally
beats/x-pack/metricbeat
PLATFORMS=linux/arm64 PACKAGES=docker mage package
kind load docker-image docker.elastic.co/beats/metricbeat:8.14.0 --name kind-cluster
metricbeat manifest for prometheus
Note: In case of dropping metrics you might need to increase mapping limit from inside Dev Console:
PUT .ds-metricbeat-8.14.0-2024.03.28-000001/_settings { "index.mapping.total_fields.limit": 99999 }
Related issues
Screenshots
Logs
With no period set: (default applies that is period:60s)
With
period: 108s
With
period: 35m