Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flush jitter causes input collection to freeze #2262

Closed
lpic10 opened this issue Jan 12, 2017 · 4 comments · Fixed by #2281
Closed

Flush jitter causes input collection to freeze #2262

lpic10 opened this issue Jan 12, 2017 · 4 comments · Fixed by #2281
Labels
bug unexpected problem or unintended behavior
Milestone

Comments

@lpic10
Copy link
Contributor

lpic10 commented Jan 12, 2017

During my tests it looks like flush_jitter causes input plugins to freeze the collection.

To reproduce this quickly is rather easy:

  1. Set interval to a small value, eg. 10s
  2. Set flush_jitter to a bigger value than interval, eg. 20s
  3. Set a frequent flush_interval, eg. 1s (so chances are higher of getting flush_jitter bigger than interval to be able to see the errors)

I could reproduce this on version 1.1.x and in the master branch.

Actual behavior

Error messages like:
E! ERROR: input [inputs.cpu] took longer to collect than collection interval (10s)

And debug messages like:
D! Input [inputs.cpu] gathered metrics, (10s interval) in 33.689740435s

Expected behavior

Collection should work as usual, flush should be handled accordingly with the jitter delay

@sparrc
Copy link
Contributor

sparrc commented Jan 12, 2017

it's not freezing collection, it's just taking that long because your jitter is very high

why would you need to do this?

@lpic10
Copy link
Contributor Author

lpic10 commented Jan 12, 2017

I don't really need to do this, but why would flush_jitter cause the inputs also to wait?

@lpic10 lpic10 closed this as completed Jan 12, 2017
@lpic10 lpic10 reopened this Jan 12, 2017
@sparrc
Copy link
Contributor

sparrc commented Jan 12, 2017

oh, sorry I misread that, it shouldn't be happening that way

what do you have collection_jitter set to? can you provide a full config file so I can reproduce?

@lpic10
Copy link
Contributor Author

lpic10 commented Jan 12, 2017

I usually have collection_jitter set to 0.

I think any config file should work with the settings I described.
I just tested this one:

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "1s"
  flush_jitter = "40s"
  debug = true
  quiet = false

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "telegraf"
  write_consistency = "any"
  timeout = "5s"

[[inputs.cpu]]
  percpu = true
  totalcpu = true

[[inputs.netstat]]
[[inputs.mem]]
[[inputs.swap]]
[[inputs.net]]

@sparrc sparrc added bug unexpected problem or unintended behavior and removed Need More Info labels Jan 13, 2017
@sparrc sparrc added this to the 1.3.0 milestone Jan 13, 2017
sparrc added a commit that referenced this issue Jan 17, 2017
doing this unblocks incoming metrics while waiting for a flush to take
place.

we have to create a semaphore so that we can
'skip' flushes that try to run while a flush is already running.

closes #2262
sparrc added a commit that referenced this issue Jan 18, 2017
doing this unblocks incoming metrics while waiting for a flush to take
place.

we have to create a semaphore so that we can
'skip' flushes that try to run while a flush is already running.

closes #2262
sparrc added a commit that referenced this issue Jan 20, 2017
doing this unblocks incoming metrics while waiting for a flush to take
place.

we have to create a semaphore so that we can
'skip' flushes that try to run while a flush is already running.

closes #2262
sparrc added a commit that referenced this issue Jan 23, 2017
doing this unblocks incoming metrics while waiting for a flush to take
place.

we have to create a semaphore so that we can
'skip' flushes that try to run while a flush is already running.

closes #2262
njwhite pushed a commit to njwhite/telegraf that referenced this issue Jan 31, 2017
doing this unblocks incoming metrics while waiting for a flush to take
place.

we have to create a semaphore so that we can
'skip' flushes that try to run while a flush is already running.

closes influxdata#2262
mlindes pushed a commit to Comcast/telegraf that referenced this issue Feb 6, 2017
doing this unblocks incoming metrics while waiting for a flush to take
place.

we have to create a semaphore so that we can
'skip' flushes that try to run while a flush is already running.

closes influxdata#2262
maxunt pushed a commit that referenced this issue Jun 26, 2018
doing this unblocks incoming metrics while waiting for a flush to take
place.

we have to create a semaphore so that we can
'skip' flushes that try to run while a flush is already running.

closes #2262
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug unexpected problem or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants