Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an aggregate metric for the theoretical write capacity #28

Merged
merged 1 commit into from
Mar 29, 2022

Conversation

mkeeler
Copy link
Member

@mkeeler mkeeler commented Jan 24, 2022

The new metric emits a sample of the number of logs per second boltdb could store if all those log write operations looked like the one currently being measured. That means that another operation would have the same number of logs per batch and that the actual txn Commit took the same amount of time. While no two operations will be identical, taking the average of the sample/summary emitted should provide a good picture of what Consul could handle with the current types of write operations being performed.

It is expected that this value will fluctuate with changes in size of the logs flowing through consul and how many logs get batched into one storage op.

If someone wanted to monitor this I think they would want to know when the actual write rate exceeds 75% of this metrics value. That could be due to an increased number of writes, or a degradation in disk performance which causes similar writes to slow down. Regardless of the cause, if you are getting close to the limit or see a drastic change in the metric it could be indicative of another issue which requires investigation.

Copy link

@markan markan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. I really appreciate it when software surfaces metrics like this. So often I look at a metric and have no idea how to relate it to reality.

A few minor tweaks, feel free to take or leave them.

bolt_store.go Outdated Show resolved Hide resolved
bolt_store.go Outdated Show resolved Hide resolved
go.mod Show resolved Hide resolved
bolt_store.go Show resolved Hide resolved
@acpana
Copy link

acpana commented Jan 31, 2022

[nit] Any chance we could add some comments around what the metric writeCapacity is and how folks should interact with it? (the PR description is great imo for this purpose)

@hashicorp-cla
Copy link

hashicorp-cla commented Mar 12, 2022

CLA assistant check
All committers have signed the CLA.

@mkeeler
Copy link
Member Author

mkeeler commented Mar 29, 2022

I added some info in the README about how to interpret metrics.

@mkeeler mkeeler merged commit 15018e9 into master Mar 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants