[mongo] Improve messaging&tags on MongoDB replset member state events #2409
Conversation
antifuchs
changed the title
| return 'Down' | ||
| elif state == 9: | ||
| return 'Rollback' | ||
| return self.REPLSET_MEMBER_STATES[state][1] |
There was a problem hiding this comment.
As for the get_state_name method, can we consider the case where state does not exist in REPLSET_MEMBER_STATES ?
if state in
pass
else:
return 'UNKNOWN'Even though, this should not happen (as opposed to get_state_name when last_state==-1 ), I think that would make the check less error prone to MongoDB updates.
There was a problem hiding this comment.
Agreed, that would be desirable. I'll fix this up.
(Actually, I'm going to pull out these methods to make them testable)
|
Thanks a lot @antifuchs. Your changes look great: I simply added a few nitpicks ✏️. We are planning a |
|
Would you mind rebasing your changes on the last |
|
@yannmh - the two fixes are in, and I rebased against master. Please let me know if the two changes are ok as-is or if you prefer them squashed into one. |
|
Squashed changes would be perfect 😄 |
This change summarized a lot of changes that improve how leadership election events look in the mongodb service check: * The event title nowrefers to the explicit hostname if checking localhost * The title now calls the member state by its official name and gives an the expanded state description in the body. * Events are only generated if there is a previous, known, state. This elimilates a lot of events being generated whenever the dd-agent is restarted. * Events now come with a set of tags featuring the previous and current states in a form that can unambiguously be queried, as well as the replSet in question.
|
Squashed! (: |
|
@antifuchs, I added a small commit on top of yours to fix the tests and restore a variable which got removed by accident in the rebase. |
This change summarized a lot of changes that improve how leadership election events look in the mongodb service check:
This change also eliminates the duplicate replica set state table that got introduced in a previous PR.
Testing
We're running this updated check in production at the moment. It generates events that we can easily overlay on top of graphs using queries like
tags:member_status:primary,previous_member_status:secondary(this finds re-elections where the primary is different from before).