Enhance /debug/health to be usable as a liveness probe #3139
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bbeaudreault
changed the title
|
I've tested this out and it works as expected in our environment. As I anticipated in my original description, there is a period of 5s or so where the state is transitioning and it returns an error. We could whitelist that state, but I'm just tuning my probes to be able to ride over it. |
sougou
reviewed
Sep 6, 2017
| tabletType := tsv.target.TabletType | ||
| tsv.mu.Unlock() | ||
| switch tabletType { | ||
| case topodatapb.TabletType_MASTER, topodatapb.TabletType_REPLICA, topodatapb.TabletType_BATCH: |
There was a problem hiding this comment.
To be future-proof, let's add experimental, which is considered a serving type.
There was a problem hiding this comment.
BATCH and RDONLY are the same value -- the switch complains if i add that. I'll do experimintal tho
There was a problem hiding this comment.
Of course :). Just experimental then.
|
added experimental. i think shard 1 is never going to go green, it's been failing all morning for multiple PRs |
|
LGTM |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Only report health for serving types, namely master, replica, batch. Fix api to return 500 if in error
This should be useful for those in kubernetes or who otherwise want to use a probe to determine whether the vttablet process is healthy. This is distinct from /healthz, which is used for load balancers and returns unhealthy in more cases.
Open question: should we also whitelist the
StateTransitioningstate? Maybe not, as this should probably be called with a heuristic that allows for a certain number of failures, like many probe implementations do.@sougou @adkhare