Skip to content
This repository has been archived by the owner on Oct 23, 2024. It is now read-only.

Implement Readiness Probe #6

Merged
merged 10 commits into from
Oct 29, 2019
Merged

Implement Readiness Probe #6

merged 10 commits into from
Oct 29, 2019

Conversation

phrynchanka
Copy link
Contributor

@phrynchanka phrynchanka commented Oct 23, 2019

What changes are proposed in this PR?

Resolves DCOS-58972

This PR Implement Readiness Probe for Kudo Cassandra operator.
LivenessProbe wasn't implemented because we need clear understand when pods definitely must be restarted.

Implementation steps :

  1. Added k8s ReadinessProbe properties to params.yml
  2. Modified stateful-set.yaml
  3. Added ReadinessProbe script similar like in DC/OS but without security support (which should be implemented and script modified)

How were these changes tested?

Manually tested

  1. Install operator
  2. Look into pod events
    kubectl get event --namespace $kudo_cassandra_instance_namespace --field-selector involvedObject.name=cassandra-node-0
    and
    kubectl describe pod cassandra-node-0
  3. Check that all pods were run and healthy.

@phrynchanka phrynchanka added the question Further information is requested label Oct 23, 2019
@phrynchanka phrynchanka changed the title [DCOS-58972] Implement Health-Checks, Readiness and Liveness Probes Implement Readiness Probe Oct 24, 2019
@phrynchanka phrynchanka added ready for review and removed question Further information is requested labels Oct 25, 2019
Copy link

@samvantran samvantran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work @phrynchanka ! Left a few comments, mostly around consistency but overall things look good!

operator/params.yaml Outdated Show resolved Hide resolved
operator/params.yaml Outdated Show resolved Hide resolved
operator/params.yaml Outdated Show resolved Hide resolved
operator/params.yaml Outdated Show resolved Hide resolved
operator/params.yaml Outdated Show resolved Hide resolved
operator/params.yaml Outdated Show resolved Hide resolved
initialDelaySeconds: {{ .Params.READINESS_CHECK_INITIAL_DELAY }}
periodSeconds: {{ .Params.READINESS_CHECK_PERIOD }}
timeoutSeconds: {{ .Params.READINESS_CHECK_TIMEOUT }}
successThreshold: {{ .Params.READINESS_SUCCESS_THRESHOLD }}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this param is even needed to be configurable since we only need 1 successful check. But in the spirit of configuring :all-the-things:...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've exposed all properties which k8s offer for readiness/liveness probe. Maybe it will be used :)

operator/operator.yaml Outdated Show resolved Hide resolved
Copy link
Contributor

@mpereira mpereira left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @phrynchanka! The changes look good to me. I pushed a PR against this PR with a few ideas for improvements. Please take a look: #8

I'll try to get the PR with the testing structure merged asap so that we can update this PR's branch with it and then we can merge it with the green checks.

Also, I noticed that a slightly different pattern is used elsewhere for grepping the nodetool output. Did it also work with the pattern we're using here?

@phrynchanka
Copy link
Contributor Author

Also, I noticed that a slightly different pattern is used elsewhere for grepping the nodetool output. Did it also work with the pattern we're using here?

Yep, our pattern works great.

phrynchanka.c and others added 2 commits October 28, 2019 11:31
* Move "readiness probe"-related params to the operator settings block.

Also, prepend "NODE" since they're for the "node" container. It's likely that
we'll add readiness probes for other containers.

* s/cassandra-readiness-probe/node-readiness-probe/

To be consistent with per-container naming.

* Simplify phrasing.

* Be consistent with adding unit suffixes to param names.
Copy link
Contributor

@mpereira mpereira left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking a look at the changes!

Let's wait for the tests to be merged to master and then we'll get this one merged as well.

@phrynchanka
Copy link
Contributor Author

Let's wait for the tests to be merged to master and then we'll get this one merged as well.
Thank you!
Do we need to implement integration test?

@mpereira
Copy link
Contributor

I think for now the existing test should be sufficient, since a successful install is dependent on the readiness probes. We might want to add more specific tests for it in the near future, but for now it should be ok.

Copy link

@samvantran samvantran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @phrynchanka !

operator/params.yaml Show resolved Hide resolved
Copy link
Contributor

@viivek46 viivek46 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @phrynchanka LGTM.

@mpereira
Copy link
Contributor

🚢

@phrynchanka phrynchanka merged commit f14d730 into master Oct 29, 2019
@phrynchanka phrynchanka deleted the DCOS-58972 branch October 29, 2019 14:51
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants