Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add terminationGracePeriod option, and use it when killing Solr. #226

Merged
merged 8 commits into from
Mar 3, 2021

Conversation

HoustonPutman
Copy link
Contributor

@HoustonPutman HoustonPutman commented Feb 25, 2021

Fixes #227

This will use the terminationGracePeriod, minus a few seconds, and give that to Solr as the SOLR_STOP_WAIT value.

That way Solr and Kubernetes are on the same page as to how much time to wait before forcefully stopping Solr.

terminationGracePeriod used to be statically set to 10 seconds. Now it defaults to 30 seconds, but is configurable.

@HoustonPutman HoustonPutman added this to the v0.3.0 milestone Feb 25, 2021
@HoustonPutman HoustonPutman requested a review from madrob March 1, 2021 16:42
@HoustonPutman HoustonPutman removed this from the v0.3.0 milestone Mar 1, 2021
...
customSolrKubeOptions:
podOptions:
terminationGracePeriodSeconds: 120
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this still means that administrators need to set this manually, rather than automatically picking up what is derived from the k8s source of truth?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a setting that users can set on pods, we merely forward this information to the pod. There is no k8s source of truth other than the value that each pod has. The default, set by the Solr Operator is 60 seconds.

Also I think the default value (for the pod) is different based on the kubernetes distribution you use, but we shouldn't be relying on a default that has nothing to do with Solr.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a user issues a command k delete --grace-period=5s than that will override our configured setting. We should figure out how to do our best to react to the new directives.

Copy link
Contributor Author

@HoustonPutman HoustonPutman Mar 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That command is kind of similar to kill -9. I'm not sure there is a way to protect against that. Is there a way to figure out what grace period is given when your pod is trying to be deleted? Pod.spec is static, so the Pod.spec.terminationGracePeriodSeconds, provided when the pod is created, will never change. If a user runs k delete --grace-period=5s, it will stay at 60 and not change to 5.

Most of the time, Pods are going to be deleted by Kubernetes (the statefulset controller) or the solrcloud controller, during managed upgrades. Neither of those use a different grace period, so the usage here is safe.

@madrob
Copy link
Contributor

madrob commented Mar 3, 2021

Please add a since 0.3.0 somewhere in the docs for this.

@HoustonPutman HoustonPutman merged commit 497630e into apache:main Mar 3, 2021
@HoustonPutman HoustonPutman deleted the solr-kill-wait branch March 3, 2021 20:23
jward-bw pushed a commit to BrandwatchLtd/solr-operator that referenced this pull request Apr 27, 2021
…apache#226)

terminationGracePeriod used to be statically set to 10 seconds. Now it defaults to 30 seconds, but is configurable.

SOLR_STOP_WAIT will be set to the terminationGracePeriod, minus a few seconds, so that kubernetes and Solr are on the same page.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Errors often occur while deleting Solr Pods
2 participants