Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configure keystonemiddleware/oslo to deal with memcached pods failures #447

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

lmiccini
Copy link

@lmiccini lmiccini commented Dec 5, 2024

Whenever one of the mecached pods disappears, because of a rolling restart during a minor update or as result of a failure, APIs can take a long time to detect that the pod went away and keep trying to reconnect.

From a quick round of tests we saw downtimes up to ~150s.

By tuning memcache_pool_dead_retry and memcache_pool_conn_get_timeout the behavior seems much more acceptable.

Since neutron also uses memcached directly we also need to tweak the [cache] section enabling the retry mechanism in the client and apply similar defaults.

Jira: https://issues.redhat.com/browse/OSPRH-11935

Copy link
Contributor

openshift-ci bot commented Dec 12, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: lmiccini
Once this PR has been reviewed and has the lgtm label, please assign abays for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Whenever one of the mecached pods disappears, because of a rolling
restart during a minor update or as result of a failure, APIs can
take a long time to detect that the pod went away and keep trying
to reconnect.

From a quick round of tests we saw downtimes up to ~150s.

By tuning memcache_pool_dead_retry and memcache_pool_conn_get_timeout
the behavior seems much more acceptable.

Since neutron also uses memcached directly we also need to tweak
the [cache] section enabling the retry mechanism in the client and
apply similar defaults.

Jira: https://issues.redhat.com/browse/OSPRH-11935
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants