Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lock not released on error; add error logging #137

Closed
chris-branch opened this issue Feb 18, 2020 · 0 comments · Fixed by #140
Closed

Lock not released on error; add error logging #137

chris-branch opened this issue Feb 18, 2020 · 0 comments · Fixed by #140

Comments

@chris-branch
Copy link

I'm trying to track down the source of an error message that appears periodically in the logs for my Kong cluster (backed by Cassandra). The message looks like this:

connector.lua:269: [cassandra] failed to refresh cluster topology: timeout, context: ngx.timer

I believe this is coming from here:

https://github.com/Kong/kong/blob/master/kong/db/strategies/cassandra/connector.lua#L273

which ends up calling _Cluster:refresh() in lua-cassandra. While reviewing the code, I noticed that that _Cluster:refresh() obtains a shared lock, but the function has several exit points that do not release the lock. In particular, there are 6 places where an error return can occur where lock:unlock() will not be called. Example:

https://github.com/thibaultcha/lua-cassandra/blob/master/lib/resty/cassandra/cluster.lua#L583

The lock will auto-release after the 60-second timeout, but any callers may block until that happens.

Also, it would be helpful if _Cluster:refresh() could log a warning/error message for each of the early returns to help diagnose the root cause of any failures.

bungle added a commit to bungle/lua-cassandra that referenced this issue Oct 6, 2020
### Summary

On thibaultcha#137 @chris-branch reported that `cluster:refresh` did not always release a lock
on error cases. This commit fixes that.

### Issues Resolved

Fix thibaultcha#137
bungle added a commit to bungle/lua-cassandra that referenced this issue Oct 6, 2020
### Summary

On thibaultcha#137 @chris-branch reported that `cluster:refresh` did not always release a lock
on error cases. This commit fixes that.

### Issues Resolved

Fix thibaultcha#137
thibaultcha pushed a commit that referenced this issue Dec 18, 2020
### Summary

On #137 @chris-branch reported that `cluster:refresh` did not always release a lock
on error cases. This commit fixes that.

### Issues Resolved

Fix #137
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant