loadBackupSession hangs in non-sticky mode

[RDiazR2]

Hello

We think we found several scenarios in which backup session can't be found and the node in which is expected to be stored is down and if those conditions are matched, loadBackupSession at MemcachedSessionService.java never finish:

while(result == null && (nextNodeId = _memcachedNodesManager.getNextAvailableNodeId(nextNodeId)) != null && !nextNodeId.equals(nodeId)) { final String newSessionId = getSessionIdFormat().createNewSessionId(requestedSessionId, nextNodeId); result = loadBackupSession(newSessionId, newNodeId); }

If we don't understand this code wrong. If the requested session is i.e: E7B85F58A0BB5C8FAAA9BB90317B0E3F-node1 and the node 1 is down, this loop never finish. Am I wrong?

We thing this can happen in several scenarios, but all of them have one thing in common: node encoded in JSESSIONID is down. In our example: node1.

The scenarios we think can generate an incorrect behavior because of the previous identified loop are:

• If node1 and node2 are down. Two failures. It's not expected scenario but It could be possible.
• If session is removed from both memcached servers after expiration is reached or eviction and node1 is down
• Finally, we reproduce and scenario in which the session seems not to be stored successfully in memcached node1 because it's down but it's returned to the browser and when a request is done with that JSESSIONID, it falls in the loop.

The steps are the following:

• Session manager selected as new node because it's still available (isNodeAvailable):
  Creating new session id with orig id 'E7B85F58A0BB5C8FAAA9BB90317B0E3F' and memcached id 'node1'.
• It's not stored:
  Could not store session E7B85F58A0BB5C8FAAA9BB90317B0E3F-node1 in memcached.
• But it's returned to browser.
  Request finished, with Set-Cookie header: JSESSIONID=E7B85F58A0BB5C8FAAA9BB90317B0E3F-node1;
• When a request is received with that JSESSSIONID:

Request starting: GET /appserver/server.do?action=requestDo (requestedSessionId E7B85F58A0BB5C8FAAA9BB90317B0E3F-node1)
Session needs to be relocated as node node1 is not available, loading backup session for E7B85F58A0BB5C8FAAA9BB90317B0E3F-node1

• It's not found in any memcached node.

No validity info (or no valid one) found for sessionId E7B85F58A0BB5C8FAAA9BB90317B0E3F-node1
No validity info (or no valid one) found for sessionId E7B85F58A0BB5C8FAAA9BB90317B0E3F-node2
No validity info (or no valid one) found for sessionId E7B85F58A0BB5C8FAAA9BB90317B0E3F-node3
No validity info (or no valid one) found for sessionId E7B85F58A0BB5C8FAAA9BB90317B0E3F-node2
No validity info (or no valid one) found for sessionId E7B85F58A0BB5C8FAAA9BB90317B0E3F-node3
...

I attach log information of this behavior and enters in that loop. Please, check JSESSION E7B85F58A0BB5C8FAAA9BB90317B0E3F-node1 in that log file.

If you need any clarification or information, please let me know.

We tested with:

• memcached-session-manager 2.3.2 (at least 3 servers to reproduce the issue)
• spymemcached 2.12.0
• memcached 1.5.9
• tomcat 8.0.20

Thanks a lot
catalina_test1.zip