bug: Use upstream keepalive, resulting in long connection reuse problem

[hahayyum]

Current Behavior

example 1:
route:/aaa upstream: test.com, mtls: use client.key and client.cert
route:/bbb upstream; test.com does not use mtls
Result 1:
When accessing /bbb, sometimes the connection of /aaa is reused, causing the request to carry client.key and client.cert .
But sometimes client.key and client.cert are lost when accessing /aaa

or
example 2:
route: https://testA.com/aaa upstream: testA.test.com, mtls: use client.key and client.cert
route: https://testB.com/aaa upstream; testB.test.com does not use mtls
os: /etc/hosts
192.168.35.3 testA.test.com testB.test.com
Result 2:
When accessing https://testB.com/aaa, sometimes the connection of testB.com/aaa is reused, causing the request to carry client.key and client.cert .
But sometimes client.key and client.cert are lost when accessing https://testA.com/aaa

Expected Behavior

expect:
When keepalive multiplexes the connection, use the connection of client.cert and client.key to multiplex this type of connection, and the connection that does not use client authentication is reused for the same type of connection。

After investigation, it is the problem caused by upstream keepalive, which can be solved by using a short connection

What is the reuse of upstream.keepalive long connections in APISIX-OpenResty? only host IP?

Error Logs

No response

Steps to Reproduce

1、create Route /aaa
curl http://localhost:9080/apisix/admin/routes/395051487660606160?api_key=admin -X PUT -i -d '
{
"uri": "/aaa",
"name": "aaa",
"methods": [
"GET",
"POST",
"PUT",
"DELETE",
"PATCH",
"HEAD",
"OPTIONS",
"CONNECT",
"TRACE"
],
"upstream": {
"nodes": [
{
"host": "test.com",
"port": 8090,
"weight": 100
}
],
"timeout": {
"connect": 6,
"send": 6,
"read": 6
},
"type": "roundrobin",
"scheme": "https",
"pass_host": "pass",
"tls": {
"client_cert": "clent_cert",
"client_key": "client_key"
},
"keepalive_pool": {
"idle_timeout": 60,
"requests": 1000,
"size": 320
}
},
"status": 1
}'
2、create Route /bbb
curl http://localhost:9080/apisix/admin/routes/39505148743344535345?api_key=admin -X PUT -i -d '
{
"uri": "/bbb",
"name": "bbb",
"methods": [
"GET",
"POST",
"PUT",
"DELETE",
"PATCH",
"HEAD",
"OPTIONS",
"CONNECT",
"TRACE"
],
"upstream": {
"nodes": [
{
"host": "test.com",
"port": 8090,
"weight": 100
}
],
"timeout": {
"connect": 6,
"send": 6,
"read": 6
},
"type": "roundrobin",
"scheme": "https",
"pass_host": "pass",
"keepalive_pool": {
"idle_timeout": 60,
"requests": 1000,
"size": 320
}
},
"status": 1
}'
3、create upstream server
1）Supports both mtls and no mtls, such as k8s apiserver
2）Different hosts, corresponding to the same service

Environment

environment:
apisix 2.7
use APISIX-OpenResty

[spacewander]

What is the reuse of upstream.keepalive long connections in APISIX-OpenResty? only host IP?

Yes.

BTW, apisix 2.7 doesn't support upstream's keepalive. Although the latest version is still using host IP as keepalive connection key.

[hahayyum]

@spacewander
So, with the same host IP, when using upstream's keepalive, is there a problem with different requests using different client.cert access?
E.g:
Request A to use the testa client certificate
Request B to use the testb client certificate
When multiple requests are made, request A may use the connection with the testb certificate, and vice versa