bug: the exception of health check in special scene

[zuiyangqingzhou]

Current Behavior

When configuring similar clb nodes and IP nodes in upstream nodes at the same time, there will be node exception errors for clb nodes.

Expected Behavior

All nodes should be normal.

Error Logs

[error] 54#54: *338255 [lua] balancer.lua:82: fetch_health_nodes(): failed to get health check target status, addr: 39.156.66.14:80, host: nil, err: target not found, client: 172.18.0.1, server: _, request: "GET /hello HTTP/1.1", host: "127.0.0.1:9080"

Steps to Reproduce

1. upstream config just like

{
    "id":"459301800810382014",
    "create_time":1683294806,
    "update_time":1683295767,
    "nodes":{
        "172.18.0.4:80":1,
        "172.18.0.5:80":1,
        "www.baidu.com:80":1
    },
    "timeout":{
        "connect":6,
        "send":6,
        "read":6
    },
    "type":"roundrobin",
    "checks":{
        "active":{
            "concurrency":10,
            "healthy":{
                "http_statuses":[
                    200,
                    302,
                    404
                ],
                "interval":1,
                "successes":2
            },
            "http_path":"/",
            "timeout":1,
            "type":"http",
            "unhealthy":{
                "http_failures":5,
                "http_statuses":[
                    500,
                    501,
                    502,
                    503,
                    504,
                    505
                ],
                "interval":1,
                "tcp_failures":2,
                "timeouts":3
            }
        }
    },
    "scheme":"http",
    "pass_host":"pass",
    "name":"test",
    "keepalive_pool":{
        "idle_timeout":60,
        "requests":1000,
        "size":320
    }
}

or the configuration in the background looks like this

2. route config just like

{
    "id":"459301861040587454",
    "create_time":1683294842,
    "update_time":1683296659,
    "uri":"/*",
    "name":"test",
    "methods":[
        "GET",
        "POST",
        "PUT",
        "DELETE",
        "PATCH",
        "HEAD",
        "OPTIONS",
        "CONNECT",
        "TRACE",
        "PURGE"
    ],
    "upstream_id":"459301800810382014",
    "status":1
}

3. Initiate a request

 curl http://127.0.0.1:9080/hello -i

4. the following error occurs after a health check is triggered by request

 [error] 54#54: *338255 [lua] balancer.lua:82: fetch_health_nodes(): failed to get health check target status, addr: 39.156.66.14:80, host: nil, err: target not found, client: 172.18.0.1, server: _, request: "GET /hello HTTP/1.1", host: "127.0.0.1:9080"

Environment

• APISIX version (run apisix version): 3.2
• Operating system (run uname -a): linux
• OpenResty / Nginx version (run openresty -V or nginx -V): openresty/1.21.4.1
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 3.5
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[pioneer-hash]

Did you solve it? We have the same problem

[Sn0rt]

thx you report. I will move this issue to backlog task.

[ZhangShangyu]

Did you solve it? We have the same problem

+1

[jujiale]

we also suffered this problem

[monkeyDluffy6017]

@ZhangShangyu @jujiale This error should only occur when multiple requests are made concurrently while creating or modifying an upstream. When the first request is in the process of creating a health check object, the second request encounters this error because it cannot access the object that hasn't been created yet. However, this error does not actually affect request forwarding. In this pull request, we will lower the error log level from 'error' to 'warn' to prevent any misunderstanding.