[🐛 BUG]: Status is not available during graceful shutdown

[cv65kr]

No duplicates 🥲.

• I have searched for a similar issue in our bug tracker and didn't find any solutions.

What happened?

During the graceful shutdown status plugins goes unavailable and that leads to problem with kubernetes.

When probes are of for k8s when SIGQUIT is sent immediately and graceful shutdown is skipped

Version (rr --version)

rr version 2024.1.2

How to reproduce the issue?

Code is from grpc repository (test catalog)

Use configuration:

version: '3'

rpc:
  listen: "tcp://127.0.0.1:6111"

server:
  command: "php ../php_test_files/worker-grpc.php"
  relay: "pipes"
  relay_timeout: "20s"

status:
  address: "127.0.0.1:35544"

# GRPC service configuration
grpc:
  listen: "tcp://127.0.0.1:9111"
  proto:
    - "../proto/service/service.proto"
  max_send_msg_size: 50
  max_recv_msg_size: 50
  max_connection_idle: 0s
  max_connection_age: 0s
  max_connection_age_grace: 0s
  max_concurrent_streams: 10
  ping_time: 1s
  timeout: 200s
  pool:
    num_workers: 1
    allocate_timeout: 60s
    destroy_timeout: 5s

endure:
  grace_period: 30s

logs:
  output: stderr
  encoding: json
  level: info
  mode: production

class EchoService implements EchoInterface
{
    public function Ping(ContextInterface $ctx, Message $in): Message
    {
        sleep(30);
        $out = new Message();
        return $out->setMsg(strtoupper($in->getMsg()));
    }
}

1. Make a long request

grpcurl -plaintext -proto ./proto/service/service.proto -d '{}'  127.0.0.1:9111 service.Echo/Ping

2. Send SIGTERM

kill -TERM {{pid}}

3. http://127.0.0.1:35544/health?plugin=grpc and http://127.0.0.1:35544/ready?plugin=grpc returns 404

Relevant log output

No response

[rustatian]

Hey @cv65kr 👋

Hm.. Plugins are stopped asynchronously (all at the same time). That means, if the Status plugin (which is responsible for these urls in p.3) stopped first, they (urls) won't be available while other plugins, like gRPC are still in the shutdown process (because you have a graceful timeout).

This is done, because graceful shutdown should be spread across all plugin simultaneously, and should not be more than configured value.

Sorry, could you please rephrase this: When probes are of for k8s when SIGQUIT is sent immediately and graceful shutdown is skipped as well. I'm not sure, that I understand you correctly. Immediately after start? Or two signals?

[cv65kr]

@rustatian sorry, I was in a hurry 😄

When /read and /health endpoints are offline then k8s sent SIGQUIT immediately and graceful shutdown is skipped

[cv65kr]

Imo

/health should continue to return a positive response as long as the application is still capable of handling requests

/ready endpoint should start returning a not ready status (e.g., HTTP 503) to inform Kubernetes that the pod should not receive new requests

[rustatian]

Makes sense.
Ok, got u. Will fix that in the next bugfix release. Thanks for the report 👍