fix incorrect usage of time.Timer

[tigrato]

Invoking timer.Stop on a timer that has already fired causes Stop to return false, and any subsequent drain operation will block because the channel remains open.

This PR addresses the blocking issue by wrapping the drain operation in a non-blocking select.

[zmb3]

Does this change still hold for Go 1.23, or do we need to leave a note to go back and update?

For a chan-based timer created with NewTimer(d), as of Go 1.23, any receive from t.C after Stop has returned is guaranteed to block rather than receive a stale time value from before the Stop; if the program has not received from t.C already and the timer is running, Stop is guaranteed to return true. Before Go 1.23, the only safe way to use Stop was insert an extra <-t.C if Stop returned false to drain a potential stale value. See the NewTimer documentation for more details.

[tigrato]

Does this change still hold for Go 1.23, or do we need to leave a note to go back and update?

For a chan-based timer created with NewTimer(d), as of Go 1.23, any receive from t.C after Stop has returned is guaranteed to block rather than receive a stale time value from before the Stop; if the program has not received from t.C already and the timer is running, Stop is guaranteed to return true. Before Go 1.23, the only safe way to use Stop was insert an extra <-t.C if Stop returned false to drain a potential stale value. See the NewTimer documentation for more details.

Unfortunately, Go 1.23 won't fix this situation directly check.
It fixes the deadlock because we no longer need to drain the channel every time we stop the timer, but for existing code, this problem would persist.

Although it doesn't fix the problem, Go 1.23 removed the misleading example where I got this incorrect behavior. check the discussion golang/go#27169

[public-teleport-github-review-bot]

@tigrato See the table below for backport results.

Branch	Result
branch/v15	Failed
branch/v16	Create PR

[public-teleport-github-review-bot]

@tigrato See the table below for backport results.

Branch	Result
branch/v15	Create PR
branch/v16	Failed