Implemented client-side per-session and per-statement configurable timeouts

[wprzytula]

Two levels of client-side timeouts were added: per Session and per Statement (PreparedStatement and Query). If no timeout is specified for a statement, a global per-session one is used. Once a timeout elapses, an RequestTimeout error is returned.

Fixes: #304

Pre-review checklist

• I have split my patch into logically separate commits.
• All commit messages clearly explain what they change and why.
• I added relevant tests for new features and bug fixes.
• All commits compile, pass static checks and pass test.
• PR description sums up the changes and reasons why they should be introduced.
• I added appropriate Fixes: annotations to PR description.

[wprzytula]

@psarna @cvybhu Maybe you could help me with understanding why test_client_timeout fails in nondeterministic way.

[cvybhu]

@psarna @cvybhu Maybe you could help me with understanding why test_client_timeout fails in nondeterministic way.

tokio::Timeout checks if the future has completed before checking if the timeout has elapsed.
With a local node and fast network it's possible that the query completes in a single Poll maybe?

https://github.com/tokio-rs/tokio/blob/de81985762a242c77361a6ab9de198372ca85987/tokio/src/time/timeout.rs#L175

[cvybhu]

Or maybe not, I added tokio::task::yield_now().await; at the beginning of runner block and it still fails.
So there are at least two Polls of the timeout future and it doesn't fail 0_o

[cvybhu]

Yeah something like this fails as well:

async fn yields_once() {
    tokio::task::yield_now().await;
}

async fn instant_timeout_test() {
    let res = tokio::time::timeout(std::time::Duration::from_secs(0), yields_once()).await;
    assert!(res.is_err());
}

Although sleeping for 1ms is enough to trigger timeout.

This might be an issue with tokio, but I'm not sure if it's a bug or a feature.
Tokio time keeps mentioning that its precision can be > 1ms, for example for sleep:

The implementation is platform specific, and some platforms (specifically Windows) will provide timers with a larger resolution than 1 ms.

https://docs.rs/tokio/latest/tokio/time/fn.sleep.html

[cvybhu]

Tokio time keeps mentioning that its precision can be > 1ms, for example for sleep:

I think we can just assume that tokio doesn't support timeouts of < 1ms and wait with the tests until proxy is merged.

[wprzytula]

Tokio time keeps mentioning that its precision can be > 1ms, for example for sleep:

I think we can just assume that tokio doesn't support timeouts of < 1ms and wait with the tests until proxy is merged.

That's right, with proxy we will easily delay the response for a couple of millis and set the timeout for 1 ms - this should work.

[psarna]

@wprzytula for now, let's just verify manually. If you create a 0ms or 1ms timeout, it should properly time out when sent to Scylla booted on a remote machine (e.g. one of the office servers).

[wprzytula]

v2:

• client_timeout -> request_timeout
• timeout test made working with 1ms timeout and remote Scylla; test marked as ignored (for normal testing with local Scylla).

[wprzytula]

@psarna This is probably complete.