Increate timeouts of tests that frequently timeout in CI

[pentschev]

In the past few weeks some tests have timed out with certain frequency in CI, probably due to its load. Attempt to avoid those by increasing timeouts from 20 to 30 seconds.

[pentschev]

Sample of such timeouts can be seen in yesterday's nightly tests.

[pentschev]

More examples of timeouts from last night: https://github.com/rapidsai/dask-cuda/actions/runs/6008227979/attempts/1

[wence-]

Those timeouts sadly still persist in this PR. Do we need to bump them further?

[pentschev]

I'm setting up locally to see if I can reproduce them and how long the average runtime seems to be on our systems. I'm wondering if this is only the timeout or there's something else hidden under the timeout error.

[pentschev]

Or perhaps an easier and more accurate way to measure this is to (maybe temporarily) increase timeouts further and print durations of all tests in CI.

[pentschev]

Of course, now that the timeout has been increased there's nothing unusual, with test_communicating_proxy_objects taking < 10s every time. I think we may need to leave --durations=0 for a few days and see it during the nightly tests, it looks to me like nightly tests fail because that's when we have the highest utilization of CI nodes when nightly tests are running for all projects, and thus things get slower.

What do you think we should do here @wence- ?

[wence-]

That sounds like a reasonable approach, yes.

[pentschev]

That sounds like a reasonable approach, yes.

Alright then, should we merge it as is then? I'll keep an eye for tonight's run and see what I can find out.

[pentschev]

/merge

[pentschev]

/merge

[pentschev]

@wence- was trying to get this merged today but it seems today's not the day to merge it. 😞