Measure GIL contention in benchmarks

[pentschev]

Add argument to measure and report GIL contention in benchmarks. This may provide some useful insight when optimizing asyncio-related changes.

Sample results observed are below.

Small messages (1 B)

$ python -m ucp.benchmarks.send_recv --backend ucp-core --n-bytes 1 --n-iter 100_000 --no-detailed-report --report-gil-contention
...
GIL contention            | 0.688429594039917

$ python -m ucp.benchmarks.send_recv --backend ucp-async --n-bytes 1 --n-iter 100_000 --no-detailed-report --report-gil-contention
...
GIL contention            | 0.4772491455078125

$ UCXPY_NON_BLOCKING_MODE=1 python -m ucp.benchmarks.send_recv --backend ucp-async --n-bytes 1 --n-iter 100_000 --no-detailed-report --report-gil-contention
...
GIL contention            | 0.8763511180877686

Medium messages (1 MiB)

$ python -m ucp.benchmarks.send_recv --backend ucp-core --n-bytes 1MiB --n-iter 100 --no-detailed-report --report-gil-contention
...
GIL contention            | 0.1721574366092682

$ python -m ucp.benchmarks.send_recv --backend ucp-async --n-bytes 1MiB --n-iter 100 --no-detailed-report --report-gil-contention
...
GIL contention            | 0.25215959548950195

$ UCXPY_NON_BLOCKING_MODE=1 python -m ucp.benchmarks.send_recv --backend ucp-async --n-bytes 1MiB --n-iter 100 --no-detailed-report --report-gil-contention
...
GIL contention            | 0.2693021595478058

Large messages (1 GiB)

$ python -m ucp.benchmarks.send_recv --backend ucp-core --n-bytes 1 GiB --n-iter 10 --no-detailed-report --report-gil-contention
...
GIL contention            | 0.5908554792404175

$ python -m ucp.benchmarks.send_recv --backend ucp-async --n-bytes 1GiB --n-iter 10 --no-detailed-report --report-gil-contention
...
GIL contention            | 0.6251853108406067

$ UCXPY_NON_BLOCKING_MODE=1 python -m ucp.benchmarks.send_recv --backend ucp-async --n-bytes 1GiB --n-iter 10 --no-detailed-report --report-gil-contention
...
GIL contention            | 0.832139790058136

[quasiben]

This is really cool! Can you describe what the output results are ? Is it time, a ratio, etc ?

[pentschev]

It is a ratio of the perceived time the user code (in this case, UCX-Py send/recv messages) had the GIL. Thus 0.0 would mean the user code never took the GIL, whereas 1.0 would mean the user code had the GIL for the entirety of time without releasing it.

[jacobtomlinson]

This looks neat! The Dask dashboard has a plot for this too, it's under the "more" list and is called "Contention". There were some performance implications with gilknocker==0.3.0 but seems much better in 0.4.0.

[pentschev]

This looks neat! The Dask dashboard has a plot for this too, it's under the "more" list and is called "Contention". There were some performance implications with gilknocker==0.3.0 but seems much better in 0.4.0.

Yes, Ben pointed me to that, and that's where I stole the idea from.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[pentschev]

/ok to test

[wence-]

Minor nits, looks good though.

Interested to know how contended things are.

[pentschev]

Interested to know how contended things are.

Not sure if you mean something different, but I reported contention numbers in the description. Did you overlook that or are you asking for something else?

[pentschev]

/ok to test

[wence-]

Interested to know how contended things are.

Not sure if you mean something different, but I reported contention numbers in the description. Did you overlook that or are you asking for something else?

Oh no, I'm just blind...

[pentschev]

Thanks @wence- !

[pentschev]

/merge