Dedupe load_bytes_with calls

[stuhood]

In order to fix #11331, we'll need to dedupe load_bytes_with calls (similar to what #12087 did for store_bytes).

Since Store::load_bytes_with backfills to the local store, the deduping should likely involve a hashmap of async mutexes keyed by digest (or a cheaper equivalent) which prevent multiple callers from concurrently fetching a single digest, and re-checking the local store afterward.

[stuhood]

Having reviewed the code, I think that most of #12087 should be reusable, with some exceptions. The implementation for uploads:

• Is not cancellation safe: if an attempt to upload bytes is cancelled, there is no Drop-guard that will remove the attempt from the in_flight_uploads map.
• Doesn't allow concurrent attempts to wait. Because it short-circuits a duplicated attempt rather than causing the caller to "block" for the attempt to complete, a caller would need to poll the method until a condition was true.

---

To resolve this, I'd suggest:

1. refactor in_flight_uploads into a parking_lot::Mutex<HashMap<Digest, tokio::sync::OnceCell<()>>>. The actual upload attempt would be guarded under the OnceCell, which would allow a caller to wait for the upload attempt to have completed successfully, or otherwise execute it itself.
2. add a second in_flight_downloads map of the same shape, and use it in Store::load_bytes_with, with the OnceCell body wrapping downloading, verifying, and storing the content.
   • Rather than having the OnceCell hold the content in memory (i.e. to continue to use OnceCell<()>), add an infallible local.load_bytes_with call after local.store_bytes. This will have the effect of copying to local disk, and then loading from disk to actually run the user function.