Inconsistency in Send/Sync requirements for async/await

[Ekleog]

The following code, when switching between lines 14 and 15 (the two versions of the for line), oscillates between compiling and not compiling because it requires Sync for the dyn Send items:

#![feature(async_await, await_macro, futures_api)]

use std::collections::VecDeque;
use std::future::Future;

pub async fn receive<HandleFn, Fut, Ret>(handle: HandleFn) -> Ret
where
    Fut: Future<Output = ()>,
    HandleFn: Fn() -> Fut,
{
    let v: VecDeque<Box<dyn Send>> = VecDeque::new();

    let l = v.len();
    for _i in 0..v.len() {
    //for _i in 0..l {
        await!(handle());
    }
    
    loop {}
}

fn do_stuff<F: Future + Send>(_f: F) {}

pub fn showcase() {
    do_stuff(async {
        match await!(receive(async move || ())) {
            true => "test",
            false => "test",
        };
    });
}

(playground)

(Note: this is a simplification of the failure that occurs on Ekleog/erlust@98c6cbc when running cargo test)

[cramertj]

I think this is working as intended-- I'd expect that in order for the future to be Send, all references held across await! points must be Send, which means the data they reference must be Sync. In the for loop here, a reference to v is being held across the await!. cc @nikomatsakis for another unfortunate side-effect of temporary references becoming extraordinarily visible under async/await!.

[cramertj]

I guess it's also worth pointing out that what isn't "working as intended" here is that the reference in question shouldn't need to be held across the await!, since it's only being used to produce a new temporary (the range).

[Ekleog]

Actually, if the reference points to inside the future itself, I think that the pointed-to stuff might only need to be Send, as there would at any point in time be only a single thread referring to them. But that is another issue (or another feature request, actually -- and I'm not even sure yet it'd be sound), and here what I wanted to point out was that the temporary borrow is propagated further than would be expected.

[earthengine]

FYI, I was posted in internals for a similar issue regarding pattern matches.

Playground

#![feature(
    await_macro,
    async_await,
    futures_api,
    optin_builtin_traits
)]
use std::future::Future;

struct Foo;
impl Foo {
    fn foo(&self) -> Option<()> { Some(()) }
}
impl !Sync for Foo{}

async fn bar() {
    let f = Foo;
    if let Some(v) = f.foo() {
        await!(async{})
    }
}

async fn buz(f: impl Future<Output=()> + Send) {
    await!(f)
}

fn main(){
    buz(bar());
}

[nikomatsakis]

I believe this is a duplicate of #57017.

[nikomatsakis]

I've marked this as deferred because of how #57017 is triaged, but I tend to think we should take some action here -- at least trying to get more precise.

question: If others agree this is a dup, maybe we can close this issue? (Perhaps move the examples over to #57017)

[Ekleog]

I'm not sure this is the same as #57017, but it does look similar -- will let someone more knowledgeable than me about compiler internals choose whether to close this :)

[Nemo157]

It seems like #57017 is more specifically about a code pattern that leads to this happening, the underlying issue here is just that a generator holding a self-reference across a yield point requires the referenced object to be Sync when it shouldn't, it just needs to be Send as the reference will move threads along with the referenced object. #57017 shows one possible way to cause this issue, because computing the live variables across the yield point inside a match includes the matched expression when it is unnecessary.

[Nemo157]

Minimised example (playground):

#![feature(async_await)]

async fn foo(x: Box<dyn Send>) {
    async fn bar() {}
    let x = &x;
    bar().await;
}

fn assert_send<T: Send>(_: T) {}

fn main() {
    assert_send(foo(Box::new(5)));
}

error[E0277]: `dyn std::marker::Send` cannot be shared between threads safely
  --> src/main.rs:12:5
   |
12 |     assert_send(foo(Box::new(5)));
   |     ^^^^^^^^^^^ `dyn std::marker::Send` cannot be shared between threads safely
   |
   = help: the trait `std::marker::Sync` is not implemented for `dyn std::marker::Send`
   = note: required because of the requirements on the impl of `std::marker::Sync` for `std::ptr::Unique<dyn std::marker::Send>`
   = note: required because it appears within the type `std::boxed::Box<dyn std::marker::Send>`
   = note: required because of the requirements on the impl of `std::marker::Send` for `&std::boxed::Box<dyn std::marker::Send>`
   = note: required because it appears within the type `for<'r, 's, 't0> {std::boxed::Box<(dyn std::marker::Send + 'r)>, &'s std::boxed::Box<(dyn std::marker::Send + 't0)>, impl std::future::Future, ()}`
   = note: required because it appears within the type `[static generator@src/main.rs:3:32: 7:2 x:std::boxed::Box<(dyn std::marker::Send + 'static)> for<'r, 's, 't0> {std::boxed::Box<(dyn std::marker::Send + 'r)>, &'s std::boxed::Box<(dyn std::marker::Send + 't0)>, impl std::future::Future, ()}]`
   = note: required because it appears within the type `std::future::GenFuture<[static generator@src/main.rs:3:32: 7:2 x:std::boxed::Box<(dyn std::marker::Send + 'static)> for<'r, 's, 't0> {std::boxed::Box<(dyn std::marker::Send + 'r)>, &'s std::boxed::Box<(dyn std::marker::Send + 't0)>, impl std::future::Future, ()}]>`
   = note: required because it appears within the type `impl std::future::Future`
   = note: required because it appears within the type `impl std::future::Future`
note: required by `assert_send`
  --> src/main.rs:9:1
   |
9  | fn assert_send<T: Send>(_: T) {}
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[Ekleog]

I've just hit this again, with this:

use std::cell::Cell;
use std::future::Future;

#[derive(Clone)]
struct NonSend {
    foo: Cell<()>,
}

impl NonSend {
    fn get_send(&self) -> Result<&(), ()> {
        Ok(&())
    }
}

async fn bar() {}

async fn foo() -> () {
    let ns = NonSend { foo: Cell::new(()) };
    loop {
        match ns.get_send() {
            Ok(r) => return *r,
            Err(_) => bar().await,
        }
    }
}

pub fn assert_send() -> impl Send + Future<Output = ()> {
    foo()
}

(playground link)

Given that #57017 appears to be in the process of living again, maybe this should be made live again? (note: I have literally no idea how labels work for async/await)

[mzabaluev]

This results in wartiness, and currently misleading compiler help, when structuring generic async code:

use std::fmt::Display;

async fn run(mut state: impl Display) {
    do_stuff(&state).await;
    // ...
}

async fn do_stuff(state: &impl Display) {
    println!("{state}");
}

fn spawn_task<T>(state: T)
where
    T: Display + Send + 'static,
{
    tokio::spawn(run(state));
}

The compiler produces an error and further suggests:

help: consider further restricting this bound
    |
14  |     T: Display + Send + 'static + std::marker::Sync,
    |                                 +++++++++++++++++++

The non-restrictive solution is to make the reference passed to do_stuff mutable (i.e. provably exclusive), even though mutability is not required by the function body.

[ds84182]

If a coroutine only captures Send types, and only has Send types + references to Send types through each yield-point, it should be Send. The references across the yield points are derived from captures, owned (self-referential) values, and globals. Captures must be Send; Globals must be Send. It boils down to whether !Send values are alive across yield points, excluding references to !Sync values.

Probably needs a new auto trait to opt out references, specifically used by coroutines. Something like ThreadUnpin which is impl for all Send types and for references to ThreadUnpin + !Sync types.

I hit this issue today with a huge chunk of async code. I am not looking forward to refactoring it.

[traviscross]

The originally-reported example in this issue now works, as do the later minimizations. So I'm going to close this issue, as it seems resolved.

If you're still hitting an issue, it may be something different, so please file a new issue and provide a minimized example if possible.

[mzabaluev]

The originally-reported example in this issue now works, as do the later minimizations.

My example still fails as of rustc 1.84.0-nightly (705cfe0 2024-11-01). Should I file a separate issue?

[traviscross]

We discussed adding a diagnostic for that over in:

• Misleading help suggests Sync bound when shareable reference is passed across or into await #129105

If you have further thoughts about that, they should probably just go there.

In terms of filing it as an ask that it "just work" somehow, I don't personally see how that could ever be supported in general. It's probably just worth tracking it in #129105 for now, unless you see some smaller special case that could conceivably be supported and want to file that separately.