Pyclasses and structs with mutable references

[entropidelic]

Hello guys, I am working on a project that requires both accessing a Rust type from Python and allowing that Rust type to invoke Python code. I've put together the smallest possible example in this repo to illustrate.

• We have a pure rust inner crate,
• We have an ffi crate, dependent on inner, which uses pyo3 to implement the FFI bindings to Outer, a struct that wraps a mutable reference to Inner,
• We have a main python process using all these. The desired code that this process should be run can be seen here

We understand that to create variables accessible by the Python interpreter in py.run() scope, this is done via a locals PyDict. To give it access to a Rust struct, the definition of this struct should be annotated with a #[pyclass] attribute. Finally, an instance of it should be wrapped in a PyCell to insert it to the locals dictionary.

At this point our confusion appears:
As the Outer struct is wrapping a mutable reference to Inner, a lifetime parameter should be specified. But when the lifetime is set along with the #[pyclass] attribute to the struct, we get an error complaining that pyclasscannot have generic parameters.

We know by design that Outer will not be mutated while it is accesible by the running python code. How can we convince pyo3 of this and pass in a pyclass-wrapped struct that references a rust-allocated data structure?

[mejrs]

Hey 👋

When you do...

#[pymethods]
impl Outer {
    #[new]
    fn new() -> Outer {
        Outer {
            inner: &mut Inner::new(),
        }
    }
}

...you're creating a new Inner, storing a mutable reference to it, and then immediately dropping it. You're create a dangling reference which is not allowed in safe Rust. If you fix that you can likely make this work.

NB: looking up things in locals() might turn out to brittle and surprising, because it's effectively global state (but worse). I'd recommend passing it as a parameter.

[entropidelic]

thank you very much for your answer @mejrs 😄. That was certainly an error in my example. It is corrected now, but sadly the problem persists: When defining the Outer struct wrapping a mutable reference to Inner, a lifetime parameter should me specified, and when I do, the compiler complains saying #[pyclass] cannot have generic parameters. How can I get this to work?

[mejrs]

Python objects having lifetimes doesn't make sense. Once you give a python object to python code, it can keep it around forever. Only the 'static lifetime makes sense.

Your best option is a design like:

#[pyclass]
pub struct Outer {
    inner: Inner,
}

( ^ would be my preference)

or

#[pyclass]
pub struct Outer {
    inner: Py<Inner>,
}

or

#[pyclass]
pub struct Outer {
    inner: Arc<Inner>,
}

Generally (in Rust, not just in pyo3) you don't want your structs to have references to other structs. Try to use composition (so, structs in other structs) rather than structs having references to other structs.