RFC: Field projection

[y86-dev]

Rendered

Also see the pre-RFC discussion.

Unresolved questions

• How should the limited negative reasoning look like? What are the limitations? Discussed at zulip
• What other alternatives are there for negative reasoning?

[y86-dev]

In ece8b0d I fixed the soundness hole found by pitaj in the internals thread.

Sadly this is to the detriment of ergonomics, but I think keeping existing behavior the same is very important.

[KirilMihaylov]

That might be nice to have for ergonomics but there is already a, mind you stable, way to do this through addr_of!(...). In other words, there already is a mechanism of doing it. IMHO, if this is added, it should just be a sugar around the already existing APIs.

[bjorn3]

addr_of! only works on raw pointers. It does nothing for Pin, MaybeUninit, Cell, UnsafeCell, ...

[Lonami]

I feel like it's missing a drawback: the implicit conversions. While supporting this for MaybeUninit is very powerful and I'd like to have something like that in the language, Option on the other hands feels a bit too "magic" (perhaps because that's not #[repr(transparent)] and incurs a hidden conversion which Rust otherwise avoids).

Rust generally prefers explicit over implicit, and this is adding implicit conversions on the very common dot operator (unlike, say, ? or .await which sticks out a fair bit more). I think this implicit conversion is not given enough weight as a drawback.

I'd also like to see a bit on "how do we teach this?" to users. How come I can access the inner field of some wrapper types but not others? The field was an usize, why is it suddenly an Option<usize>?

[y86-dev]

Option on the other hands feels a bit too "magic" (perhaps because that's not #[repr(transparent)] and incurs a hidden conversion which Rust otherwise avoids).

To me there is no hidden conversion. If I have Option<&mut Struct> and turn it into Option<&mut Field> via struct.field, then it essentially is just doing this:

match struct {
    Some(struct) => Some(&mut struct.field),
    None => None,
}

To me this conversion is fully expressed by the member access syntax (I think the argument for different syntax can be made). It also is functionally very similar to Pin.

(I do not know if it is clear from the RFC, but Option and Pin are #[inner_projecting], which means that they only allow projection when the generic parameter is a Pointer type.)

[KirilMihaylov]

addr_of! only works on raw pointers. It does nothing for Pin, MaybeUninit, Cell, UnsafeCell, ...

True, yet can't you call get() on UnsafeCell? It is how you are supposed to initialize a structure inside MaybeUninit.

[y86-dev]

yes you can do

struct Foo {
    bar: Bar,
}
struct Bar {
    a: usize
}
let data = UnsafeCell::new(...);
let a: &UnsafeCell<usize> = unsafe { &*addr_of_mut!((*data.get()).bar.a).cast::<UnsafeCell<usize>>() }

But

1. you need unsafe
2. it is very verbose

[nbdd0121]

The trait suggested here isn't actually implementable: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=23e3e48fbc2816cf3f6460bceeba6dd2

Probably it should have an extra lifetime param: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=23e3e48fbc2816cf3f6460bceeba6dd2

pub trait FieldProjecting<'a, T> where Self: 'a {
    type Inner<U: 'a>;
    unsafe fn project<U>(self, proj: impl FnOnce(*mut T) -> *mut U) -> Self::Inner<U>;
}

[y86-dev]

The trait must also be unsafe. Because it is only allowed to do projections.

[Lonami]

To me there is no hidden conversion

The layout of T is not guaranteed to be the same as the layout of Option<T>. The cost may neglible, and it may even be optimized out in most cases, but it's still there, because Option is not #[repr(transparent)]. For example, if T is u64, we're talking going from 8 to 16 bytes (twice as big!).

Perhaps support for inner_projecting could be a future possibility instead (but this is just my opinion).

As later pointed I missed the detail where this only applies to pointers, my bad! (I read about it but my brain chose to ignore that for this example.)

[y86-dev]

The layout of T is not guaranteed to be the same as the layout of Option. The cost may neglible, and it may even be optimized out in most cases, but it's still there, because Option is not #[repr(transparent)]. For example, if T is u64, we're talking going from 8 to 16 bytes (twice as big!).

There are no conversions from Option<T> to Option<U>. I am only proposing Option<&mut T> to Option<&mut U> where U is a field of T. &mut T and &mut U have the same layout (this is only available for thin pointers, I do not know of any fat pointers where the pointee has fields, please let me know if you know any!). Additionally I propose allowing this conversion for the pointer types listed in the supported pointers section. These are:

• &mut T, &T
• *mut T, *const T, NonNull<T>, AtomicPtr<T>
• Pin<P> where P is from above

Note that all of these types have the same layout as *mut T.

[KirilMihaylov]

yes you can do

struct Foo {
    bar: Bar,
}
struct Bar {
    a: usize
}
let data = UnsafeCell::new(...);
let a: &UnsafeCell<usize> = unsafe { &*addr_of_mut!((*data.get()).bar.a).cast::<UnsafeCell<usize>>() }

But

1. you need unsafe
2. it is very verbose

My point there was that: (1) with the example of UnsafeCell, you might as well use unsafe for the rest too and (2) while it's very verbose in this plain form, it's not quite that hard to abstract it away in a trait from a crate.

There are already crates providing lens as a functionality, this is no different. This would be a great crate, but I personally don't think it has a place in the standard library, or at least not like this.

[y86-dev]

My point there was that: (1) with the example of UnsafeCell, you might as well use unsafe for the rest too and (2) while it's very verbose in this plain form, it's not quite that hard to abstract it away in a trait from a crate.

1. I think isolating unsafe is important and we should strive to make every operation safe, if it can be done.
2. I have a demo crate doing this, but it still needs some type hints and the syntax is not optimal (postfix macros would help there), a language feature would improve on both points (and fits well together with Pin projections)

There are already crates providing lens as a functionality, this is no different. This would be a great crate, but I personally don't think it has a place in the standard library, or at least not like this.

To me, field projection feels different compared to lenses. It also is something native to Rust. We have these different wrapper types (Cell,UnsafeCell, etc.) that impose something on the type they contain (be it mutability or the ability to be partially initialized). They also affect the fields of structs they wrap and so producing a pointer to a wrapper just around the field is to me something we should support.

This is something general that many Rust programs will need to do, so in my opinion it is something the language should provide. Most of the impact will be by allowing this for Pin and MaybeUninit. I am open to the idea of moving Option, Cell etc. into the future possibilities section and having their projections be unstable for longer.

[Jules-Bertholet]

I do not know of any fat pointers where the pointee has fields, please let me know if you know any

struct Foo {
    a: i32,
    b: dyn Send,
}

[y86-dev]

I think for the beginning we are not going to support those. But conversion can only happen in one way, right? (so from fat => thin) So it might be alright to allow that at some point (it would be weird to not support pin projection for Foo).

[ChayimFriedman2]

Option projection can be almost as easy with try blocks, and given that there is a branch there I feel like it suits Rust model of showing costs explicitly better.

[y86-dev]

There is also the big question of how we want pin projection to look like. Here is a zulip post about that

[matthieu-m]

I do like the idea of field projection in general, but I am not at all convinced that making . more magical than it already is (Deref/DerefMut/partial moves, ...) is the way to go there.

You reference RFC 2708, and I cannot help but feel that re-ifying field access is a direction that should be explored more: it is useful on its own, and it seems like it could power field projection.

From a pure syntax perspective, it would certain be more verbose, especially in the absence of aliasing, yet:

maybe_uninit.field

maybe_uninit.project(Type::field)

The latter is indubitably more verbose, but at the same time it's within a factor of x2 and has zero magic.

---

Is Field Projection frequent enough to warrant special treatment?

I believe not. It is used, certainly, but UnsafeCell and MaybeUninit are used sparingly -- being unsafe -- and even though Cell and Pin are used more frequently, they are not that frequent either.

Contrast with Deref, which is used all the time.

It's clear why Deref warrants "magic" in the . operator, and it's not clear why field projection would.

Should Field Projection be implementable safely?

It's a laudable goal, yet at the same time all the types you wish to implement for already have an unsafe implementation to start with, so one more method with an unsafe implementation on them certainly won't be a deal-breaker.

---

So I'll propose an alternative, lightweight design:

• Compiler: Type::field is now valid syntax, and desugars to fn(*mut Type) -> *mut FieldType.
• Standard Library: UnsafeCell, Cell, MaybeUninit, Pin, Option, ... all implement an inherent project function.

That's it. No complex attributes for the compiler to implement, no trait.

Any additional implementation cost, any additional language or library change, should be justified as providing sufficient added value above what this lightweight alternative offers.

[mkroening]

This would not apply to slices and other Aligned types (#3319), though, right?

[y86-dev]

Not so sure how knowing alignment will change this. For example if I have a &dyn Any how do I even know that there is a field of some type present?

[mkroening]

I am coming here from a separate discussion on a potential VolatileCell type in the language (rust-lang/unsafe-code-guidelines#411 (comment)).

It would be great to have some kind of field projection on a volatile type that is correct and ergonomic.

An ergonomic, completely separate approach from this RFC to this would be allowing the following:

let foo: &mut VolatileCell<Foo> = magic!();
let bar = &mut foo.bar; // &mut VolatileCell<Bar>

This should verify borrows as usual: mutable references must be exclusive, and all references must satisfy any lifetime requirements. Specifically, projecting to different fields should work and non-mutable borrowing are allowed to overlap.

Going by the VolatileMem example from the guide-level explanation, this RFC does not seem to help with this, since VolatileMem::map takes self by value. The VolatileMem abstraction would have to reinvent references and account for both read and write types, as drafted in rust-osdev/volatile#29/src/volatile_ref.rs. This would still not allow for disjoint simultaneous mutable borrowing of fields, though, I think.

Would this reference magic be part of the "general custom field projection" that is referenced in the motivation, or would this be similar to "the very first design [that] only supported transparent wrapper types" design referenced in the RFC History?

[y86-dev]

I am coming here from a separate discussion on a potential VolatileCell type in the language (rust-lang/unsafe-code-guidelines#411 (comment)).

It would be great to have some kind of field projection on a volatile type that is correct and ergonomic.

An ergonomic, completely separate approach from this RFC to this would be allowing the following:

It is not completely separate :)
At the beginning, I also thought about overloading the . operator, but it was not well received.

let foo: &mut VolatileCell<Foo> = magic!();
let bar = &mut foo.bar; // &mut VolatileCell<Bar>

This should verify borrows as usual: mutable references must be exclusive, and all references must satisfy any lifetime requirements. Specifically, projecting to different fields should work and non-mutable borrowing are allowed to overlap.

Going by the VolatileMem example from the guide-level explanation, this RFC does not seem to help with this, since VolatileMem::map takes self by value. The VolatileMem abstraction would have to reinvent references and account for both read and write types, as drafted in rust-osdev/volatile#29/src/volatile_ref.rs. This would still not allow for disjoint simultaneous mutable borrowing of fields, though, I think.

Note that VolatileMem<T> from the guide-level explanation is a smart pointer (it contains a *mut T), so it makes sense to map self.

Would this reference magic be part of the "general custom field projection" that is referenced in the motivation, or would this be similar to "the very first design [that] only supported transparent wrapper types" design referenced in the RFC History?

For your VolatileCell<T> example, you do this:

impl<T> VolatileCell<T> {
    pub fn map<F: Field<Base = T>>(&mut self) -> &mut VolatileCell<F::Type> {
        todo!()
    }
}

Then you could do:

let foo: &mut VolatileCell<Foo> = magic!();
let bar: &mut VolatileCell<Bar> = foo.map::<field_of!(Foo, bar)>();

An ergonomics problem that this has is that you cannot borrow multiple fields at the same time:

let foo: &mut VolatileCell<Foo> = magic!();
let bar: &mut VolatileCell<Bar> = foo.map::<field_of!(Foo, bar)>();
let baz: &mut VolatileCell<Baz> = foo.map::<field_of!(Foo, baz)>();
// cannot use `bar` here.

If we would get some better custom references support, this could be fixed orthogonally.

[tgross35]

I think we may not need field_of if we come up with type-level field access syntax, aka field paths (swift) or something like member pointers (C++).

This came up during discussion of offset_of at rust-lang/rust#106655 and I think we may want it at some point.

(Needless to say, this would definitely not block an initial implementation using field_of)

[Nadrieril]

I notice that the current proposal doesn't really need special compiler support (playground). The easiest way to help move this RFC forward would be to define a crate with some version of Field/field_of, and have people use it to see if the design is right.

[y86-dev]

That implementation is not sound, I can decide to call field_of!(Foo<u32>, field: bool). That can be fixed by using a mechanism to check for the correct type though.

The real problem with the implementation is that it is not actually useful in the sense that it does not allow projections where we need to store some information on the field. For example in the case of Pin projections. Since you cannot implement a trait on field_of!(...) and then expect another invocation of field_of! to observe that same trait.

So yes you can implement it like this, but I would say that the implementation that I linked in the crates section by @nbdd0121 is much more mature.

[Nadrieril]

The real problem with the implementation is that it is not actually useful in the sense that it does not allow projections where we need to store some information on the field.

Ah, I did not realize this was a need; is it load-bearing? As in, is it required for meaningful experimentation?

AFAIU for pin projections this macro+trait would get us a safe project<F: Field>(Pin<&mut T>) -> Pin<&mut S>, which is the footgunny part of pin projection. Going from Pin<&mut S> to &mut S on fields that aren't structurally pinned feels easier (and a lot less unsafe), maybe a wrapper type could do that. And for MaybeUninit, ManuallyDrop, UnsafeCell etc afaik there's no need for field-specific data.

The current proposal is so clean and minimal, it gave me hope that a crate that implements it even partially could solve many problems long before we get compiler support.

[y86-dev]

The real problem with the implementation is that it is not actually useful in the sense that it does not allow projections where we need to store some information on the field.

Ah, I did not realize this was a need; is it load-bearing? As in, is it required for meaningful experimentation?

You can experiment with that API, but it does not offer you the same amount of freedom in what you can achieve with it.

AFAIU for pin projections this macro+trait would get us a safe project<F: Field>(Pin<&mut T>) -> Pin<&mut S>, which is the footgunny part of pin projection. Going from Pin<&mut S> to &mut S on fields that aren't structurally pinned feels easier (and a lot less unsafe), maybe a wrapper type could do that. And for MaybeUninit, ManuallyDrop, UnsafeCell etc afaik there's no need for field-specific data.

I agree, this simpler version already seems useful for the basic types that have constant projections (so the projection does not depend on the field type).
The wrapper concept for handling not structurally pinned fields that are !Unpin might need some more inspection, IIRC that was also brought up in the design meeting, but it seems a bit hacky to me.

Rust for linux has some more applications of non-constant projections, for example we might want to use it to make the RCU API more ergonomic. And I can imagine this being useful for memory mapped registers too.

[y86-dev]

I have taken some time to dig through the compiler source code and I have actually managed to implement the field_of! macro. You can take a look at the code here.
I have very little experience with the rustc codebase, so I have probably done something wrong or in a weird way. If anyone has any suggestions, feel free to leave a comment/issue there.

[Nadrieril]

Excellent! You could submit this as a PR. Adding a feature gate requires a lot less procedure than getting an RFC accepted. They'll tell you if this is acceptable (you'll definitely have to add tests!) and we'll all get to experiment with it

[Jules-Bertholet]

Also, it's possible to exhaustively enumerate all possible values of a type, but there is no way to enumerate all implementors of a trait.

To expand on this: I see this proposal as a first step toward a more general compile-time reflection mechanism, in the style of "A Mirror for Rust". That more general feature will require a mechanism for performing an operation with all a struct's fields. If the way to refer to "a field of type T" is "type implementing Field { Base = T }", expressing this will likely be difficult or impossible, because (from the type system's perspective) any downstream crate could introduce new implementations of Field with arbitrary associated types. In contrast, if the way to refer to stuct fields is via concrete values of a particular type, enumerating all fields should be far simpler, as the set of possible values of a type is bounded.

[Nadrieril]

This isn't anything like a concrete proposal, but I wanted to note the opposite direction of what @Jules-Bertholet just mentioned: arguably the only thing that projections need is "this value has some data of type T at offset OFFSET that can be retrieved safely". That would make projection possible for enums and unsized types and weirder things. I have no idea what the API could look like. Of course simple structs could still have a mechanism to enumerate their fields, each of which would carry such a typed offset thingie.

[y86-dev]

Also, it's possible to exhaustively enumerate all possible values of a type, but there is no way to enumerate all implementors of a trait.

To expand on this: I see this proposal as a first step toward a more general compile-time reflection mechanism, in the style of "A Mirror for Rust".That more general feature will require a mechanism for performing an operation with all a struct's fields. If the way to refer to "a field of type T" is "type implementing Field { Base = T }", expressing this will likely be difficult or impossible, because (from the type system's perspective) any downstream crate could introduce new implementations of Field with arbitrary associated types. In contrast, if the way to refer to stuct fields is via concrete values of a particular type, enumerating all fields should be far simpler, as the set of possible values of a type is bounded.

I do not have the bandwidth of changing this proposal to be part of a bigger system of compile-time reflection. If you (or anyone else) wants to tackle the problem, feel free to reach out to me, I can take a look if what you are proposing will cover the use-cases of this RFC.

I also think that the general compile-time reflection feature is going to take a long time to develop. RFL needs these projection as soon as we can get them. I do not want that trying to find a "perfect" general compile-time reflection implementation prevents us from having field projections in a couple of months (of course I am talking about it being an unstable feature).

[y86-dev]

This isn't anything like a concrete proposal, but I wanted to note the opposite direction of what @Jules-Bertholet just mentioned: arguably the only thing that projections need is "this value has some data of type T at offset OFFSET that can be retrieved safely". That would make projection possible for enums and unsized types and weirder things. I have no idea what the API could look like. Of course simple structs could still have a mechanism to enumerate their fields, each of which would carry such a typed offset thingie.

As I said before, our projections need more than just the offset and the type of the field. We also need to store information on each field, be it for pin projections, RCU projections or memory mapped registers.

For enums I do not see any issue with the current proposal. I haven't implemented them yet, but they could definitely work. Unsized types where we know the layout (i.e. structs where the last field is unsized) should also be no problem.

[Jules-Bertholet]

I do not want that trying to find a "perfect" general compile-time reflection implementation prevents us from having field projections in a couple of months (of course I am talking about it being an unstable feature).

I have no objection to having an imperfect API as an experimental feature gate, especially if it gives feedback that improves the final result. However, the current lang team policy seems to be that RFCs should present a polished and complete design.

[y86-dev]

I do not want that trying to find a "perfect" general compile-time reflection implementation prevents us from having field projections in a couple of months (of course I am talking about it being an unstable feature).

I have no objection to having an imperfect API as an experimental feature gate, especially if it gives feedback that improves the final result. However, the current lang team policy seems to be that RFCs should present a polished and complete design.

Sure that is fine. When I opened this RFC, I thought that making an RFC is the only way to introduce these changes (note that at the beginning the RFC had many more things compared to now). Also at the time I had not yet implemented anything and tinkering on the compiler seemed too daunting.

I have no idea what you are trying to convey by this link. Can you please point me to a specific comment?

[GKFX]

Could I suggest updating this so the second argument is parsed identically to that of offset_of? This keeps the syntax for nested fields, enums, etc. consistent, even if they are not actually implemented in the first release of field_of. The macro argument becomes $($fields:expr)+ and this is parsed by parse_floating_field_access.

Enum support would be useful in e.g. Option::as_slice. Currently that uses a bare offset_of and is dependent on getting two casts and a ptr::byte_add right; writing that sort of code could be much more robust with proper field projection.

[y86-dev]

I am picking up this work again. See this zulip topic for more info. I am starting out by first collecting the use cases, as I feel that last time, the motivation for this feature was not sufficient enough. Here is the document presenting all current use cases for field projections (also linked from the zulip topic, please leave your feedback there).