RFC: Rust Has Provenance

[RalfJung]

Pointers (this includes values of reference type) in Rust have two components.

• The pointer's "address" says where in memory the pointer is currently pointing.
• The pointer's "provenance" says where and when the pointer is allowed to access memory.

(This is disregarding any "metadata" that may come with wide pointers, it only talks about thin pointers / the data part of a wide pointer.)

Whether a memory access with a given pointer causes undefined behavior (UB) depends on both the address and the provenance:
the same address may be fine to access with one provenance, and UB to access with another provenance.

In contrast, integers do not have a provenance component.

Most of the rest of the details, such as a specific provenance model, are intentionally left unspecified.

This RFC very deliberately aims to be as minimal as possible, to just get the entire Rust Project on the "same page" about the long-term future development of the language.

📖 RFC book

Rendered

Tracking issue

[WaffleLapkin]

Thanks for writing this! I think it's a good idea to set in stone this fact :)

[mattheww]

Note that "provenance" is a somewhat unfortunate term.

The above is true.

Is this RFC declaring only that Rust's abstract machine includes a notion of provenance, or is it also declaring that "provenance" is going to be the public-facing term?

(I think "place" has been a valuable improvement on "lvalue", so maybe there's an opportunity to make a similar improvement here.)

[RalfJung]

Is this RFC declaring only that Rust's abstract machine includes a notion of provenance, or is it also declaring that "provenance" is going to be the public-facing term?

The RFC is primarily concerned with the Abstract Machine.
However, it is also meant to pave the way for the stabilization of the strict provenance APIs, which would obviously use the term in public ways (and may even include it in the names of some functions).

[scottmcm]

Thanks for writing this up! It was a good read.

After the design meeting we had about this, I agree that there's no practical alternative to having provenance in Rust, and thus we ought acknowledge that fact in a way that's visible like this. Let's see if others agree:
@rfcbot fcp merge

[rfcbot]

Team member @scottmcm has proposed to merge this. The next step is review by the rest of the tagged team members:

• @joshtriplett
• @nikomatsakis
• @pnkfelix
• @scottmcm
• @tmandry

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.

[tmandry]

+1 to what @scottmcm said and thanks for writing this up, @RalfJung. This reflects the understanding I took away from the lang design meeting last year and I think the RFC speaks for itself in terms of motivation.

@rfcbot reviewed

[traviscross]

@rustbot labels +I-lang-nominated

Even though we've already proposed FCP merge here, let's nominate so we discuss it in one meeting for at least general awareness.

Huge thanks to @RalfJung for writing this up. I earlier reviewed this at the draft stage and am +1 on it.

[jrtc27]

This reads strangely. Per your definition, if it was always undefined behaviour, it was always prescriptive, regardless of whether implementations exploited it. If it was not always undefined behaviour, the important thing is that it became undefined, not that implementations exploited it. The way you've written it reads to me like you're saying it was always undefined behaviour, but was prescriptive until compilers exploited it for optimisations?

[RalfJung]

Standard C was never worded precise enough to really say whether this alias analysis is allowed. The wording about one-past-the-end pointers hints at prescriptive provenance, but it's not sufficiently clear.

So as is often the case with C, we have to look at implementations to see how the standard is interpreted.
In de-facto-C, I think this was descriptive for a while, but at some point (decades ago) alias analysis made it prescriptive. The example from the RFC shows prescriptive provenance on the oldest version of clang for which godbolt can still execute code (clang 3.4.1). I couldn't find an exact release date but it seems to be from around 2013/2014.

[lonjil]

When C was standardized, they didn't really consider provenance, but many basic optimizations that C compilers do, such as register allocating local variables, both seem "obviously" correct to everyone and imply provenance.

[RalfJung]

Register allocation of variables that never have their address taken, or where no information about the address is ever used to influence what happens with other unrelated pointers, is fine without provenance under the non-deterministic-allocation model.

[VitWW]

I think its useful to add in the RFC alternative names of a term:

• provenance
• place
• entity
• gist
• responsibility
• conformity
• association
• essence
• ...

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[VitWW]

As title for RFC topic "Rust Has Provenance" is a good and loud name, but for language feature this title is self-referencing, trivial and meaningless same time.
It is better to change the title into something more meaningful before merging, like "Adding Provenance to Pointer Model" or "Pointer's Provenance"

[Lokathor]

The "point" of the RFC, at least in the initial draft that I sent to Ralf, and I think it's also fair to say in Ralf's draft that is now in FCP here, is that we're not actually adding anything.

We're just admitting, concretely, what is already the case.

Merging the RFC doesn't change the language, it just focuses future discussions about the development of the language (which otherwise often stall out).

[traviscross]

@rustbot labels -I-lang-nominated

We mentioned this for awareness in the meeting last week, and it's now in FCP, so we can unnominate.

Thanks for @RalfJung for pushing this forward, and to @Lokathor for the initial draft work.

[pnkfelix]

A point that came out of a zulip conversation that I felt worth lifting up to this less transient forum: The sentence "Most of the rest of the details, such as a specific provenance model, are intentionally left unspecified" is carrying a lot of weight.

In particular, one cannot immediately generalize the given shrptr example to other arbitrary *mut T types and make the same conclusion, for all choices of T, that an analogous shrptr.write(...) line for that other T is also UB. In particular, there are some choices for T where, under all memory models under consideration currently, we would almost certainly consider the behavior to be well-defined.

(This is all consistent with the text of the RFC, since all the RFC is saying is that "Provenance Exists", and is not making statements about what precise effect that has on the rest of the language; more precise statements are left for future work in defining the specific provenance (+ memory) model.)

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

This will be merged soon.

[scottmcm]

Huzzah! The @rust-lang/lang team has decided to accept this RFC.

Thank you @RalfJung and everyone who contributed!

To track further discussion, subscribe to the tracking issue here:
rust-lang/rust#121243

[jsgf]

Just got around to reading this. It's very similar to the Annelid tool @nnethercote and I did for Valgrind, where pointers were tagged by a "segment" (eg a segment corresponding to a malloc), and all memory load/stores were checked against the pointer's segment. In practice this didn't work all that well, as it was trying to track the segment through arbitrary raw assembler instructions which often entangled pointers from different segments (eg xor pointer swap), but it was very interesting to play with.

Discussed in Nick's thesis https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-606.pdf

[jsgf]

"The data part of a wide pointer" seems ambiguous - I read this as referring to the metadata, but from context I assume it means the address/pointer part of it.

[WaffleLapkin]

I think it is the usual way to refer to the address part (at least in Rust). For example from_raw_parts:

pub fn from_raw_parts<T>(
    data_address: *const (),
    metadata: <T as Pointee>::Metadata
) -> *const T
where
    T: ?Sized,

"data" refers to the fact that the "data" address points to the data. This is in contrast to the metadata which for trait objects is a pointer pointing to the vtable.

[jsgf]

I guess, but I think "address" is the significant part of that name (though I realize the whole point of the RFC is to make a clear distinction between "address" and "pointer").

[RalfJung]

When we have "metadata", usually the things that this is "meta about" is the "data". I have not seen any context where one would abbreviate "metadata" as "data", that is losing the key distinction of it being "meta"!

Calling it an "address" definitely does not work in the context of this RFC. On nightly, that function signature has been changed:

pub fn from_raw_parts<T>(
    data_pointer: *const (),
    metadata: <T as Pointee>::Metadata
) -> *const T
where
    T: ?Sized,

[RalfJung]

Just got around to reading this. It's very similar to the Annelid tool @nnethercote and I did for Valgrind, where pointers were tagged by a "segment" (eg a segment corresponding to a malloc), and all memory load/stores were checked against the pointer's segment. In practice this didn't work all that well, as it was trying to track the segment through arbitrary raw assembler instructions which often entangled pointers from different segments (eg xor pointer swap), but it was very interesting to play with.

That's cool, I didn't know about this tool!

The key difference is that here we are making this extra information (tag/provenance/segment) part of the spec, and in the spec we have all the information we need to define how it propagates. XOR of pointers is one of the very few patterns that need "exposed" pointer semantics, lucky enough it is very rare. Specifying this fully is still not a solved problem, but this is "just" part of figuring out how to make int2ptr casts work in a world with provenance.

Doing this kind of tracking in valgrind after the compiler erased all information is of course a lot harder. @pnkfelix' valgrind tool for Stacked Borrows must be doing something similar.