Allow null-pointer-optimized enums in FFI if their underlying representation is FFI safe

[mjbshaw]

I'm not sure if this requires an RFC. I attempted to start a discussion on internals.rust-lang.org and when no one really objected I figured I'd go ahead and try implementing this.

This allows types like Option<NonZeroU8> to be used in FFI without triggering the improper_ctypes lint. This works by changing the is_repr_nullable_ptr function to consider an enum E to be FFI-safe if:

• E has no explicit #[repr(...)].
• It only has two variants.
• One of those variants is empty (meaning it has no fields).
• The other variant has only one field.
• That field is one of the following:
  • &T
  • &mut T
  • extern "C" fn
  • core::num::NonZero*
  • core::ptr::NonNull<T>
  • #[repr(transparent)] struct wrapper around one of the types in this list.
• The size of E and its field are both known and are both the same size (implying E is participating in the nonnull optimization).

This logic seems consistent with the Rust nomicon.

[rust-highfive]

r? @estebank

(rust_highfive has picked a reviewer for you, use r? to override)

[Centril]

r? @pnkfelix

[Centril]

cc @rust-lang/wg-unsafe-code-guidelines

[hanna-kruppe]

I do think we should change the lint to accepting types that are already effectively guaranteed to participate in Option layout optimizations (at minimum, the NonZero* types and repr(transparent) newtypes). However, there are also situations where rustc does apply layout optimizations but we don't obviously (i.e., without an RFC) want to commit to those optimizations.

Relatedly, there may be repr(C) library-defined types which happen to have a niche in a private field but whose author didn't intend to guarantee that Option<Foo> is FFI-safe. For example, a library author may at first use bool for a private field but later decide to use uint8_t to be defensive against the C side storing a byte other than 0 or 1, without expecting this to break clients.

So I think the check "Option<T> is the same size as T" that this PR currently implements is too broad. I would be more comfortable to have us start out with something more restrictive. I am not sure what that should be, though -- that uncertainty might be why this generalization hasn't happened yet.

[mjbshaw]

@Centril Thanks; I've made the changes you recommended. Let me know if there are others.

@rkruppe I agree. I've revised the code to now only allow:

• &T and &mut T
• extern "C" fn
• core::num::NonZero*
• core::ptr::NonNull<T>
• #[repr(transparent)] struct wrapper around one of the types in this list.

I'll update the top-level comment to reflect this change.

[bors]

☔ The latest upstream changes (presumably #60435) made this pull request unmergeable. Please resolve the merge conflicts.

[joshtriplett]

The idea of this change looks great to me. We want to define this behavior, carefully, and not leave it undefined. And this allows people to define safer types for FFI functions.

We do need further review on the code itself, but let's get consensus on the concept:

@rfcbot merge

[rfcbot]

Team member @joshtriplett has proposed to merge this. The next step is review by the rest of the tagged team members:

• @Centril
• @aturon
• @cramertj
• @eddyb
• @joshtriplett
• @nikomatsakis
• @pnkfelix
• @scottmcm
• @withoutboats

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[scottmcm]

👍 now that this is the "careful" version (following #60300 (comment))

[pnkfelix]

... why isn't there the standard list of checkboxes for all members of labelled teams (T-lang in this case) on the rfcbot comment

[rust-highfive]

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:0b529774:start=1557164028516611654,finish=1557164116066617906,duration=87550006252
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
[00:14:49]    Compiling rustc_mir v0.0.0 (/checkout/src/librustc_mir)
[00:14:49]    Compiling rustc_typeck v0.0.0 (/checkout/src/librustc_typeck)
[00:14:49]    Compiling rustc_allocator v0.0.0 (/checkout/src/librustc_allocator)
[00:20:20]    Compiling rustc_lint v0.0.0 (/checkout/src/librustc_lint)
[00:20:20] error: expected one of `::`, `=`, or `|`, found `!=`
[00:20:20]     |
[00:20:20]     |
[00:20:20] 535 |             if let AdtKind::Struct != field_def.adt_kind() {
[00:20:20]     |                                    ^^ expected one of `::`, `=`, or `|` here
[00:20:21]    Compiling rustc_traits v0.0.0 (/checkout/src/librustc_traits)
[00:20:22] error: aborting due to previous error
[00:20:22] 
[00:20:22] error: Could not compile `rustc_lint`.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[nikomatsakis]

@pnkfelix perhaps it was slow -- the list exists now

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[bors]

☔ The latest upstream changes (presumably #60630) made this pull request unmergeable. Please resolve the merge conflicts.

[gnzlbg]

One of those variants is empty.

Here empty means that the variant has no fields, right? That is, variants with zero-sized fields are not empty, and, for example, Result<(), NonZeroU32> still is an improper ctype, right? If so, is there a test covering this?

---

EDIT: many C APIs return an error code of type int that is zero on success and non-zero on error, so the Result<(), NonZeroCInt> case is something that users might attempt to use in C FFI directly. However, we currently do not guarantee any niche optimizations for Result-like types, so it makes IMO sense to add an ui test that checks that the improper ctypes lint fails for these types until we start guaranteeing them.

[mjbshaw]

this branch has merge commits, that's generally discouraged, please rebase (and while at it, you could squash some back-and-forth in the history, but no big deal)

Thanks, I wasn't sure what Rust's policy was on this. I'll keep that in mind going forward (also: done).

the new rustc_* attribute doesn't have a feature gate test, please add one

Done.

[hanna-kruppe]

Thank you for your work and patience!

@bors r+

[bors]

📌 Commit a31dc8e has been approved by rkruppe

[oli-obk]

Could this just reuse the attribute in the line above?

[RalfJung]

I think the goal was to separate "performing the optimization" from "guaranteeing the optimization".