safe transmute: support non-ZST, variantful, uninhabited enums

[jswrenn]

Previously, Tree::from_enum's implementation branched into three disjoint cases:

1. enums that uninhabited
2. enums for which all but one variant is uninhabited
3. enums with multiple variants

This branching (incorrectly) did not differentiate between variantful and variantless uninhabited enums. In both cases, we assumed (and asserted) that uninhabited enums are zero-sized types. This assumption is false for enums like:

enum Uninhabited { A(!, u128) }

...which, currently, has the same size as u128. This faulty assumption manifested as the ICE reported in #126460.

In this PR, we revise the first case of Tree::from_enum to consider only the narrow category of "enums that are uninhabited ZSTs". These enums, whose layouts are described with Variants::Single { index }, are special in their layouts otherwise resemble the ! type and cannot be descended into like typical enums. This first case captures uninhabited enums like:

enum Uninhabited { A(!, !), B(!) }

The second case is revised to consider the broader category of "enums that defer their layout to one of their variants"; i.e., enums whose layouts are described with Variants::Single { index } and that do have a variant at index. This second case captures uninhabited enums that are not ZSTs, like:

enum Uninhabited { A(!, u128) }

...which represent their variants with Variants::Single.

Finally, the third case is revised to cover the broader category of "enums with multiple variants", which captures uninhabited enums like:

enum Uninhabited { A(u8, !), B(!, u32) }

...which represent their variants with Variants::Multiple.

This PR also adds a comment requested by @RalfJung in his review of #126358 to compiler/rustc_const_eval/src/interpret/discriminant.rs.

Fixes #126460

r? @compiler-errors

[rustbot]

Some changes occurred to the CTFE / Miri engine

cc @rust-lang/miri

[jswrenn]

Requested by @RalfJung here: #126358 (comment)

[RalfJung]

What about enums like enum E { A(!, u64), B(!, u32) }? These use Multiple variants despite all variants being uninhabited. See here for the layout debug info.

[jswrenn]

I've amended this PR to add a test case for such enums.

Those enums are supported, as before; they're covered by the Variants::Multiple case here:

rust/compiler/rustc_transmute/src/layout/tree.rs

Lines 363 to 380 in 4a39faa

	Variants::Multiple { tag_field, .. } => {
	// `Variants::Multiple` denotes an enum with multiple
	// inhabited variants. The layout of such an enum is the
	// disjunction of the layouts of its tagged variants.
	// For enums (but not coroutines), the tag field is
	// currently always the first field of the layout.
	assert_eq!(*tag_field, 0);
	let variants = def.discriminants(cx.tcx()).try_fold(
	Self::uninhabited(),
	|variants, (idx, ref discriminant)| {
	let variant = layout_of_variant(idx)?;
	Result::<Self, Err>::Ok(variants.or(variant))
	},
	)?;
	return Ok(Self::def(Def::Adt(def)).then(variants));

They don't require any special handling, because they don't get the !-like representation implemented here:

rust/compiler/rustc_abi/src/layout.rs

Lines 195 to 197 in f8e5660

	None if is_enum => {
	return Some(self.layout_of_never_type());
	}

[RalfJung]

So in which case do they fall from the PR description?

2. enums for which all but one variant is uninhabited
3. enums with multiple inhabited variants

[jswrenn]

Whoops, that needs to be updated.

[RalfJung]

Btw I am getting a notification from github for each time you push a commit that has @RalfJung in the commit message... I'd appreciate if you could remove the ping from the commit message. :)

[jswrenn]

Whoops, that needs to be updated.

Done

Btw I am getting a notification from github for each time you push a commit that has @RalfJung in the commit message... I'd appreciate if you could remove the ping from the commit message. :)

Yikes! Sorry. Done!

[RalfJung]

Whoops, that needs to be updated.

Ah okay. :)
If that exactly corresponds to Variants::Single and Variants::Multiple, then that is worth mentioning.

[jswrenn]

Agreed. I've updated the PR description and commit message to note the correspondence to Variants::Single and Variants::Multiple.

[compiler-errors]

@bors r+ rollup

[bors]

📌 Commit df1d616 has been approved by compiler-errors

It is now in the queue for this repository.