Extend stack probe support to non-tier-1 platforms, and clarify policy for mitigating LLVM-dependent unsafety

[bstrie]

As of #42816 , all (current) tier-1 platforms are protected against the potential memory unsafety demonstrated in #16012 . However, this protection is dependent upon platform-dependent LLVM support, so AFAIK most of our supported tier-2 and tier-3 platforms ( https://forge.rust-lang.org/platform-support.html ) are still theoretically unprotected.

Proper resolution of this issue will depend on extending LLVM's stack probe support for various platforms, (which should be done regardless of the discussion in the next paragraph). See https://reviews.llvm.org/D34387 for pcwalton's original x86 implementation, https://reviews.llvm.org/D34386 for the implementation of the attribute itself, and #42816 for the work needed on rustc's side.

But here's the more important concern: the list of potential platforms is unbounded, and we need to decide where to draw the line. As minor as the safety hole may be, is it an abrogation of Rust's claim to guarantee memory safety if that safety is only enforced on "popular" platforms? If so, is it worth considering some sort of blanket check that can be implemented in the frontend for use only on platforms where stack probes are unimplemented? For instance, given that we statically know the stack size of all types, could we perhaps enforce an upper bound on the size of a type (perhaps only in debug mode)? I think this discussion may be relevant to #10184 as well.

[whitequark]

so AFAIK most of our supported tier-2 and tier-3 platforms

All of them. LLVM only currently has support for x86.

[cuviper]

For instance, given that we statically know the stack size of all types, could we perhaps enforce an upper bound on the size of a type (perhaps only in debug mode)?

GCC has -Wframe-larger-than=len, so I could see rustc having something similar. That can't count alloca/VLAs/etc. though. It's also hard to say what a reasonable limit is generally, so I think it would have to be user configurable. e.g. a 1MB stack item might be fine in isolation, but will blow up in mild recursion, or if a few such functions are nested. A kernel probably wants more like ~1KB max.

[bstrie]

@cuviper Would it suffice to give the user the ability to enforce that no given function ever exceeds the size of the guard page? Maybe I'm naive, but I'm assuming that our guard page support is platform-independent, and that we can calculate the maximum size of any given monomorphized function (I can't think of a reason why we couldn't given that we don't support alloca).

[cuviper]

Page size is not a static value on some arches, although for this purpose you could probably get away with just assuming a minimum 4KB. Stack probing on those arches will probably have to use 4KB strides as a worst case, even when the real page size is 64KB/etc., unless we want those to query sysconf at runtime.

Also, as we learned in #43052, the Linux kernel now implements a much larger effective guard size, but we have no way to detect that at runtime, unless you tried to probe a fault on purpose.

Anyway, 4KB seems to me like a pretty low stack-frame threshold for userspace programs. I guess some users might be happy with that, but I think a tunable threshold would be more useful.

[cuviper]

There's also different behavior between the main thread's growable stack and the fixed stack space of other threads. See pthread_attr_setguardsize -- defaults to one page, which Rust doesn't change.

[bstrie]

I suppose another difficulty is that LLVM can inline functions as it pleases, which would poke a hole in any attempt to statically lint against large functions in the frontend.

[whitequark]

Would it suffice to give the user the ability to enforce that no given function ever exceeds the size of the guard page?

This is not realistic given the way LLVM is designed. You'd have to plug your pass very late into the MI pipeline, which by itself will need target-specific LLVM patches (that are quite a bit more complex than stack probes themselves).

[nikomatsakis]

triage: P-medium

[GreenReaper]

so AFAIK most of our supported tier-2 and tier-3 platforms

All of them. LLVM only currently has support for x86.

Well, gnux32 (Tier 2) is x86_64, with 32-bit pointers, and theory has support.
But in practice, LLVM is shaky here, too; see #59674.

[whitequark]

This is not realistic given the way LLVM is designed.

Revisiting: this statement is wrong. There is a way to ask LLVM to emit this information because it gained exactly such a pass plugged very late into MC pipeline. See this.

[cuviper]

Right now we implement X86 probes with a call to compiler-builtins' __rust_probestack, but there is support for inline stack-probes coming in LLVM 11:
https://reviews.llvm.org/rGe67cbac81211d40332a79d98c9d5953624cc1202

That's also X86-only so far, but this should be the path forward for other targets.

[bstrie]

Does #77885 mean that stack probes are now enabled on all LLVM targets? If so, should this be closed?

[cuviper]

Not yet -- I would like to test additional architectures natively before enabling them, because we did uncover bugs in X86 codegen. Also, PowerPC and SystemZ (s390x) are the only other targets that have implemented stack probes at all.

[cuviper]

As of LLVM 18, "probe-stack"="inline-asm" is supported by AArch64, PowerPC, SystemZ, and X86, and we are using that (StackProbeType::Inline) for almost all related target triples. The exceptions are Windows and UEFI targets that use __chkstk calls, and Fortanix, Illumos, and AIX that might just be missing this on accident.