Inlining an always_inline call site leads to a crash in the backend

[kazutakahirata]

Consider:

target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-unknown-linux-gnu"

define void @caller(ptr %0) {
  call void @callee(ptr %0, i64 0, i32 0) #1
  ret void
}

define available_externally void @callee(ptr %0, i64 %1, i32 %2) #0 {
  %4 = call <16 x float> @llvm.x86.avx512.min.ps.512(<16 x float> zeroinitializer, <16 x float> zeroinitializer, i32 0)
  store <16 x float> %4, ptr %0, align 1
  ret void
}

declare <16 x float> @llvm.x86.avx512.min.ps.512(<16 x float>, <16 x float>, i32 immarg)

attributes #0 = { "target-features"="+aes,+avx,+avx2,+avx512bw,+avx512dq,+avx512f,+avx512vl,+bmi,+bmi2,+crc32,+cx16,+cx8,+f16c,+fma,+fsgsbase,+fxsr,+invpcid,+lzcnt,+mmx,+movbe,+pclmul,+popcnt,+rdrnd,+sahf,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+ssse3,+x87,+xsave,+xsaveopt" }
attributes #1 = { alwaysinline }

Compile like so:

$ clang -O3 -S bug.ll

I get:

SplitVectorResult #0: t7: v16f32 = llvm.x86.avx512.min.ps.512 TargetConstant:i64<10845>, t5, t5, TargetConstant:i32<0>

fatal error: error in backend: Do not know how to split the result of this operator!

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
:
:

[kazutakahirata]

The root cause is that callee in the input IR is asking the compiler to always inline itself via the alwaysinline call site attribute even though caller is not allowed to use the AVX512 instruction sets.

Then llvm::getAttributeBasedInliningDecision in InlineCost.cpp looks at the alwaysinline attribute and decides to inline callee.

Note that in llvm::getAttributeBasedInliningDecision the alwaysinline attribute takes precedence over the checks performed by functionsHaveCompatibleAttributes. Here are the excerpt:

  // Calls to functions with always-inline attributes should be inlined
  // whenever possible.
  if (Call.hasFnAttr(Attribute::AlwaysInline)) {
    :
    :
    :
  }

  // Never inline functions with conflicting attributes (unless callee has
  // always-inline attribute).
  Function *Caller = Call.getCaller();
  if (!functionsHaveCompatibleAttributes(Caller, Callee, CalleeTTI, GetTLI))
    return InlineResult::failure("conflicting attributes");

FWIW, the two functions in the input IR above comes from two different modules and merged into one module by ThinLTO.

caller is declared with __attribute__((flatten)), which puts alwaysinline at call sites.

One possible fix is to call TTI.areInlineCompatible(Caller, Callee)) before we decide to inline on the basis of Attribute::AlwaysInline. That is, we could have something like:

  if (!TTI.areInlineCompatible(Caller, Callee))
    return InlineResult::failure("conflicting target features");

  // Calls to functions with always-inline attributes should be inlined
  // whenever possible.
  if (Call.hasFnAttr(Attribute::AlwaysInline)) {
    :
    :
  }

  // Never inline functions with conflicting attributes (unless callee has
  // always-inline attribute).
  Function *Caller = Call.getCaller();
  if (!functionsHaveCompatibleAttributes(Caller, Callee, CalleeTTI, GetTLI))
    return InlineResult::failure("conflicting attributes");

That is, we are essentially peeling one of the checks from functionsHaveCompatibleAttributes and doing it before handling Attribute::AlwaysInline.

[kazutakahirata]

I've uploaded: https://reviews.llvm.org/D150396