[VE] udiv / urem crashes backend with assertion LHS.getValueType().isVector() == VT.isVector() && "Cannot compare scalars to vectors" failed

[HazyFish]

Description

When targeting ve, using udiv or urem with the last operand being a constant vector with all 1's (examples listed below) crashes backend with assertion LHS.getValueType().isVector() == VT.isVector() && "Cannot compare scalars to vectors" failed during pass VE DAG->DAG Pattern Instruction Selection.

Here are some vector operands that crash the backend:

v4i1: <i1 true, i1 true, i1 true, i1 true>
v4i8: <i8 255, i8 255, i8 255, i8 255>
v4i16: <i16 65535, i16 65535, i16 65535, i16 65535>

The problem doesn't exist when using sdiv or srem.
The problem doens't exist when targeting x86_64, aarch64, or riscv64.

Minimal Reproduction

https://godbolt.org/z/4Kr9xMEb8

Code

define <4 x i8> @f(<4 x i8> %0) {
BB:
  %B = udiv <4 x i8> %0, <i8 255, i8 255, i8 255, i8 255>
  ret <4 x i8> %B
}

Stack Trace

llc: /home/henry/aflplusplus-isel/llvm-project/llvm/include/llvm/CodeGen/SelectionDAG.h:1124: llvm::SDValue llvm::SelectionDAG::getSetCC(const llvm::SDLoc &, llvm::EVT, llvm::SDValue, llvm::SDValue, ISD::CondCode, llvm::SDValue, bool): Assertion `LHS.getValueType().isVector() == VT.isVector() && "Cannot compare scalars to vectors"' failed.
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.	Program arguments: ./llvm-project/build-debug/bin/llc -mtriple=ve ./crash-reports/dagisel-ve/1.ll
1.	Running pass 'Function Pass Manager' on module './crash-reports/dagisel-ve/1.ll'.
2.	Running pass 'VE DAG->DAG Pattern Instruction Selection' on function '@f'
 #0 0x00000000047ada8a llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /home/henry/aflplusplus-isel/llvm-project/llvm/lib/Support/Unix/Signals.inc:569:11
 #1 0x00000000047adc3b PrintStackTraceSignalHandler(void*) /home/henry/aflplusplus-isel/llvm-project/llvm/lib/Support/Unix/Signals.inc:636:1
 #2 0x00000000047ac286 llvm::sys::RunSignalHandlers() /home/henry/aflplusplus-isel/llvm-project/llvm/lib/Support/Signals.cpp:104:5
 #3 0x00000000047ae365 SignalHandler(int) /home/henry/aflplusplus-isel/llvm-project/llvm/lib/Support/Unix/Signals.inc:407:1
 #4 0x00007f66fcb92980 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x12980)
 #5 0x00007f66fba82e87 raise /build/glibc-CVJwZb/glibc-2.27/signal/../sysdeps/unix/sysv/linux/raise.c:51:0
 #6 0x00007f66fba847f1 abort /build/glibc-CVJwZb/glibc-2.27/stdlib/abort.c:81:0
 #7 0x00007f66fba743fa __assert_fail_base /build/glibc-CVJwZb/glibc-2.27/assert/assert.c:89:0
 #8 0x00007f66fba74472 (/lib/x86_64-linux-gnu/libc.so.6+0x30472)
 #9 0x000000000127cb1a llvm::SelectionDAG::getSetCC(llvm::SDLoc const&, llvm::EVT, llvm::SDValue, llvm::SDValue, llvm::ISD::CondCode, llvm::SDValue, bool) /home/henry/aflplusplus-isel/llvm-project/llvm/include/llvm/CodeGen/SelectionDAG.h:1125:5
#10 0x00000000042c87fb (anonymous namespace)::DAGCombiner::visitUDIV(llvm::SDNode*) /home/henry/aflplusplus-isel/llvm-project/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp:4481:38
#11 0x00000000042b9bf6 (anonymous namespace)::DAGCombiner::visit(llvm::SDNode*) /home/henry/aflplusplus-isel/llvm-project/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp:1705:40
#12 0x00000000042b93a8 (anonymous namespace)::DAGCombiner::combine(llvm::SDNode*) /home/henry/aflplusplus-isel/llvm-project/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp:1828:10
#13 0x00000000042b89b6 (anonymous namespace)::DAGCombiner::Run(llvm::CombineLevel) /home/henry/aflplusplus-isel/llvm-project/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp:1626:18
#14 0x00000000042b829f llvm::SelectionDAG::Combine(llvm::CombineLevel, llvm::AAResults*, llvm::CodeGenOpt::Level) /home/henry/aflplusplus-isel/llvm-project/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp:25478:3
#15 0x000000000451175f llvm::SelectionDAGISel::CodeGenAndEmitDAG() /home/henry/aflplusplus-isel/llvm-project/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:772:3
#16 0x000000000451130d llvm::SelectionDAGISel::SelectBasicBlock(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::Instruction, true, false, void>, false, true>, llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::Instruction, true, false, void>, false, true>, bool&) /home/henry/aflplusplus-isel/llvm-project/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:688:1
#17 0x0000000004510dab llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&) /home/henry/aflplusplus-isel/llvm-project/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:1603:11
#18 0x000000000450e356 llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&) /home/henry/aflplusplus-isel/llvm-project/llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:467:3
#19 0x0000000002465705 (anonymous namespace)::VEDAGToDAGISel::runOnMachineFunction(llvm::MachineFunction&) /home/henry/aflplusplus-isel/llvm-project/llvm/lib/Target/VE/VEISelDAGToDAG.cpp:38:5
#20 0x00000000035783e5 llvm::MachineFunctionPass::runOnFunction(llvm::Function&) /home/henry/aflplusplus-isel/llvm-project/llvm/lib/CodeGen/MachineFunctionPass.cpp:91:8
#21 0x0000000003c55c66 llvm::FPPassManager::runOnFunction(llvm::Function&) /home/henry/aflplusplus-isel/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1430:23
#22 0x0000000003c5aa92 llvm::FPPassManager::runOnModule(llvm::Module&) /home/henry/aflplusplus-isel/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1476:16
#23 0x0000000003c56539 (anonymous namespace)::MPPassManager::runOnModule(llvm::Module&) /home/henry/aflplusplus-isel/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1545:23
#24 0x0000000003c560ad llvm::legacy::PassManagerImpl::run(llvm::Module&) /home/henry/aflplusplus-isel/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:535:16
#25 0x0000000003c5ad71 llvm::legacy::PassManager::run(llvm::Module&) /home/henry/aflplusplus-isel/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1672:3
#26 0x0000000000d38a5c compileModule(char**, llvm::LLVMContext&) /home/henry/aflplusplus-isel/llvm-project/llvm/tools/llc/llc.cpp:737:41
#27 0x0000000000d36e02 main /home/henry/aflplusplus-isel/llvm-project/llvm/tools/llc/llc.cpp:418:13
#28 0x00007f66fba65c87 __libc_start_main /build/glibc-CVJwZb/glibc-2.27/csu/../csu/libc-start.c:344:0
#29 0x0000000000d3660a _start (./llvm-project/build-debug/bin/llc+0xd3660a)

[rotateright]

We have a truncating build vector which doesn't seem like an issue by itself.
The getSetCCResultType() call is returning a scalar type for the vector type input though. That is unexpected.

[rotateright]

This seems very wrong:

EVT VETargetLowering::getSetCCResultType(const DataLayout &, LLVMContext &,
                                         EVT VT) const {
  return MVT::i32;
}

...but I guess we have to guard against this kind of thing in the generic code.

[rotateright]

cc @kaz7 @simoll based on patch history.
I've prevented the crashing, but I'm not sure what is expected on examples like these.

[rotateright]

Closing since the crash is solved. Please re-open or file another bug if there's more to do.