[TableGen] Add MacroFusions to SchedMachineModel #72223

wangpc-pp · 2023-11-14T08:12:22Z

A list of MacroFusion is added to SchedMachineModel to indicate
supported macro fusions of this processor model.

For visibility, C++ enums of all MacroFusion are generated and new
hasMacroFusion is added to Subtarget to test if a MacroFusion is
supported.

This PR is stacked on #72219, #72222

llvmbot · 2023-11-14T08:12:50Z

@llvm/pr-subscribers-backend-aarch64
@llvm/pr-subscribers-backend-risc-v
@llvm/pr-subscribers-mc
@llvm/pr-subscribers-backend-x86

@llvm/pr-subscribers-backend-arm

Author: Wang Pengcheng (wangpc-pp)

Changes

A list of MacroFusion is added to SchedMachineModel to indicate
supported macro fusions of this processor model.

For visibility, C++ enums of all MacroFusion are generated and new
hasMacroFusion is added to Subtarget to test if a MacroFusion is
supported.

This PR is stacked on #72219, #72222

Patch is 34.86 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/72223.diff

22 Files Affected:

(modified) llvm/include/llvm/CodeGen/MacroFusion.h (+9-9)
(modified) llvm/include/llvm/CodeGen/TargetSubtargetInfo.h (+4)
(modified) llvm/include/llvm/MC/MCSchedule.h (+9)
(modified) llvm/include/llvm/MC/MCSubtargetInfo.h (+13)
(modified) llvm/include/llvm/Target/TargetInstrPredicate.td (+6)
(modified) llvm/include/llvm/Target/TargetSchedule.td (+65)
(modified) llvm/lib/CodeGen/MacroFusion.cpp (+27-12)
(modified) llvm/lib/MC/MCSchedule.cpp (+1)
(modified) llvm/lib/Target/AArch64/AArch64MacroFusion.cpp (+1-1)
(modified) llvm/lib/Target/AMDGPU/AMDGPUMacroFusion.cpp (+2-2)
(modified) llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp (+2-2)
(modified) llvm/lib/Target/ARM/ARMMacroFusion.cpp (+2-2)
(modified) llvm/lib/Target/PowerPC/PPCMacroFusion.cpp (+2-2)
(modified) llvm/lib/Target/RISCV/RISCVMacroFusion.cpp (+1-1)
(modified) llvm/lib/Target/X86/X86MacroFusion.cpp (+2-3)
(modified) llvm/utils/TableGen/CMakeLists.txt (+1)
(modified) llvm/utils/TableGen/CodeGenSchedule.cpp (+16)
(modified) llvm/utils/TableGen/CodeGenSchedule.h (+11)
(added) llvm/utils/TableGen/MacroFusionPredicatorEmitter.cpp (+208)
(modified) llvm/utils/TableGen/PredicateExpander.cpp (+8)
(modified) llvm/utils/TableGen/PredicateExpander.h (+1)
(modified) llvm/utils/TableGen/SubtargetEmitter.cpp (+46-1)

diff --git a/llvm/include/llvm/CodeGen/MacroFusion.h b/llvm/include/llvm/CodeGen/MacroFusion.h
index ea2c7a5faae385a..a97f776335368c7 100644
--- a/llvm/include/llvm/CodeGen/MacroFusion.h
+++ b/llvm/include/llvm/CodeGen/MacroFusion.h
@@ -14,8 +14,8 @@
 #ifndef LLVM_CODEGEN_MACROFUSION_H
 #define LLVM_CODEGEN_MACROFUSION_H
 
-#include <functional>
 #include <memory>
+#include <vector>
 
 namespace llvm {
 
@@ -29,10 +29,10 @@ class SUnit;
 /// Check if the instr pair, FirstMI and SecondMI, should be fused
 /// together. Given SecondMI, when FirstMI is unspecified, then check if
 /// SecondMI may be part of a fused pair at all.
-using ShouldSchedulePredTy = std::function<bool(const TargetInstrInfo &TII,
-                                                const TargetSubtargetInfo &TSI,
-                                                const MachineInstr *FirstMI,
-                                                const MachineInstr &SecondMI)>;
+using MacroFusionPredTy = bool (*)(const TargetInstrInfo &TII,
+                                   const TargetSubtargetInfo &STI,
+                                   const MachineInstr *FirstMI,
+                                   const MachineInstr &SecondMI);
 
 /// Checks if the number of cluster edges between SU and its predecessors is
 /// less than FuseLimit
@@ -48,15 +48,15 @@ bool fuseInstructionPair(ScheduleDAGInstrs &DAG, SUnit &FirstSU,
 
 /// Create a DAG scheduling mutation to pair instructions back to back
 /// for instructions that benefit according to the target-specific
-/// shouldScheduleAdjacent predicate function.
+/// predicate functions.
 std::unique_ptr<ScheduleDAGMutation>
-createMacroFusionDAGMutation(ShouldSchedulePredTy shouldScheduleAdjacent);
+createMacroFusionDAGMutation(std::vector<MacroFusionPredTy> Predicates);
 
 /// Create a DAG scheduling mutation to pair branch instructions with one
 /// of their predecessors back to back for instructions that benefit according
-/// to the target-specific shouldScheduleAdjacent predicate function.
+/// to the target-specific predicate functions.
 std::unique_ptr<ScheduleDAGMutation>
-createBranchMacroFusionDAGMutation(ShouldSchedulePredTy shouldScheduleAdjacent);
+createBranchMacroFusionDAGMutation(std::vector<MacroFusionPredTy> Predicates);
 
 } // end namespace llvm
 
diff --git a/llvm/include/llvm/CodeGen/TargetSubtargetInfo.h b/llvm/include/llvm/CodeGen/TargetSubtargetInfo.h
index 55ef95c28543190..7c76293f3e5eaea 100644
--- a/llvm/include/llvm/CodeGen/TargetSubtargetInfo.h
+++ b/llvm/include/llvm/CodeGen/TargetSubtargetInfo.h
@@ -16,6 +16,7 @@
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/CodeGen/MacroFusion.h"
 #include "llvm/CodeGen/PBQPRAConstraint.h"
 #include "llvm/CodeGen/SchedulerRegistry.h"
 #include "llvm/IR/GlobalValue.h"
@@ -323,6 +324,9 @@ class TargetSubtargetInfo : public MCSubtargetInfo {
   /// helps removing redundant copies generated by register allocator when
   /// handling complex eviction chains.
   virtual bool enableSpillageCopyElimination() const { return false; }
+
+  /// Get the list of MacroFusion predicates.
+  virtual std::vector<MacroFusionPredTy> getMacroFusions() const { return {}; }
 };
 
 } // end namespace llvm
diff --git a/llvm/include/llvm/MC/MCSchedule.h b/llvm/include/llvm/MC/MCSchedule.h
index 98ebe42cfd133b5..aa187e5cb400672 100644
--- a/llvm/include/llvm/MC/MCSchedule.h
+++ b/llvm/include/llvm/MC/MCSchedule.h
@@ -14,6 +14,7 @@
 #ifndef LLVM_MC_MCSCHEDULE_H
 #define LLVM_MC_MCSCHEDULE_H
 
+#include "llvm/ADT/Bitset.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/Support/DataTypes.h"
 #include <cassert>
@@ -196,6 +197,9 @@ struct MCExtraProcessorInfo {
   unsigned StoreQueueID;
 };
 
+const unsigned MaxMacroFusions = 256;
+using MacroFusionBitset = Bitset<MaxMacroFusions>;
+
 /// Machine model for scheduling, bundling, and heuristics.
 ///
 /// The machine model directly provides basic information about the
@@ -325,9 +329,14 @@ struct MCSchedModel {
   const InstrItinerary *InstrItineraries;
 
   const MCExtraProcessorInfo *ExtraProcessorInfo;
+  const MacroFusionBitset *MacroFusionBits;
 
   bool hasExtraProcessorInfo() const { return ExtraProcessorInfo; }
 
+  const MacroFusionBitset *getMacroFusionBits() const {
+    return MacroFusionBits;
+  }
+
   unsigned getProcessorID() const { return ProcID; }
 
   /// Does this machine model include instruction-level scheduling.
diff --git a/llvm/include/llvm/MC/MCSubtargetInfo.h b/llvm/include/llvm/MC/MCSubtargetInfo.h
index f172a799aa3331c..66fb6c9383272e1 100644
--- a/llvm/include/llvm/MC/MCSubtargetInfo.h
+++ b/llvm/include/llvm/MC/MCSubtargetInfo.h
@@ -120,6 +120,12 @@ class MCSubtargetInfo {
     return FeatureBits[Feature];
   }
 
+  bool hasMacroFusion(unsigned MacroFusion) const {
+    const MacroFusionBitset *MacroFusionBits =
+        CPUSchedModel->getMacroFusionBits();
+    return MacroFusionBits && MacroFusionBits->test(MacroFusion);
+  }
+
 protected:
   /// Initialize the scheduling model and feature bits.
   ///
@@ -295,6 +301,13 @@ class MCSubtargetInfo {
 
   /// \return if target want to issue a prefetch in address space \p AS.
   virtual bool shouldPrefetchAddressSpace(unsigned AS) const;
+
+  /// Enable macro fusion for this subtarget.
+  virtual bool enableMacroFusion() const {
+    const MacroFusionBitset *MacroFusionBits =
+        CPUSchedModel->getMacroFusionBits();
+    return MacroFusionBits && MacroFusionBits->any();
+  }
 };
 
 } // end namespace llvm
diff --git a/llvm/include/llvm/Target/TargetInstrPredicate.td b/llvm/include/llvm/Target/TargetInstrPredicate.td
index 9f2cde9d923050a..f7e6a390045b520 100644
--- a/llvm/include/llvm/Target/TargetInstrPredicate.td
+++ b/llvm/include/llvm/Target/TargetInstrPredicate.td
@@ -95,6 +95,12 @@ class MCOperandPredicate<int Index> : MCInstPredicate {
 // Return true if machine operand at position `Index` is a register operand.
 class CheckIsRegOperand<int Index> : MCOperandPredicate<Index>;
 
+// Return true if machine operand at position `Index` is a virtual register operand.
+class CheckIsVRegOperand<int Index> : MCOperandPredicate<Index>;
+
+// Return true if machine operand at position `Index` is not a virtual register operand.
+class CheckIsNotVirtualRegOperand<int Index> : CheckNot<CheckIsVRegOperand<Index>>;
+
 // Return true if machine operand at position `Index` is an immediate operand.
 class CheckIsImmOperand<int Index> : MCOperandPredicate<Index>;
 
diff --git a/llvm/include/llvm/Target/TargetSchedule.td b/llvm/include/llvm/Target/TargetSchedule.td
index 949baa5d2105c45..ed07010dfb2d39f 100644
--- a/llvm/include/llvm/Target/TargetSchedule.td
+++ b/llvm/include/llvm/Target/TargetSchedule.td
@@ -53,6 +53,7 @@
 include "llvm/Target/TargetItinerary.td"
 
 class Predicate; // Forward def
+class MacroFusion;
 
 // DAG operator that interprets the DAG args as Instruction defs.
 def instrs;
@@ -122,6 +123,9 @@ class SchedMachineModel {
   // using intervals via ResourceSegments (see
   // llvm/include/llvm/CodeGen/MachineScheduler.h).
   bit EnableIntervals = false;
+
+  // List of MacroFusion.
+  list<MacroFusion> MacroFusions = [];
 }
 
 def NoSchedModel : SchedMachineModel {
@@ -469,6 +473,67 @@ class SchedAlias<SchedReadWrite match, SchedReadWrite alias> {
   SchedMachineModel SchedModel = ?;
 }
 
+// Base class of MacroFusionPredicate, etc. The avaliable variables are:
+// * const TargetInstrInfo &TII
+// * const TargetSubtargetInfo &STI
+// * const MachineRegisterInfo &MRI
+// * const MachineInstr *FirstMI
+// * const MachineInstr &SecondMI
+class MacroFusionPredicateBase;
+
+// MacroFusionPredicate with raw code predicate.
+class MacroFusionPredicate<code pred> : MacroFusionPredicateBase {
+  code Predicate = pred;
+}
+
+// Binds firstOpIdx and secondOpIdx. The operand of `FirstMI` at position
+// `firstOpIdx` should be the same as the operand of `SenondMI` at position
+// `secondOpIdx`.
+class BindReg<int firstOpIdx, int secondOpIdx> : MacroFusionPredicateBase {
+  int FirstOpIdx = firstOpIdx;
+  int SecondOpIdx = secondOpIdx;
+}
+
+// A predicate for wildcard. The generated code will be like:
+// ```
+// if (!FirstMI)
+//   return ReturnValue;
+// ```
+class WildcardPred<bit ret> : MacroFusionPredicateBase {
+  bit ReturnValue = ret;
+}
+def WildcardFalse : WildcardPred<0>;
+def WildcardTrue : WildcardPred<1>;
+
+// Indicates that the destination register of `FirstMI` should be have one
+// use if it is an virtual register.
+class OneUsePred : MacroFusionPredicateBase;
+def OneUse : OneUsePred;
+
+// Handled by MacroFusionPredicatorEmitter backend.
+// The generated predicator will be like:
+// ```
+// bool isNAME(const TargetInstrInfo &TII,
+//             const TargetSubtargetInfo &STI,
+//             const MachineInstr *FirstMI,
+//             const MachineInstr &SecondMI) {
+//   auto &MRI = SecondMI.getMF()->getRegInfo();
+//   /* Prolog */
+//   /* Predicate for `FirstMI` */
+//   /* Predicate for `SecondMI` */
+//   /* Epilog */
+//   return true;
+// }
+// ```
+class MacroFusion<MCInstPredicate first, MCInstPredicate second,
+                  list<MacroFusionPredicateBase> prolog = [],
+                  list<MacroFusionPredicateBase> epilog = []> {
+  MCInstPredicate First = first;
+  MCInstPredicate Second = second;
+  list<MacroFusionPredicateBase> Prolog = prolog;
+  list<MacroFusionPredicateBase> Epilog = epilog;
+}
+
 // Allow the definition of processor register files for register renaming
 // purposes.
 //
diff --git a/llvm/lib/CodeGen/MacroFusion.cpp b/llvm/lib/CodeGen/MacroFusion.cpp
index fa5df68b8abcc0f..63ea53f9cc32f97 100644
--- a/llvm/lib/CodeGen/MacroFusion.cpp
+++ b/llvm/lib/CodeGen/MacroFusion.cpp
@@ -137,19 +137,35 @@ namespace {
 /// Post-process the DAG to create cluster edges between instrs that may
 /// be fused by the processor into a single operation.
 class MacroFusion : public ScheduleDAGMutation {
-  ShouldSchedulePredTy shouldScheduleAdjacent;
+  std::vector<MacroFusionPredTy> Predicates;
   bool FuseBlock;
   bool scheduleAdjacentImpl(ScheduleDAGInstrs &DAG, SUnit &AnchorSU);
 
 public:
-  MacroFusion(ShouldSchedulePredTy shouldScheduleAdjacent, bool FuseBlock)
-    : shouldScheduleAdjacent(shouldScheduleAdjacent), FuseBlock(FuseBlock) {}
+  MacroFusion(std::vector<MacroFusionPredTy> Predicates, bool FuseBlock)
+      : Predicates(Predicates), FuseBlock(FuseBlock) {}
 
   void apply(ScheduleDAGInstrs *DAGInstrs) override;
+
+  bool shouldScheduleAdjacent(const TargetInstrInfo &TII,
+                              const TargetSubtargetInfo &STI,
+                              const MachineInstr *FirstMI,
+                              const MachineInstr &SecondMI);
 };
 
 } // end anonymous namespace
 
+bool MacroFusion::shouldScheduleAdjacent(const TargetInstrInfo &TII,
+                                         const TargetSubtargetInfo &STI,
+                                         const MachineInstr *FirstMI,
+                                         const MachineInstr &SecondMI) {
+  for (MacroFusionPredTy Predicate : Predicates) {
+    if (Predicate(TII, STI, FirstMI, SecondMI))
+      return true;
+  }
+  return false;
+}
+
 void MacroFusion::apply(ScheduleDAGInstrs *DAG) {
   if (FuseBlock)
     // For each of the SUnits in the scheduling block, try to fuse the instr in
@@ -197,17 +213,16 @@ bool MacroFusion::scheduleAdjacentImpl(ScheduleDAGInstrs &DAG, SUnit &AnchorSU)
 }
 
 std::unique_ptr<ScheduleDAGMutation>
-llvm::createMacroFusionDAGMutation(
-     ShouldSchedulePredTy shouldScheduleAdjacent) {
-  if(EnableMacroFusion)
-    return std::make_unique<MacroFusion>(shouldScheduleAdjacent, true);
+llvm::createMacroFusionDAGMutation(std::vector<MacroFusionPredTy> Predicates) {
+  if (EnableMacroFusion) {
+    return std::make_unique<MacroFusion>(Predicates, true);
+  }
   return nullptr;
 }
 
-std::unique_ptr<ScheduleDAGMutation>
-llvm::createBranchMacroFusionDAGMutation(
-     ShouldSchedulePredTy shouldScheduleAdjacent) {
-  if(EnableMacroFusion)
-    return std::make_unique<MacroFusion>(shouldScheduleAdjacent, false);
+std::unique_ptr<ScheduleDAGMutation> llvm::createBranchMacroFusionDAGMutation(
+    std::vector<MacroFusionPredTy> Predicates) {
+  if (EnableMacroFusion)
+    return std::make_unique<MacroFusion>(Predicates, false);
   return nullptr;
 }
diff --git a/llvm/lib/MC/MCSchedule.cpp b/llvm/lib/MC/MCSchedule.cpp
index 990a693559a7776..19c36cb0e58d9c1 100644
--- a/llvm/lib/MC/MCSchedule.cpp
+++ b/llvm/lib/MC/MCSchedule.cpp
@@ -37,6 +37,7 @@ const MCSchedModel MCSchedModel::Default = {DefaultIssueWidth,
                                             0,
                                             0,
                                             nullptr,
+                                            nullptr,
                                             nullptr};
 
 int MCSchedModel::computeInstrLatency(const MCSubtargetInfo &STI,
diff --git a/llvm/lib/Target/AArch64/AArch64MacroFusion.cpp b/llvm/lib/Target/AArch64/AArch64MacroFusion.cpp
index 05d60872bf51aca..8f46f3eabb3ef45 100644
--- a/llvm/lib/Target/AArch64/AArch64MacroFusion.cpp
+++ b/llvm/lib/Target/AArch64/AArch64MacroFusion.cpp
@@ -478,5 +478,5 @@ static bool shouldScheduleAdjacent(const TargetInstrInfo &TII,
 
 std::unique_ptr<ScheduleDAGMutation>
 llvm::createAArch64MacroFusionDAGMutation() {
-  return createMacroFusionDAGMutation(shouldScheduleAdjacent);
+  return createMacroFusionDAGMutation({shouldScheduleAdjacent});
 }
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUMacroFusion.cpp b/llvm/lib/Target/AMDGPU/AMDGPUMacroFusion.cpp
index c15c94ee17f8b1d..b2b11d661523e9c 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUMacroFusion.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUMacroFusion.cpp
@@ -59,8 +59,8 @@ static bool shouldScheduleAdjacent(const TargetInstrInfo &TII_,
 
 namespace llvm {
 
-std::unique_ptr<ScheduleDAGMutation> createAMDGPUMacroFusionDAGMutation () {
-  return createMacroFusionDAGMutation(shouldScheduleAdjacent);
+std::unique_ptr<ScheduleDAGMutation> createAMDGPUMacroFusionDAGMutation() {
+  return createMacroFusionDAGMutation({shouldScheduleAdjacent});
 }
 
 } // end namespace llvm
diff --git a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
index 29c9b9ccf27614f..0bddeeef9e9b1a3 100644
--- a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
@@ -142,10 +142,10 @@ namespace {
 /// be turned into VOPD instructions
 /// Greedily pairs instruction candidates. O(n^2) algorithm.
 struct VOPDPairingMutation : ScheduleDAGMutation {
-  ShouldSchedulePredTy shouldScheduleAdjacent; // NOLINT: function pointer
+  MacroFusionPredTy shouldScheduleAdjacent; // NOLINT: function pointer
 
   VOPDPairingMutation(
-      ShouldSchedulePredTy shouldScheduleAdjacent) // NOLINT: function pointer
+      MacroFusionPredTy shouldScheduleAdjacent) // NOLINT: function pointer
       : shouldScheduleAdjacent(shouldScheduleAdjacent) {}
 
   void apply(ScheduleDAGInstrs *DAG) override {
diff --git a/llvm/lib/Target/ARM/ARMMacroFusion.cpp b/llvm/lib/Target/ARM/ARMMacroFusion.cpp
index 38bf28ba8219b90..7de117925e464fe 100644
--- a/llvm/lib/Target/ARM/ARMMacroFusion.cpp
+++ b/llvm/lib/Target/ARM/ARMMacroFusion.cpp
@@ -62,8 +62,8 @@ static bool shouldScheduleAdjacent(const TargetInstrInfo &TII,
   return false;
 }
 
-std::unique_ptr<ScheduleDAGMutation> createARMMacroFusionDAGMutation () {
-  return createMacroFusionDAGMutation(shouldScheduleAdjacent);
+std::unique_ptr<ScheduleDAGMutation> createARMMacroFusionDAGMutation() {
+  return createMacroFusionDAGMutation({shouldScheduleAdjacent});
 }
 
 } // end namespace llvm
diff --git a/llvm/lib/Target/PowerPC/PPCMacroFusion.cpp b/llvm/lib/Target/PowerPC/PPCMacroFusion.cpp
index bf1c39a3a3a2d47..d6a4a5dd5faabae 100644
--- a/llvm/lib/Target/PowerPC/PPCMacroFusion.cpp
+++ b/llvm/lib/Target/PowerPC/PPCMacroFusion.cpp
@@ -286,8 +286,8 @@ static bool shouldScheduleAdjacent(const TargetInstrInfo &TII,
 
 namespace llvm {
 
-std::unique_ptr<ScheduleDAGMutation> createPowerPCMacroFusionDAGMutation () {
-  return createMacroFusionDAGMutation(shouldScheduleAdjacent);
+std::unique_ptr<ScheduleDAGMutation> createPowerPCMacroFusionDAGMutation() {
+  return createMacroFusionDAGMutation({shouldScheduleAdjacent});
 }
 
 } // end namespace llvm
diff --git a/llvm/lib/Target/RISCV/RISCVMacroFusion.cpp b/llvm/lib/Target/RISCV/RISCVMacroFusion.cpp
index 02a8d5c18fe1a0e..1b82cc8b5b858f4 100644
--- a/llvm/lib/Target/RISCV/RISCVMacroFusion.cpp
+++ b/llvm/lib/Target/RISCV/RISCVMacroFusion.cpp
@@ -65,5 +65,5 @@ static bool shouldScheduleAdjacent(const TargetInstrInfo &TII,
 }
 
 std::unique_ptr<ScheduleDAGMutation> llvm::createRISCVMacroFusionDAGMutation() {
-  return createMacroFusionDAGMutation(shouldScheduleAdjacent);
+  return createMacroFusionDAGMutation({shouldScheduleAdjacent});
 }
diff --git a/llvm/lib/Target/X86/X86MacroFusion.cpp b/llvm/lib/Target/X86/X86MacroFusion.cpp
index aa6e8645e0927f5..382cc9a71c2a184 100644
--- a/llvm/lib/Target/X86/X86MacroFusion.cpp
+++ b/llvm/lib/Target/X86/X86MacroFusion.cpp
@@ -67,9 +67,8 @@ static bool shouldScheduleAdjacent(const TargetInstrInfo &TII,
 
 namespace llvm {
 
-std::unique_ptr<ScheduleDAGMutation>
-createX86MacroFusionDAGMutation () {
-  return createBranchMacroFusionDAGMutation(shouldScheduleAdjacent);
+std::unique_ptr<ScheduleDAGMutation> createX86MacroFusionDAGMutation() {
+  return createBranchMacroFusionDAGMutation({shouldScheduleAdjacent});
 }
 
 } // end namespace llvm
diff --git a/llvm/utils/TableGen/CMakeLists.txt b/llvm/utils/TableGen/CMakeLists.txt
index 071ea3bc07054bb..f765cc36d3bebed 100644
--- a/llvm/utils/TableGen/CMakeLists.txt
+++ b/llvm/utils/TableGen/CMakeLists.txt
@@ -72,6 +72,7 @@ add_tablegen(llvm-tblgen LLVM
   PredicateExpander.cpp
   PseudoLoweringEmitter.cpp
   CompressInstEmitter.cpp
+  MacroFusionPredicatorEmitter.cpp
   RegisterBankEmitter.cpp
   RegisterInfoEmitter.cpp
   SearchableTableEmitter.cpp
diff --git a/llvm/utils/TableGen/CodeGenSchedule.cpp b/llvm/utils/TableGen/CodeGenSchedule.cpp
index c3c5e4f8eb2d8c3..98bf245697b77db 100644
--- a/llvm/utils/TableGen/CodeGenSchedule.cpp
+++ b/llvm/utils/TableGen/CodeGenSchedule.cpp
@@ -208,6 +208,9 @@ CodeGenSchedModels::CodeGenSchedModels(RecordKeeper &RK,
   // (For per-operand resources mapped to itinerary classes).
   collectProcUnsupportedFeatures();
 
+  // Find MacroFusion records for each processor.
+  collectMacroFusions();
+
   // Infer new SchedClasses from SchedVariant.
   inferSchedClasses();
 
@@ -1150,6 +1153,13 @@ bool CodeGenSchedModels::hasItineraries() const {
   return false;
 }
 
+bool CodeGenSchedModels::hasMacroFusions() const {
+  for (const CodeGenProcModel &PM : ProcModels)
+    if (PM.hasMacroFusions())
+      return true;
+  return false;
+}
+
 // Gather the processor itineraries.
 void CodeGenSchedModels::collectProcItins() {
   LLVM_DEBUG(dbgs() << "\n+++ PROBLEM ITINERARIES (collectProcItins) +++\n");
@@ -1220,6 +1230,12 @@ void CodeGenSchedModels::collectProcUnsupportedFeatures() {
         ProcModel.ModelDef->getValueAsListOfDefs("UnsupportedFeatures"));
 }
 
+void CodeGenSchedModels::collectMacroFusions() {
+  for (CodeGenProcModel &ProcModel : ProcModels)
+    append_range(ProcModel.MacroFusions,
+                 ProcModel.ModelDef->getValueAsListOfDefs("MacroFusions"));
+}
+
 /// Infer new classes from existing classes. In the process, this may create new
 /// SchedWrites from sequences of existing SchedWrites.
 void CodeGenSchedModels::inferSchedClasses() {
diff --git a/llvm/utils/TableGen/CodeGenSchedule.h b/llvm/utils/TableGen/CodeGenSchedule.h
index 76ef1e43953078e..317558c52fbef15 100644
--- a/llvm/utils/TableGen/CodeGenSchedule.h
+++ b/llvm/utils/TableGen/CodeGenSchedule.h
@@ -238,6 +238,10 @@ struct CodeGenProcModel {
   // This list is empty if the Processor has no UnsupportedFeatures.
   RecVec UnsupportedFeaturesDefs;
 
+  // List of MacroFusion.
+  // This list is empty if the Processor has no MacroFusion.
+  RecVec MacroFusions;
+
   // All read/write resources associated with this processor.
   RecVec WriteResDefs;
   RecVec ReadAdvanceDefs;
@@ -260,6 +264,8 @@ struct CodeGenProcModel {
     Index(Idx), ModelName(std::move(Name)), ModelDef(MDef), ItinsDef(IDef),
     RetireControlUnit(nullptr), LoadQueue(nullptr), StoreQueue(nullptr) {}
 
+  bool hasMacroFusions() const { return !MacroFusio...
[truncated]

llvmbot · 2023-11-14T08:12:50Z

@llvm/pr-subscribers-backend-amdgpu

Author: Wang Pengcheng (wangpc-pp)

Changes

A list of MacroFusion is added to SchedMachineModel to indicate
supported macro fusions of this processor model.

For visibility, C++ enums of all MacroFusion are generated and new
hasMacroFusion is added to Subtarget to test if a MacroFusion is
supported.

This PR is stacked on #72219, #72222

Patch is 34.86 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/72223.diff

22 Files Affected:

(modified) llvm/include/llvm/CodeGen/MacroFusion.h (+9-9)
(modified) llvm/include/llvm/CodeGen/TargetSubtargetInfo.h (+4)
(modified) llvm/include/llvm/MC/MCSchedule.h (+9)
(modified) llvm/include/llvm/MC/MCSubtargetInfo.h (+13)
(modified) llvm/include/llvm/Target/TargetInstrPredicate.td (+6)
(modified) llvm/include/llvm/Target/TargetSchedule.td (+65)
(modified) llvm/lib/CodeGen/MacroFusion.cpp (+27-12)
(modified) llvm/lib/MC/MCSchedule.cpp (+1)
(modified) llvm/lib/Target/AArch64/AArch64MacroFusion.cpp (+1-1)
(modified) llvm/lib/Target/AMDGPU/AMDGPUMacroFusion.cpp (+2-2)
(modified) llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp (+2-2)
(modified) llvm/lib/Target/ARM/ARMMacroFusion.cpp (+2-2)
(modified) llvm/lib/Target/PowerPC/PPCMacroFusion.cpp (+2-2)
(modified) llvm/lib/Target/RISCV/RISCVMacroFusion.cpp (+1-1)
(modified) llvm/lib/Target/X86/X86MacroFusion.cpp (+2-3)
(modified) llvm/utils/TableGen/CMakeLists.txt (+1)
(modified) llvm/utils/TableGen/CodeGenSchedule.cpp (+16)
(modified) llvm/utils/TableGen/CodeGenSchedule.h (+11)
(added) llvm/utils/TableGen/MacroFusionPredicatorEmitter.cpp (+208)
(modified) llvm/utils/TableGen/PredicateExpander.cpp (+8)
(modified) llvm/utils/TableGen/PredicateExpander.h (+1)
(modified) llvm/utils/TableGen/SubtargetEmitter.cpp (+46-1)

diff --git a/llvm/include/llvm/CodeGen/MacroFusion.h b/llvm/include/llvm/CodeGen/MacroFusion.h
index ea2c7a5faae385a..a97f776335368c7 100644
--- a/llvm/include/llvm/CodeGen/MacroFusion.h
+++ b/llvm/include/llvm/CodeGen/MacroFusion.h
@@ -14,8 +14,8 @@
 #ifndef LLVM_CODEGEN_MACROFUSION_H
 #define LLVM_CODEGEN_MACROFUSION_H
 
-#include <functional>
 #include <memory>
+#include <vector>
 
 namespace llvm {
 
@@ -29,10 +29,10 @@ class SUnit;
 /// Check if the instr pair, FirstMI and SecondMI, should be fused
 /// together. Given SecondMI, when FirstMI is unspecified, then check if
 /// SecondMI may be part of a fused pair at all.
-using ShouldSchedulePredTy = std::function<bool(const TargetInstrInfo &TII,
-                                                const TargetSubtargetInfo &TSI,
-                                                const MachineInstr *FirstMI,
-                                                const MachineInstr &SecondMI)>;
+using MacroFusionPredTy = bool (*)(const TargetInstrInfo &TII,
+                                   const TargetSubtargetInfo &STI,
+                                   const MachineInstr *FirstMI,
+                                   const MachineInstr &SecondMI);
 
 /// Checks if the number of cluster edges between SU and its predecessors is
 /// less than FuseLimit
@@ -48,15 +48,15 @@ bool fuseInstructionPair(ScheduleDAGInstrs &DAG, SUnit &FirstSU,
 
 /// Create a DAG scheduling mutation to pair instructions back to back
 /// for instructions that benefit according to the target-specific
-/// shouldScheduleAdjacent predicate function.
+/// predicate functions.
 std::unique_ptr<ScheduleDAGMutation>
-createMacroFusionDAGMutation(ShouldSchedulePredTy shouldScheduleAdjacent);
+createMacroFusionDAGMutation(std::vector<MacroFusionPredTy> Predicates);
 
 /// Create a DAG scheduling mutation to pair branch instructions with one
 /// of their predecessors back to back for instructions that benefit according
-/// to the target-specific shouldScheduleAdjacent predicate function.
+/// to the target-specific predicate functions.
 std::unique_ptr<ScheduleDAGMutation>
-createBranchMacroFusionDAGMutation(ShouldSchedulePredTy shouldScheduleAdjacent);
+createBranchMacroFusionDAGMutation(std::vector<MacroFusionPredTy> Predicates);
 
 } // end namespace llvm
 
diff --git a/llvm/include/llvm/CodeGen/TargetSubtargetInfo.h b/llvm/include/llvm/CodeGen/TargetSubtargetInfo.h
index 55ef95c28543190..7c76293f3e5eaea 100644
--- a/llvm/include/llvm/CodeGen/TargetSubtargetInfo.h
+++ b/llvm/include/llvm/CodeGen/TargetSubtargetInfo.h
@@ -16,6 +16,7 @@
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/CodeGen/MacroFusion.h"
 #include "llvm/CodeGen/PBQPRAConstraint.h"
 #include "llvm/CodeGen/SchedulerRegistry.h"
 #include "llvm/IR/GlobalValue.h"
@@ -323,6 +324,9 @@ class TargetSubtargetInfo : public MCSubtargetInfo {
   /// helps removing redundant copies generated by register allocator when
   /// handling complex eviction chains.
   virtual bool enableSpillageCopyElimination() const { return false; }
+
+  /// Get the list of MacroFusion predicates.
+  virtual std::vector<MacroFusionPredTy> getMacroFusions() const { return {}; }
 };
 
 } // end namespace llvm
diff --git a/llvm/include/llvm/MC/MCSchedule.h b/llvm/include/llvm/MC/MCSchedule.h
index 98ebe42cfd133b5..aa187e5cb400672 100644
--- a/llvm/include/llvm/MC/MCSchedule.h
+++ b/llvm/include/llvm/MC/MCSchedule.h
@@ -14,6 +14,7 @@
 #ifndef LLVM_MC_MCSCHEDULE_H
 #define LLVM_MC_MCSCHEDULE_H
 
+#include "llvm/ADT/Bitset.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/Support/DataTypes.h"
 #include <cassert>
@@ -196,6 +197,9 @@ struct MCExtraProcessorInfo {
   unsigned StoreQueueID;
 };
 
+const unsigned MaxMacroFusions = 256;
+using MacroFusionBitset = Bitset<MaxMacroFusions>;
+
 /// Machine model for scheduling, bundling, and heuristics.
 ///
 /// The machine model directly provides basic information about the
@@ -325,9 +329,14 @@ struct MCSchedModel {
   const InstrItinerary *InstrItineraries;
 
   const MCExtraProcessorInfo *ExtraProcessorInfo;
+  const MacroFusionBitset *MacroFusionBits;
 
   bool hasExtraProcessorInfo() const { return ExtraProcessorInfo; }
 
+  const MacroFusionBitset *getMacroFusionBits() const {
+    return MacroFusionBits;
+  }
+
   unsigned getProcessorID() const { return ProcID; }
 
   /// Does this machine model include instruction-level scheduling.
diff --git a/llvm/include/llvm/MC/MCSubtargetInfo.h b/llvm/include/llvm/MC/MCSubtargetInfo.h
index f172a799aa3331c..66fb6c9383272e1 100644
--- a/llvm/include/llvm/MC/MCSubtargetInfo.h
+++ b/llvm/include/llvm/MC/MCSubtargetInfo.h
@@ -120,6 +120,12 @@ class MCSubtargetInfo {
     return FeatureBits[Feature];
   }
 
+  bool hasMacroFusion(unsigned MacroFusion) const {
+    const MacroFusionBitset *MacroFusionBits =
+        CPUSchedModel->getMacroFusionBits();
+    return MacroFusionBits && MacroFusionBits->test(MacroFusion);
+  }
+
 protected:
   /// Initialize the scheduling model and feature bits.
   ///
@@ -295,6 +301,13 @@ class MCSubtargetInfo {
 
   /// \return if target want to issue a prefetch in address space \p AS.
   virtual bool shouldPrefetchAddressSpace(unsigned AS) const;
+
+  /// Enable macro fusion for this subtarget.
+  virtual bool enableMacroFusion() const {
+    const MacroFusionBitset *MacroFusionBits =
+        CPUSchedModel->getMacroFusionBits();
+    return MacroFusionBits && MacroFusionBits->any();
+  }
 };
 
 } // end namespace llvm
diff --git a/llvm/include/llvm/Target/TargetInstrPredicate.td b/llvm/include/llvm/Target/TargetInstrPredicate.td
index 9f2cde9d923050a..f7e6a390045b520 100644
--- a/llvm/include/llvm/Target/TargetInstrPredicate.td
+++ b/llvm/include/llvm/Target/TargetInstrPredicate.td
@@ -95,6 +95,12 @@ class MCOperandPredicate<int Index> : MCInstPredicate {
 // Return true if machine operand at position `Index` is a register operand.
 class CheckIsRegOperand<int Index> : MCOperandPredicate<Index>;
 
+// Return true if machine operand at position `Index` is a virtual register operand.
+class CheckIsVRegOperand<int Index> : MCOperandPredicate<Index>;
+
+// Return true if machine operand at position `Index` is not a virtual register operand.
+class CheckIsNotVirtualRegOperand<int Index> : CheckNot<CheckIsVRegOperand<Index>>;
+
 // Return true if machine operand at position `Index` is an immediate operand.
 class CheckIsImmOperand<int Index> : MCOperandPredicate<Index>;
 
diff --git a/llvm/include/llvm/Target/TargetSchedule.td b/llvm/include/llvm/Target/TargetSchedule.td
index 949baa5d2105c45..ed07010dfb2d39f 100644
--- a/llvm/include/llvm/Target/TargetSchedule.td
+++ b/llvm/include/llvm/Target/TargetSchedule.td
@@ -53,6 +53,7 @@
 include "llvm/Target/TargetItinerary.td"
 
 class Predicate; // Forward def
+class MacroFusion;
 
 // DAG operator that interprets the DAG args as Instruction defs.
 def instrs;
@@ -122,6 +123,9 @@ class SchedMachineModel {
   // using intervals via ResourceSegments (see
   // llvm/include/llvm/CodeGen/MachineScheduler.h).
   bit EnableIntervals = false;
+
+  // List of MacroFusion.
+  list<MacroFusion> MacroFusions = [];
 }
 
 def NoSchedModel : SchedMachineModel {
@@ -469,6 +473,67 @@ class SchedAlias<SchedReadWrite match, SchedReadWrite alias> {
   SchedMachineModel SchedModel = ?;
 }
 
+// Base class of MacroFusionPredicate, etc. The avaliable variables are:
+// * const TargetInstrInfo &TII
+// * const TargetSubtargetInfo &STI
+// * const MachineRegisterInfo &MRI
+// * const MachineInstr *FirstMI
+// * const MachineInstr &SecondMI
+class MacroFusionPredicateBase;
+
+// MacroFusionPredicate with raw code predicate.
+class MacroFusionPredicate<code pred> : MacroFusionPredicateBase {
+  code Predicate = pred;
+}
+
+// Binds firstOpIdx and secondOpIdx. The operand of `FirstMI` at position
+// `firstOpIdx` should be the same as the operand of `SenondMI` at position
+// `secondOpIdx`.
+class BindReg<int firstOpIdx, int secondOpIdx> : MacroFusionPredicateBase {
+  int FirstOpIdx = firstOpIdx;
+  int SecondOpIdx = secondOpIdx;
+}
+
+// A predicate for wildcard. The generated code will be like:
+// ```
+// if (!FirstMI)
+//   return ReturnValue;
+// ```
+class WildcardPred<bit ret> : MacroFusionPredicateBase {
+  bit ReturnValue = ret;
+}
+def WildcardFalse : WildcardPred<0>;
+def WildcardTrue : WildcardPred<1>;
+
+// Indicates that the destination register of `FirstMI` should be have one
+// use if it is an virtual register.
+class OneUsePred : MacroFusionPredicateBase;
+def OneUse : OneUsePred;
+
+// Handled by MacroFusionPredicatorEmitter backend.
+// The generated predicator will be like:
+// ```
+// bool isNAME(const TargetInstrInfo &TII,
+//             const TargetSubtargetInfo &STI,
+//             const MachineInstr *FirstMI,
+//             const MachineInstr &SecondMI) {
+//   auto &MRI = SecondMI.getMF()->getRegInfo();
+//   /* Prolog */
+//   /* Predicate for `FirstMI` */
+//   /* Predicate for `SecondMI` */
+//   /* Epilog */
+//   return true;
+// }
+// ```
+class MacroFusion<MCInstPredicate first, MCInstPredicate second,
+                  list<MacroFusionPredicateBase> prolog = [],
+                  list<MacroFusionPredicateBase> epilog = []> {
+  MCInstPredicate First = first;
+  MCInstPredicate Second = second;
+  list<MacroFusionPredicateBase> Prolog = prolog;
+  list<MacroFusionPredicateBase> Epilog = epilog;
+}
+
 // Allow the definition of processor register files for register renaming
 // purposes.
 //
diff --git a/llvm/lib/CodeGen/MacroFusion.cpp b/llvm/lib/CodeGen/MacroFusion.cpp
index fa5df68b8abcc0f..63ea53f9cc32f97 100644
--- a/llvm/lib/CodeGen/MacroFusion.cpp
+++ b/llvm/lib/CodeGen/MacroFusion.cpp
@@ -137,19 +137,35 @@ namespace {
 /// Post-process the DAG to create cluster edges between instrs that may
 /// be fused by the processor into a single operation.
 class MacroFusion : public ScheduleDAGMutation {
-  ShouldSchedulePredTy shouldScheduleAdjacent;
+  std::vector<MacroFusionPredTy> Predicates;
   bool FuseBlock;
   bool scheduleAdjacentImpl(ScheduleDAGInstrs &DAG, SUnit &AnchorSU);
 
 public:
-  MacroFusion(ShouldSchedulePredTy shouldScheduleAdjacent, bool FuseBlock)
-    : shouldScheduleAdjacent(shouldScheduleAdjacent), FuseBlock(FuseBlock) {}
+  MacroFusion(std::vector<MacroFusionPredTy> Predicates, bool FuseBlock)
+      : Predicates(Predicates), FuseBlock(FuseBlock) {}
 
   void apply(ScheduleDAGInstrs *DAGInstrs) override;
+
+  bool shouldScheduleAdjacent(const TargetInstrInfo &TII,
+                              const TargetSubtargetInfo &STI,
+                              const MachineInstr *FirstMI,
+                              const MachineInstr &SecondMI);
 };
 
 } // end anonymous namespace
 
+bool MacroFusion::shouldScheduleAdjacent(const TargetInstrInfo &TII,
+                                         const TargetSubtargetInfo &STI,
+                                         const MachineInstr *FirstMI,
+                                         const MachineInstr &SecondMI) {
+  for (MacroFusionPredTy Predicate : Predicates) {
+    if (Predicate(TII, STI, FirstMI, SecondMI))
+      return true;
+  }
+  return false;
+}
+
 void MacroFusion::apply(ScheduleDAGInstrs *DAG) {
   if (FuseBlock)
     // For each of the SUnits in the scheduling block, try to fuse the instr in
@@ -197,17 +213,16 @@ bool MacroFusion::scheduleAdjacentImpl(ScheduleDAGInstrs &DAG, SUnit &AnchorSU)
 }
 
 std::unique_ptr<ScheduleDAGMutation>
-llvm::createMacroFusionDAGMutation(
-     ShouldSchedulePredTy shouldScheduleAdjacent) {
-  if(EnableMacroFusion)
-    return std::make_unique<MacroFusion>(shouldScheduleAdjacent, true);
+llvm::createMacroFusionDAGMutation(std::vector<MacroFusionPredTy> Predicates) {
+  if (EnableMacroFusion) {
+    return std::make_unique<MacroFusion>(Predicates, true);
+  }
   return nullptr;
 }
 
-std::unique_ptr<ScheduleDAGMutation>
-llvm::createBranchMacroFusionDAGMutation(
-     ShouldSchedulePredTy shouldScheduleAdjacent) {
-  if(EnableMacroFusion)
-    return std::make_unique<MacroFusion>(shouldScheduleAdjacent, false);
+std::unique_ptr<ScheduleDAGMutation> llvm::createBranchMacroFusionDAGMutation(
+    std::vector<MacroFusionPredTy> Predicates) {
+  if (EnableMacroFusion)
+    return std::make_unique<MacroFusion>(Predicates, false);
   return nullptr;
 }
diff --git a/llvm/lib/MC/MCSchedule.cpp b/llvm/lib/MC/MCSchedule.cpp
index 990a693559a7776..19c36cb0e58d9c1 100644
--- a/llvm/lib/MC/MCSchedule.cpp
+++ b/llvm/lib/MC/MCSchedule.cpp
@@ -37,6 +37,7 @@ const MCSchedModel MCSchedModel::Default = {DefaultIssueWidth,
                                             0,
                                             0,
                                             nullptr,
+                                            nullptr,
                                             nullptr};
 
 int MCSchedModel::computeInstrLatency(const MCSubtargetInfo &STI,
diff --git a/llvm/lib/Target/AArch64/AArch64MacroFusion.cpp b/llvm/lib/Target/AArch64/AArch64MacroFusion.cpp
index 05d60872bf51aca..8f46f3eabb3ef45 100644
--- a/llvm/lib/Target/AArch64/AArch64MacroFusion.cpp
+++ b/llvm/lib/Target/AArch64/AArch64MacroFusion.cpp
@@ -478,5 +478,5 @@ static bool shouldScheduleAdjacent(const TargetInstrInfo &TII,
 
 std::unique_ptr<ScheduleDAGMutation>
 llvm::createAArch64MacroFusionDAGMutation() {
-  return createMacroFusionDAGMutation(shouldScheduleAdjacent);
+  return createMacroFusionDAGMutation({shouldScheduleAdjacent});
 }
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUMacroFusion.cpp b/llvm/lib/Target/AMDGPU/AMDGPUMacroFusion.cpp
index c15c94ee17f8b1d..b2b11d661523e9c 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUMacroFusion.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUMacroFusion.cpp
@@ -59,8 +59,8 @@ static bool shouldScheduleAdjacent(const TargetInstrInfo &TII_,
 
 namespace llvm {
 
-std::unique_ptr<ScheduleDAGMutation> createAMDGPUMacroFusionDAGMutation () {
-  return createMacroFusionDAGMutation(shouldScheduleAdjacent);
+std::unique_ptr<ScheduleDAGMutation> createAMDGPUMacroFusionDAGMutation() {
+  return createMacroFusionDAGMutation({shouldScheduleAdjacent});
 }
 
 } // end namespace llvm
diff --git a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
index 29c9b9ccf27614f..0bddeeef9e9b1a3 100644
--- a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
@@ -142,10 +142,10 @@ namespace {
 /// be turned into VOPD instructions
 /// Greedily pairs instruction candidates. O(n^2) algorithm.
 struct VOPDPairingMutation : ScheduleDAGMutation {
-  ShouldSchedulePredTy shouldScheduleAdjacent; // NOLINT: function pointer
+  MacroFusionPredTy shouldScheduleAdjacent; // NOLINT: function pointer
 
   VOPDPairingMutation(
-      ShouldSchedulePredTy shouldScheduleAdjacent) // NOLINT: function pointer
+      MacroFusionPredTy shouldScheduleAdjacent) // NOLINT: function pointer
       : shouldScheduleAdjacent(shouldScheduleAdjacent) {}
 
   void apply(ScheduleDAGInstrs *DAG) override {
diff --git a/llvm/lib/Target/ARM/ARMMacroFusion.cpp b/llvm/lib/Target/ARM/ARMMacroFusion.cpp
index 38bf28ba8219b90..7de117925e464fe 100644
--- a/llvm/lib/Target/ARM/ARMMacroFusion.cpp
+++ b/llvm/lib/Target/ARM/ARMMacroFusion.cpp
@@ -62,8 +62,8 @@ static bool shouldScheduleAdjacent(const TargetInstrInfo &TII,
   return false;
 }
 
-std::unique_ptr<ScheduleDAGMutation> createARMMacroFusionDAGMutation () {
-  return createMacroFusionDAGMutation(shouldScheduleAdjacent);
+std::unique_ptr<ScheduleDAGMutation> createARMMacroFusionDAGMutation() {
+  return createMacroFusionDAGMutation({shouldScheduleAdjacent});
 }
 
 } // end namespace llvm
diff --git a/llvm/lib/Target/PowerPC/PPCMacroFusion.cpp b/llvm/lib/Target/PowerPC/PPCMacroFusion.cpp
index bf1c39a3a3a2d47..d6a4a5dd5faabae 100644
--- a/llvm/lib/Target/PowerPC/PPCMacroFusion.cpp
+++ b/llvm/lib/Target/PowerPC/PPCMacroFusion.cpp
@@ -286,8 +286,8 @@ static bool shouldScheduleAdjacent(const TargetInstrInfo &TII,
 
 namespace llvm {
 
-std::unique_ptr<ScheduleDAGMutation> createPowerPCMacroFusionDAGMutation () {
-  return createMacroFusionDAGMutation(shouldScheduleAdjacent);
+std::unique_ptr<ScheduleDAGMutation> createPowerPCMacroFusionDAGMutation() {
+  return createMacroFusionDAGMutation({shouldScheduleAdjacent});
 }
 
 } // end namespace llvm
diff --git a/llvm/lib/Target/RISCV/RISCVMacroFusion.cpp b/llvm/lib/Target/RISCV/RISCVMacroFusion.cpp
index 02a8d5c18fe1a0e..1b82cc8b5b858f4 100644
--- a/llvm/lib/Target/RISCV/RISCVMacroFusion.cpp
+++ b/llvm/lib/Target/RISCV/RISCVMacroFusion.cpp
@@ -65,5 +65,5 @@ static bool shouldScheduleAdjacent(const TargetInstrInfo &TII,
 }
 
 std::unique_ptr<ScheduleDAGMutation> llvm::createRISCVMacroFusionDAGMutation() {
-  return createMacroFusionDAGMutation(shouldScheduleAdjacent);
+  return createMacroFusionDAGMutation({shouldScheduleAdjacent});
 }
diff --git a/llvm/lib/Target/X86/X86MacroFusion.cpp b/llvm/lib/Target/X86/X86MacroFusion.cpp
index aa6e8645e0927f5..382cc9a71c2a184 100644
--- a/llvm/lib/Target/X86/X86MacroFusion.cpp
+++ b/llvm/lib/Target/X86/X86MacroFusion.cpp
@@ -67,9 +67,8 @@ static bool shouldScheduleAdjacent(const TargetInstrInfo &TII,
 
 namespace llvm {
 
-std::unique_ptr<ScheduleDAGMutation>
-createX86MacroFusionDAGMutation () {
-  return createBranchMacroFusionDAGMutation(shouldScheduleAdjacent);
+std::unique_ptr<ScheduleDAGMutation> createX86MacroFusionDAGMutation() {
+  return createBranchMacroFusionDAGMutation({shouldScheduleAdjacent});
 }
 
 } // end namespace llvm
diff --git a/llvm/utils/TableGen/CMakeLists.txt b/llvm/utils/TableGen/CMakeLists.txt
index 071ea3bc07054bb..f765cc36d3bebed 100644
--- a/llvm/utils/TableGen/CMakeLists.txt
+++ b/llvm/utils/TableGen/CMakeLists.txt
@@ -72,6 +72,7 @@ add_tablegen(llvm-tblgen LLVM
   PredicateExpander.cpp
   PseudoLoweringEmitter.cpp
   CompressInstEmitter.cpp
+  MacroFusionPredicatorEmitter.cpp
   RegisterBankEmitter.cpp
   RegisterInfoEmitter.cpp
   SearchableTableEmitter.cpp
diff --git a/llvm/utils/TableGen/CodeGenSchedule.cpp b/llvm/utils/TableGen/CodeGenSchedule.cpp
index c3c5e4f8eb2d8c3..98bf245697b77db 100644
--- a/llvm/utils/TableGen/CodeGenSchedule.cpp
+++ b/llvm/utils/TableGen/CodeGenSchedule.cpp
@@ -208,6 +208,9 @@ CodeGenSchedModels::CodeGenSchedModels(RecordKeeper &RK,
   // (For per-operand resources mapped to itinerary classes).
   collectProcUnsupportedFeatures();
 
+  // Find MacroFusion records for each processor.
+  collectMacroFusions();
+
   // Infer new SchedClasses from SchedVariant.
   inferSchedClasses();
 
@@ -1150,6 +1153,13 @@ bool CodeGenSchedModels::hasItineraries() const {
   return false;
 }
 
+bool CodeGenSchedModels::hasMacroFusions() const {
+  for (const CodeGenProcModel &PM : ProcModels)
+    if (PM.hasMacroFusions())
+      return true;
+  return false;
+}
+
 // Gather the processor itineraries.
 void CodeGenSchedModels::collectProcItins() {
   LLVM_DEBUG(dbgs() << "\n+++ PROBLEM ITINERARIES (collectProcItins) +++\n");
@@ -1220,6 +1230,12 @@ void CodeGenSchedModels::collectProcUnsupportedFeatures() {
         ProcModel.ModelDef->getValueAsListOfDefs("UnsupportedFeatures"));
 }
 
+void CodeGenSchedModels::collectMacroFusions() {
+  for (CodeGenProcModel &ProcModel : ProcModels)
+    append_range(ProcModel.MacroFusions,
+                 ProcModel.ModelDef->getValueAsListOfDefs("MacroFusions"));
+}
+
 /// Infer new classes from existing classes. In the process, this may create new
 /// SchedWrites from sequences of existing SchedWrites.
 void CodeGenSchedModels::inferSchedClasses() {
diff --git a/llvm/utils/TableGen/CodeGenSchedule.h b/llvm/utils/TableGen/CodeGenSchedule.h
index 76ef1e43953078e..317558c52fbef15 100644
--- a/llvm/utils/TableGen/CodeGenSchedule.h
+++ b/llvm/utils/TableGen/CodeGenSchedule.h
@@ -238,6 +238,10 @@ struct CodeGenProcModel {
   // This list is empty if the Processor has no UnsupportedFeatures.
   RecVec UnsupportedFeaturesDefs;
 
+  // List of MacroFusion.
+  // This list is empty if the Processor has no MacroFusion.
+  RecVec MacroFusions;
+
   // All read/write resources associated with this processor.
   RecVec WriteResDefs;
   RecVec ReadAdvanceDefs;
@@ -260,6 +264,8 @@ struct CodeGenProcModel {
     Index(Idx), ModelName(std::move(Name)), ModelDef(MDef), ItinsDef(IDef),
     RetireControlUnit(nullptr), LoadQueue(nullptr), StoreQueue(nullptr) {}
 
+  bool hasMacroFusions() const { return !MacroFusio...
[truncated]

llvm/utils/TableGen/CodeGenSchedule.cpp

github-actions · 2023-11-20T03:49:04Z

✅ With the latest revision this PR passed the C/C++ code formatter.

The user can provide multiple predicators to MacroFusion and the DAG mutation will be applied if one of them is evalated to true. `ShouldSchedulePredTy` is renamed to `MacroFusionPredTy`.

`FusionPredicate` is used to predicate if target instruction matches the requirement. The targets can be firstMI, secondMI or both. The `Fusion` contains a list of `FusionPredicate`. The generated code will be like: ``` bool isNAME(const TargetInstrInfo &TII, const TargetSubtargetInfo &STI, const MachineInstr *FirstMI, const MachineInstr &SecondMI) { auto &MRI = SecondMI.getMF()->getRegInfo(); /* Predicates */ return true; } ``` A boilerplate class called `SimpleFusion` is added. `SimpleFusion` has a predefined structure of predicates and accepts predicate for `firstMI`, predicate for `secondMI` and epilog/prolog as arguments. The generated code for `SimpleFusion` will be like: ``` bool isNAME(const TargetInstrInfo &TII, const TargetSubtargetInfo &STI, const MachineInstr *FirstMI, const MachineInstr &SecondMI) { auto &MRI = SecondMI.getMF()->getRegInfo(); /* Prolog */ /* Predicate for `SecondMI` */ /* Wildcard */ /* Predicate for `FirstMI` */ /* Check One Use */ /* Tie registers */ /* Epilog */ return true; } ```

A list of `MacroFusion` is added to `SchedMachineModel` to indicate supported macro fusions of this processor model. For visibility, C++ enums of all MacroFusion are generated and new `hasMacroFusion` is added to Subtarget to test if a MacroFusion is supported.

Doc: https://xiangshan-doc.readthedocs.io/zh-cn/latest/frontend/decode/ This PR is to show the usage of TableGen-based macro fusions. Some instrcution pairs can be folded into one MacroFusion definition but I leave them standalone to show the different ways to define a macro fusion. This PR is stacked on llvm#72219, llvm#72222, llvm#72223, llvm#72224, llvm#72227

We add an option `-mfusion` like `-mattr` to add/remove supported macro fusions, so that we can test each macro fusion separately via `llc`. This PR is stacked on llvm#72219, llvm#72222, llvm#72223

wangpc-pp · 2023-11-22T13:17:53Z

Close this PR as I prefer #73115.

Doc: https://xiangshan-doc.readthedocs.io/zh-cn/latest/frontend/decode/ This PR is to show the usage of TableGen-based macro fusions. Some instrcution pairs can be folded into one MacroFusion definition but I leave them standalone to show the different ways to define a macro fusion. This PR is stacked on llvm#72219, llvm#72222, llvm#72223, llvm#72224, llvm#72227

llvmbot added backend:ARM backend:AArch64 backend:AMDGPU backend:RISC-V backend:X86 mc Machine (object) code labels Nov 14, 2023

wangpc-pp requested review from asb, preames, Sisyph, topperc and zixuan-wu November 14, 2023 08:13

This was referenced Nov 14, 2023

[RISCV] Use TableGen-based macro fusion #72224

Merged

[Sched] Add MacroFusion mutation if fusions are not empty #72227

Merged

topperc reviewed Nov 14, 2023

View reviewed changes

llvm/utils/TableGen/CodeGenSchedule.cpp Outdated Show resolved Hide resolved

wangpc-pp force-pushed the main-tablegen-macro-fusion-sched branch 2 times, most recently from ae0b1a1 to 63e24a3 Compare November 15, 2023 06:46

wangpc-pp mentioned this pull request Nov 15, 2023

[RISCV] Add macro fusions for Xiangshan #72362

Open

wangpc-pp force-pushed the main-tablegen-macro-fusion-sched branch from 63e24a3 to 56e1b5a Compare November 20, 2023 03:46

wangpc-pp added 3 commits November 21, 2023 16:17

[MacroFusion] Support multiple predicators

e910bae

The user can provide multiple predicators to MacroFusion and the DAG mutation will be applied if one of them is evalated to true. `ShouldSchedulePredTy` is renamed to `MacroFusionPredTy`.

wangpc-pp force-pushed the main-tablegen-macro-fusion-sched branch from 56e1b5a to f4cd61c Compare November 21, 2023 08:26

wangpc-pp mentioned this pull request Nov 22, 2023

[TableGen] Enhance testability of TableGen-based macro fusion #73075

Closed

wangpc-pp closed this Nov 22, 2023

wangpc-pp deleted the main-tablegen-macro-fusion-sched branch March 29, 2024 04:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TableGen] Add MacroFusions to SchedMachineModel #72223

[TableGen] Add MacroFusions to SchedMachineModel #72223

wangpc-pp commented Nov 14, 2023

llvmbot commented Nov 14, 2023 •

edited

Loading

llvmbot commented Nov 14, 2023

github-actions bot commented Nov 20, 2023 •

edited

Loading

wangpc-pp commented Nov 22, 2023

[TableGen] Add MacroFusions to SchedMachineModel #72223

[TableGen] Add MacroFusions to SchedMachineModel #72223

Conversation

wangpc-pp commented Nov 14, 2023

llvmbot commented Nov 14, 2023 • edited Loading

llvmbot commented Nov 14, 2023

github-actions bot commented Nov 20, 2023 • edited Loading

wangpc-pp commented Nov 22, 2023

llvmbot commented Nov 14, 2023 •

edited

Loading

github-actions bot commented Nov 20, 2023 •

edited

Loading