Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AMDGPU: Set max supported div/rem size to 64 #80669

Merged
merged 2 commits into from
Feb 5, 2024
Merged

AMDGPU: Set max supported div/rem size to 64 #80669

merged 2 commits into from
Feb 5, 2024

Conversation

arsenm
Copy link
Contributor

@arsenm arsenm commented Feb 5, 2024

This enables IR expansion for i128 divisions. The vector case is still broken because ExpandLargeDivRem doesn't try to handle them.

Fixes: SWDEV-426193

@arsenm
AMDGPU: Set max supported div/rem size to 64
bd26bbb 
This enables IR expansion for i128 divisions. The vector case is
still broken because ExpandLargeDivRem doesn't try to handle them.

Fixes: SWDEV-426193
@arsenm arsenm requested review from yxsamliu and Pierre-vh February 5, 2024 12:01
@llvmbot
Copy link
Member

llvmbot commented Feb 5, 2024

@llvm/pr-subscribers-backend-amdgpu

Author: Matt Arsenault (arsenm)

Changes

This enables IR expansion for i128 divisions. The vector case is still broken because ExpandLargeDivRem doesn't try to handle them.

Fixes: SWDEV-426193

Patch is 87.05 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/80669.diff

3 Files Affected:

  • (modified) llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp (+1)
  • (modified) llvm/test/CodeGen/AMDGPU/div_i128.ll (+1643-4)
  • (added) llvm/test/CodeGen/AMDGPU/div_v2i128.ll (+25)
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp b/llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp
index b420e72d87ed0..10569d97248b9 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp
@@ -577,6 +577,7 @@ AMDGPUTargetLowering::AMDGPUTargetLowering(const TargetMachine &TM,
                        ISD::AssertSext, ISD::INTRINSIC_WO_CHAIN});
 
   setMaxAtomicSizeInBitsSupported(64);
+  setMaxDivRemBitWidthSupported(64);
 }
 
 bool AMDGPUTargetLowering::mayIgnoreSignedZero(SDValue Op) const {
diff --git a/llvm/test/CodeGen/AMDGPU/div_i128.ll b/llvm/test/CodeGen/AMDGPU/div_i128.ll
index 4aa97c57cbd9c..c673ac8e03ff7 100644
--- a/llvm/test/CodeGen/AMDGPU/div_i128.ll
+++ b/llvm/test/CodeGen/AMDGPU/div_i128.ll
@@ -1,9 +1,1648 @@
-; RUN: not --crash llc -global-isel=0 -mtriple=amdgcn-amd-amdhsa -verify-machineinstrs -o - %s 2>&1 | FileCheck -check-prefix=SDAG-ERR %s
-; RUN: not --crash llc -global-isel=1 -mtriple=amdgcn-amd-amdhsa -verify-machineinstrs -o - %s 2>&1 | FileCheck -check-prefix=GISEL-ERR %s
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 4
+; RUN: llc -global-isel=0 -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -o - %s | FileCheck -check-prefixes=GFX9,GFX9-SDAG %s
+; RUN: llc -global-isel=1 -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -o - %s | FileCheck -check-prefixes=GFX9,GFX9-GISEL %s
 
-; SDAG-ERR: LLVM ERROR: unsupported libcall legalization
-; GISEL-ERR: LLVM ERROR: unable to legalize instruction: %{{[0-9]+}}:_(s128) = G_SDIV %{{[0-9]+}}:_, %{{[0-9]+}}:_ (in function: v_sdiv_i128_vv)
 define i128 @v_sdiv_i128_vv(i128 %lhs, i128 %rhs) {
+; GFX9-SDAG-LABEL: v_sdiv_i128_vv:
+; GFX9-SDAG:       ; %bb.0: ; %_udiv-special-cases
+; GFX9-SDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-SDAG-NEXT:    v_ashrrev_i32_e32 v16, 31, v3
+; GFX9-SDAG-NEXT:    v_xor_b32_e32 v0, v16, v0
+; GFX9-SDAG-NEXT:    v_xor_b32_e32 v1, v16, v1
+; GFX9-SDAG-NEXT:    v_sub_co_u32_e32 v8, vcc, v0, v16
+; GFX9-SDAG-NEXT:    v_xor_b32_e32 v2, v16, v2
+; GFX9-SDAG-NEXT:    v_subb_co_u32_e32 v9, vcc, v1, v16, vcc
+; GFX9-SDAG-NEXT:    v_ashrrev_i32_e32 v17, 31, v7
+; GFX9-SDAG-NEXT:    v_xor_b32_e32 v3, v16, v3
+; GFX9-SDAG-NEXT:    v_subb_co_u32_e32 v10, vcc, v2, v16, vcc
+; GFX9-SDAG-NEXT:    v_subb_co_u32_e32 v11, vcc, v3, v16, vcc
+; GFX9-SDAG-NEXT:    v_xor_b32_e32 v3, v17, v4
+; GFX9-SDAG-NEXT:    v_xor_b32_e32 v2, v17, v5
+; GFX9-SDAG-NEXT:    v_sub_co_u32_e32 v20, vcc, v3, v17
+; GFX9-SDAG-NEXT:    v_xor_b32_e32 v0, v17, v6
+; GFX9-SDAG-NEXT:    v_subb_co_u32_e32 v21, vcc, v2, v17, vcc
+; GFX9-SDAG-NEXT:    v_xor_b32_e32 v1, v17, v7
+; GFX9-SDAG-NEXT:    v_subb_co_u32_e32 v0, vcc, v0, v17, vcc
+; GFX9-SDAG-NEXT:    v_subb_co_u32_e32 v1, vcc, v1, v17, vcc
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v3, v21, v1
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v2, v20, v0
+; GFX9-SDAG-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[2:3]
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v3, v9, v11
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v2, v8, v10
+; GFX9-SDAG-NEXT:    v_cmp_eq_u64_e64 s[4:5], 0, v[2:3]
+; GFX9-SDAG-NEXT:    v_ffbh_u32_e32 v2, v0
+; GFX9-SDAG-NEXT:    v_add_u32_e32 v2, 32, v2
+; GFX9-SDAG-NEXT:    v_ffbh_u32_e32 v3, v1
+; GFX9-SDAG-NEXT:    v_min_u32_e32 v2, v2, v3
+; GFX9-SDAG-NEXT:    v_ffbh_u32_e32 v3, v20
+; GFX9-SDAG-NEXT:    v_add_u32_e32 v3, 32, v3
+; GFX9-SDAG-NEXT:    v_ffbh_u32_e32 v4, v21
+; GFX9-SDAG-NEXT:    v_min_u32_e32 v3, v3, v4
+; GFX9-SDAG-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
+; GFX9-SDAG-NEXT:    v_add_co_u32_e32 v3, vcc, 64, v3
+; GFX9-SDAG-NEXT:    v_addc_co_u32_e64 v4, s[6:7], 0, 0, vcc
+; GFX9-SDAG-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
+; GFX9-SDAG-NEXT:    v_ffbh_u32_e32 v5, v11
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e32 v2, v3, v2, vcc
+; GFX9-SDAG-NEXT:    v_ffbh_u32_e32 v3, v10
+; GFX9-SDAG-NEXT:    v_add_u32_e32 v3, 32, v3
+; GFX9-SDAG-NEXT:    v_min_u32_e32 v3, v3, v5
+; GFX9-SDAG-NEXT:    v_ffbh_u32_e32 v5, v8
+; GFX9-SDAG-NEXT:    v_add_u32_e32 v5, 32, v5
+; GFX9-SDAG-NEXT:    v_ffbh_u32_e32 v6, v9
+; GFX9-SDAG-NEXT:    v_min_u32_e32 v5, v5, v6
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e64 v4, v4, 0, vcc
+; GFX9-SDAG-NEXT:    v_add_co_u32_e32 v5, vcc, 64, v5
+; GFX9-SDAG-NEXT:    v_addc_co_u32_e64 v6, s[6:7], 0, 0, vcc
+; GFX9-SDAG-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[10:11]
+; GFX9-SDAG-NEXT:    s_mov_b64 s[6:7], 0x7f
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e32 v3, v5, v3, vcc
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e64 v6, v6, 0, vcc
+; GFX9-SDAG-NEXT:    v_sub_co_u32_e32 v2, vcc, v2, v3
+; GFX9-SDAG-NEXT:    v_subb_co_u32_e32 v3, vcc, v4, v6, vcc
+; GFX9-SDAG-NEXT:    v_mov_b32_e32 v5, 0
+; GFX9-SDAG-NEXT:    v_subbrev_co_u32_e32 v4, vcc, 0, v5, vcc
+; GFX9-SDAG-NEXT:    v_subbrev_co_u32_e32 v5, vcc, 0, v5, vcc
+; GFX9-SDAG-NEXT:    v_cmp_lt_u64_e32 vcc, s[6:7], v[2:3]
+; GFX9-SDAG-NEXT:    v_mov_b32_e32 v18, v16
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
+; GFX9-SDAG-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[4:5]
+; GFX9-SDAG-NEXT:    v_mov_b32_e32 v19, v17
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; GFX9-SDAG-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[4:5]
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e32 v6, v7, v6, vcc
+; GFX9-SDAG-NEXT:    v_and_b32_e32 v6, 1, v6
+; GFX9-SDAG-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v6
+; GFX9-SDAG-NEXT:    v_xor_b32_e32 v6, 0x7f, v2
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v7, v3, v5
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v6, v6, v4
+; GFX9-SDAG-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
+; GFX9-SDAG-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[6:7]
+; GFX9-SDAG-NEXT:    s_xor_b64 s[6:7], s[4:5], -1
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e64 v13, v11, 0, s[4:5]
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e64 v12, v10, 0, s[4:5]
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e64 v7, v9, 0, s[4:5]
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e64 v6, v8, 0, s[4:5]
+; GFX9-SDAG-NEXT:    s_and_b64 s[4:5], s[6:7], vcc
+; GFX9-SDAG-NEXT:    s_and_saveexec_b64 s[8:9], s[4:5]
+; GFX9-SDAG-NEXT:    s_cbranch_execz .LBB0_6
+; GFX9-SDAG-NEXT:  ; %bb.1: ; %udiv-bb1
+; GFX9-SDAG-NEXT:    v_add_co_u32_e32 v22, vcc, 1, v2
+; GFX9-SDAG-NEXT:    v_addc_co_u32_e32 v23, vcc, 0, v3, vcc
+; GFX9-SDAG-NEXT:    v_addc_co_u32_e32 v24, vcc, 0, v4, vcc
+; GFX9-SDAG-NEXT:    v_sub_u32_e32 v7, 0x7f, v2
+; GFX9-SDAG-NEXT:    v_addc_co_u32_e32 v25, vcc, 0, v5, vcc
+; GFX9-SDAG-NEXT:    v_sub_u32_e32 v12, 64, v7
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v4, v23, v25
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v3, v22, v24
+; GFX9-SDAG-NEXT:    v_lshlrev_b64 v[5:6], v7, v[10:11]
+; GFX9-SDAG-NEXT:    v_lshrrev_b64 v[12:13], v12, v[8:9]
+; GFX9-SDAG-NEXT:    v_sub_u32_e32 v2, 63, v2
+; GFX9-SDAG-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[3:4]
+; GFX9-SDAG-NEXT:    v_lshlrev_b64 v[2:3], v2, v[8:9]
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v4, v6, v13
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v5, v5, v12
+; GFX9-SDAG-NEXT:    v_cmp_gt_u32_e64 s[4:5], 64, v7
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e64 v3, v3, v4, s[4:5]
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e64 v2, v2, v5, s[4:5]
+; GFX9-SDAG-NEXT:    v_lshlrev_b64 v[4:5], v7, v[8:9]
+; GFX9-SDAG-NEXT:    v_cmp_eq_u32_e64 s[6:7], 0, v7
+; GFX9-SDAG-NEXT:    v_mov_b32_e32 v6, 0
+; GFX9-SDAG-NEXT:    v_mov_b32_e32 v12, 0
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e64 v3, v3, v11, s[6:7]
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e64 v2, v2, v10, s[6:7]
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e64 v5, 0, v5, s[4:5]
+; GFX9-SDAG-NEXT:    v_mov_b32_e32 v7, 0
+; GFX9-SDAG-NEXT:    v_mov_b32_e32 v13, 0
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e64 v4, 0, v4, s[4:5]
+; GFX9-SDAG-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; GFX9-SDAG-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
+; GFX9-SDAG-NEXT:    s_cbranch_execz .LBB0_5
+; GFX9-SDAG-NEXT:  ; %bb.2: ; %udiv-preheader
+; GFX9-SDAG-NEXT:    v_sub_u32_e32 v12, 64, v22
+; GFX9-SDAG-NEXT:    v_lshrrev_b64 v[6:7], v22, v[8:9]
+; GFX9-SDAG-NEXT:    v_lshlrev_b64 v[12:13], v12, v[10:11]
+; GFX9-SDAG-NEXT:    v_cmp_gt_u32_e32 vcc, 64, v22
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v12, v6, v12
+; GFX9-SDAG-NEXT:    v_subrev_u32_e32 v6, 64, v22
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v13, v7, v13
+; GFX9-SDAG-NEXT:    v_lshrrev_b64 v[6:7], v6, v[10:11]
+; GFX9-SDAG-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v22
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e32 v7, v7, v13, vcc
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e64 v9, v7, v9, s[4:5]
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e32 v12, v6, v12, vcc
+; GFX9-SDAG-NEXT:    v_lshrrev_b64 v[6:7], v22, v[10:11]
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e64 v8, v12, v8, s[4:5]
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e32 v11, 0, v7, vcc
+; GFX9-SDAG-NEXT:    v_cndmask_b32_e32 v10, 0, v6, vcc
+; GFX9-SDAG-NEXT:    v_add_co_u32_e32 v26, vcc, -1, v20
+; GFX9-SDAG-NEXT:    v_addc_co_u32_e32 v27, vcc, -1, v21, vcc
+; GFX9-SDAG-NEXT:    v_addc_co_u32_e32 v28, vcc, -1, v0, vcc
+; GFX9-SDAG-NEXT:    v_mov_b32_e32 v14, 0
+; GFX9-SDAG-NEXT:    v_mov_b32_e32 v12, 0
+; GFX9-SDAG-NEXT:    v_addc_co_u32_e32 v29, vcc, -1, v1, vcc
+; GFX9-SDAG-NEXT:    s_mov_b64 s[4:5], 0
+; GFX9-SDAG-NEXT:    v_mov_b32_e32 v15, 0
+; GFX9-SDAG-NEXT:    v_mov_b32_e32 v13, 0
+; GFX9-SDAG-NEXT:    v_mov_b32_e32 v7, 0
+; GFX9-SDAG-NEXT:  .LBB0_3: ; %udiv-do-while
+; GFX9-SDAG-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-SDAG-NEXT:    v_lshrrev_b32_e32 v6, 31, v5
+; GFX9-SDAG-NEXT:    v_lshlrev_b64 v[4:5], 1, v[4:5]
+; GFX9-SDAG-NEXT:    v_lshlrev_b64 v[10:11], 1, v[10:11]
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v4, v14, v4
+; GFX9-SDAG-NEXT:    v_lshrrev_b32_e32 v14, 31, v9
+; GFX9-SDAG-NEXT:    v_lshlrev_b64 v[8:9], 1, v[8:9]
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v10, v10, v14
+; GFX9-SDAG-NEXT:    v_lshrrev_b32_e32 v14, 31, v3
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v8, v8, v14
+; GFX9-SDAG-NEXT:    v_sub_co_u32_e32 v14, vcc, v26, v8
+; GFX9-SDAG-NEXT:    v_subb_co_u32_e32 v14, vcc, v27, v9, vcc
+; GFX9-SDAG-NEXT:    v_subb_co_u32_e32 v14, vcc, v28, v10, vcc
+; GFX9-SDAG-NEXT:    v_subb_co_u32_e32 v14, vcc, v29, v11, vcc
+; GFX9-SDAG-NEXT:    v_ashrrev_i32_e32 v30, 31, v14
+; GFX9-SDAG-NEXT:    v_and_b32_e32 v14, v30, v20
+; GFX9-SDAG-NEXT:    v_sub_co_u32_e32 v8, vcc, v8, v14
+; GFX9-SDAG-NEXT:    v_and_b32_e32 v14, v30, v21
+; GFX9-SDAG-NEXT:    v_subb_co_u32_e32 v9, vcc, v9, v14, vcc
+; GFX9-SDAG-NEXT:    v_and_b32_e32 v14, v30, v0
+; GFX9-SDAG-NEXT:    v_subb_co_u32_e32 v10, vcc, v10, v14, vcc
+; GFX9-SDAG-NEXT:    v_and_b32_e32 v14, v30, v1
+; GFX9-SDAG-NEXT:    v_subb_co_u32_e32 v11, vcc, v11, v14, vcc
+; GFX9-SDAG-NEXT:    v_add_co_u32_e32 v22, vcc, -1, v22
+; GFX9-SDAG-NEXT:    v_addc_co_u32_e32 v23, vcc, -1, v23, vcc
+; GFX9-SDAG-NEXT:    v_addc_co_u32_e32 v24, vcc, -1, v24, vcc
+; GFX9-SDAG-NEXT:    v_addc_co_u32_e32 v25, vcc, -1, v25, vcc
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v5, v15, v5
+; GFX9-SDAG-NEXT:    v_lshlrev_b64 v[2:3], 1, v[2:3]
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v14, v22, v24
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v15, v23, v25
+; GFX9-SDAG-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[14:15]
+; GFX9-SDAG-NEXT:    v_or3_b32 v2, v2, v6, v12
+; GFX9-SDAG-NEXT:    v_and_b32_e32 v6, 1, v30
+; GFX9-SDAG-NEXT:    v_mov_b32_e32 v15, v7
+; GFX9-SDAG-NEXT:    v_or3_b32 v3, v3, 0, v13
+; GFX9-SDAG-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
+; GFX9-SDAG-NEXT:    v_mov_b32_e32 v14, v6
+; GFX9-SDAG-NEXT:    s_andn2_b64 exec, exec, s[4:5]
+; GFX9-SDAG-NEXT:    s_cbranch_execnz .LBB0_3
+; GFX9-SDAG-NEXT:  ; %bb.4: ; %Flow
+; GFX9-SDAG-NEXT:    s_or_b64 exec, exec, s[4:5]
+; GFX9-SDAG-NEXT:  .LBB0_5: ; %Flow2
+; GFX9-SDAG-NEXT:    s_or_b64 exec, exec, s[6:7]
+; GFX9-SDAG-NEXT:    v_lshlrev_b64 v[0:1], 1, v[4:5]
+; GFX9-SDAG-NEXT:    v_lshlrev_b64 v[2:3], 1, v[2:3]
+; GFX9-SDAG-NEXT:    v_lshrrev_b32_e32 v4, 31, v5
+; GFX9-SDAG-NEXT:    v_or3_b32 v13, v3, 0, v13
+; GFX9-SDAG-NEXT:    v_or3_b32 v12, v2, v4, v12
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v7, v7, v1
+; GFX9-SDAG-NEXT:    v_or_b32_e32 v6, v6, v0
+; GFX9-SDAG-NEXT:  .LBB0_6: ; %Flow3
+; GFX9-SDAG-NEXT:    s_or_b64 exec, exec, s[8:9]
+; GFX9-SDAG-NEXT:    v_xor_b32_e32 v2, v17, v16
+; GFX9-SDAG-NEXT:    v_xor_b32_e32 v3, v19, v18
+; GFX9-SDAG-NEXT:    v_xor_b32_e32 v0, v6, v2
+; GFX9-SDAG-NEXT:    v_xor_b32_e32 v1, v7, v3
+; GFX9-SDAG-NEXT:    v_sub_co_u32_e32 v0, vcc, v0, v2
+; GFX9-SDAG-NEXT:    v_xor_b32_e32 v5, v12, v2
+; GFX9-SDAG-NEXT:    v_subb_co_u32_e32 v1, vcc, v1, v3, vcc
+; GFX9-SDAG-NEXT:    v_xor_b32_e32 v4, v13, v3
+; GFX9-SDAG-NEXT:    v_subb_co_u32_e32 v2, vcc, v5, v2, vcc
+; GFX9-SDAG-NEXT:    v_subb_co_u32_e32 v3, vcc, v4, v3, vcc
+; GFX9-SDAG-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX9-GISEL-LABEL: v_sdiv_i128_vv:
+; GFX9-GISEL:       ; %bb.0: ; %_udiv-special-cases
+; GFX9-GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
+; GFX9-GISEL-NEXT:    v_ashrrev_i32_e32 v16, 31, v3
+; GFX9-GISEL-NEXT:    v_xor_b32_e32 v0, v16, v0
+; GFX9-GISEL-NEXT:    v_xor_b32_e32 v1, v16, v1
+; GFX9-GISEL-NEXT:    v_sub_co_u32_e32 v10, vcc, v0, v16
+; GFX9-GISEL-NEXT:    v_xor_b32_e32 v2, v16, v2
+; GFX9-GISEL-NEXT:    v_subb_co_u32_e32 v11, vcc, v1, v16, vcc
+; GFX9-GISEL-NEXT:    v_ashrrev_i32_e32 v17, 31, v7
+; GFX9-GISEL-NEXT:    v_xor_b32_e32 v3, v16, v3
+; GFX9-GISEL-NEXT:    v_subb_co_u32_e32 v12, vcc, v2, v16, vcc
+; GFX9-GISEL-NEXT:    v_subb_co_u32_e32 v13, vcc, v3, v16, vcc
+; GFX9-GISEL-NEXT:    v_xor_b32_e32 v0, v17, v4
+; GFX9-GISEL-NEXT:    v_xor_b32_e32 v1, v17, v5
+; GFX9-GISEL-NEXT:    v_sub_co_u32_e32 v18, vcc, v0, v17
+; GFX9-GISEL-NEXT:    v_xor_b32_e32 v2, v17, v6
+; GFX9-GISEL-NEXT:    v_subb_co_u32_e32 v19, vcc, v1, v17, vcc
+; GFX9-GISEL-NEXT:    v_xor_b32_e32 v3, v17, v7
+; GFX9-GISEL-NEXT:    v_subb_co_u32_e32 v4, vcc, v2, v17, vcc
+; GFX9-GISEL-NEXT:    v_subb_co_u32_e32 v5, vcc, v3, v17, vcc
+; GFX9-GISEL-NEXT:    v_or_b32_e32 v0, v18, v4
+; GFX9-GISEL-NEXT:    v_or_b32_e32 v1, v19, v5
+; GFX9-GISEL-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[0:1]
+; GFX9-GISEL-NEXT:    v_or_b32_e32 v0, v10, v12
+; GFX9-GISEL-NEXT:    v_or_b32_e32 v1, v11, v13
+; GFX9-GISEL-NEXT:    v_cmp_eq_u64_e64 s[4:5], 0, v[0:1]
+; GFX9-GISEL-NEXT:    v_ffbh_u32_e32 v1, v18
+; GFX9-GISEL-NEXT:    v_ffbh_u32_e32 v0, v19
+; GFX9-GISEL-NEXT:    v_add_u32_e32 v1, 32, v1
+; GFX9-GISEL-NEXT:    v_ffbh_u32_e32 v2, v4
+; GFX9-GISEL-NEXT:    v_min_u32_e32 v0, v0, v1
+; GFX9-GISEL-NEXT:    v_ffbh_u32_e32 v1, v5
+; GFX9-GISEL-NEXT:    v_add_u32_e32 v2, 32, v2
+; GFX9-GISEL-NEXT:    v_cmp_eq_u64_e64 s[6:7], 0, v[4:5]
+; GFX9-GISEL-NEXT:    v_add_u32_e32 v0, 64, v0
+; GFX9-GISEL-NEXT:    v_min_u32_e32 v1, v1, v2
+; GFX9-GISEL-NEXT:    v_ffbh_u32_e32 v2, v10
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e64 v0, v1, v0, s[6:7]
+; GFX9-GISEL-NEXT:    v_ffbh_u32_e32 v1, v11
+; GFX9-GISEL-NEXT:    v_add_u32_e32 v2, 32, v2
+; GFX9-GISEL-NEXT:    v_ffbh_u32_e32 v3, v12
+; GFX9-GISEL-NEXT:    v_min_u32_e32 v1, v1, v2
+; GFX9-GISEL-NEXT:    v_ffbh_u32_e32 v2, v13
+; GFX9-GISEL-NEXT:    v_add_u32_e32 v3, 32, v3
+; GFX9-GISEL-NEXT:    v_cmp_eq_u64_e64 s[6:7], 0, v[12:13]
+; GFX9-GISEL-NEXT:    v_add_u32_e32 v1, 64, v1
+; GFX9-GISEL-NEXT:    v_min_u32_e32 v2, v2, v3
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e64 v1, v2, v1, s[6:7]
+; GFX9-GISEL-NEXT:    v_sub_co_u32_e64 v0, s[6:7], v0, v1
+; GFX9-GISEL-NEXT:    v_subb_co_u32_e64 v1, s[6:7], 0, 0, s[6:7]
+; GFX9-GISEL-NEXT:    v_mov_b32_e32 v6, 0x7f
+; GFX9-GISEL-NEXT:    v_subb_co_u32_e64 v2, s[6:7], 0, 0, s[6:7]
+; GFX9-GISEL-NEXT:    v_mov_b32_e32 v7, 0
+; GFX9-GISEL-NEXT:    v_subb_co_u32_e64 v3, s[6:7], 0, 0, s[6:7]
+; GFX9-GISEL-NEXT:    v_cmp_gt_u64_e64 s[6:7], v[0:1], v[6:7]
+; GFX9-GISEL-NEXT:    s_or_b64 s[4:5], vcc, s[4:5]
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e64 v6, 0, 1, s[6:7]
+; GFX9-GISEL-NEXT:    v_cmp_lt_u64_e64 s[6:7], 0, v[2:3]
+; GFX9-GISEL-NEXT:    v_or_b32_e32 v15, v1, v3
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, s[6:7]
+; GFX9-GISEL-NEXT:    v_cmp_eq_u64_e64 s[6:7], 0, v[2:3]
+; GFX9-GISEL-NEXT:    s_mov_b64 s[8:9], 0
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e64 v6, v7, v6, s[6:7]
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, s[4:5]
+; GFX9-GISEL-NEXT:    v_or_b32_e32 v20, v7, v6
+; GFX9-GISEL-NEXT:    v_xor_b32_e32 v6, 0x7f, v0
+; GFX9-GISEL-NEXT:    v_or_b32_e32 v14, v6, v2
+; GFX9-GISEL-NEXT:    v_and_b32_e32 v6, 1, v20
+; GFX9-GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v6
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e64 v6, v10, 0, vcc
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e64 v7, v11, 0, vcc
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e64 v8, v12, 0, vcc
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e64 v9, v13, 0, vcc
+; GFX9-GISEL-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[14:15]
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e64 v14, 0, 1, vcc
+; GFX9-GISEL-NEXT:    v_or_b32_e32 v14, v20, v14
+; GFX9-GISEL-NEXT:    v_and_b32_e32 v14, 1, v14
+; GFX9-GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v14
+; GFX9-GISEL-NEXT:    s_xor_b64 s[4:5], vcc, -1
+; GFX9-GISEL-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
+; GFX9-GISEL-NEXT:    s_cbranch_execz .LBB0_6
+; GFX9-GISEL-NEXT:  ; %bb.1: ; %udiv-bb1
+; GFX9-GISEL-NEXT:    v_add_co_u32_e32 v20, vcc, 1, v0
+; GFX9-GISEL-NEXT:    v_addc_co_u32_e32 v21, vcc, 0, v1, vcc
+; GFX9-GISEL-NEXT:    v_addc_co_u32_e32 v22, vcc, 0, v2, vcc
+; GFX9-GISEL-NEXT:    v_addc_co_u32_e32 v23, vcc, 0, v3, vcc
+; GFX9-GISEL-NEXT:    s_xor_b64 s[4:5], vcc, -1
+; GFX9-GISEL-NEXT:    v_sub_co_u32_e32 v8, vcc, 0x7f, v0
+; GFX9-GISEL-NEXT:    v_sub_u32_e32 v0, 64, v8
+; GFX9-GISEL-NEXT:    v_lshrrev_b64 v[0:1], v0, v[10:11]
+; GFX9-GISEL-NEXT:    v_lshlrev_b64 v[2:3], v8, v[12:13]
+; GFX9-GISEL-NEXT:    v_subrev_u32_e32 v9, 64, v8
+; GFX9-GISEL-NEXT:    v_lshlrev_b64 v[6:7], v8, v[10:11]
+; GFX9-GISEL-NEXT:    v_or_b32_e32 v2, v0, v2
+; GFX9-GISEL-NEXT:    v_or_b32_e32 v3, v1, v3
+; GFX9-GISEL-NEXT:    v_lshlrev_b64 v[0:1], v9, v[10:11]
+; GFX9-GISEL-NEXT:    v_cmp_gt_u32_e32 vcc, 64, v8
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e32 v6, 0, v6, vcc
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e32 v7, 0, v7, vcc
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e32 v1, v1, v3, vcc
+; GFX9-GISEL-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v8
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e32 v8, v0, v12, vcc
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e32 v9, v1, v13, vcc
+; GFX9-GISEL-NEXT:    s_mov_b64 s[10:11], s[8:9]
+; GFX9-GISEL-NEXT:    v_mov_b32_e32 v0, s8
+; GFX9-GISEL-NEXT:    v_mov_b32_e32 v1, s9
+; GFX9-GISEL-NEXT:    v_mov_b32_e32 v2, s10
+; GFX9-GISEL-NEXT:    v_mov_b32_e32 v3, s11
+; GFX9-GISEL-NEXT:    s_and_saveexec_b64 s[8:9], s[4:5]
+; GFX9-GISEL-NEXT:    s_xor_b64 s[12:13], exec, s[8:9]
+; GFX9-GISEL-NEXT:    s_cbranch_execz .LBB0_5
+; GFX9-GISEL-NEXT:  ; %bb.2: ; %udiv-preheader
+; GFX9-GISEL-NEXT:    v_sub_u32_e32 v2, 64, v20
+; GFX9-GISEL-NEXT:    v_lshrrev_b64 v[0:1], v20, v[10:11]
+; GFX9-GISEL-NEXT:    v_lshlrev_b64 v[2:3], v2, v[12:13]
+; GFX9-GISEL-NEXT:    v_subrev_u32_e32 v24, 64, v20
+; GFX9-GISEL-NEXT:    v_lshrrev_b64 v[14:15], v20, v[12:13]
+; GFX9-GISEL-NEXT:    v_or_b32_e32 v2, v0, v2
+; GFX9-GISEL-NEXT:    v_or_b32_e32 v3, v1, v3
+; GFX9-GISEL-NEXT:    v_lshrrev_b64 v[0:1], v24, v[12:13]
+; GFX9-GISEL-NEXT:    v_cmp_gt_u32_e32 vcc, 64, v20
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e32 v1, v1, v3, vcc
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e32 v14, 0, v14, vcc
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e32 v15, 0, v15, vcc
+; GFX9-GISEL-NEXT:    v_add_co_u32_e32 v24, vcc, -1, v18
+; GFX9-GISEL-NEXT:    s_mov_b64 s[8:9], 0
+; GFX9-GISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v20
+; GFX9-GISEL-NEXT:    v_addc_co_u32_e32 v25, vcc, -1, v19, vcc
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e64 v12, v0, v10, s[4:5]
+; GFX9-GISEL-NEXT:    v_cndmask_b32_e64 v13, v1, v11, s[4:5]
+; GFX9-GISEL-NEXT:    v_addc_co_u32_e32 v26, vcc, -1, v4, vcc
+; GFX9-GISEL-NEXT:    s_mov_b64 s[10:11], s[8:9]
+; GFX9-GISEL-NEXT:    v_mov_b32_e32 v0, s8
+; GFX9-GISEL-NEXT:    v_addc_co_u32_e32 v27, vcc, -1, v5, vcc
+; GFX9-GISEL-NEXT:    v_mov_b32_e32 v11, 0
+; GFX9-GISEL-NEXT:    v_mov_b32_e32 v1, s9
+; GFX9-GISEL-NEXT:    v_mov_b32_e32 v2, s10
+; GFX9-GISEL-NEXT:    v_mov_b32_e32 v3, s11
+; GFX9-GISEL-NEXT:  .LBB0_3: ; %udiv-do-while
+; GFX9-GISEL-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-GISEL-NEXT:    v_lshlrev_b64 v[2:3], 1, v[6:7]
+; GFX9-GISEL-NEXT:    v_lshrrev_b32_e32 v10, 31, v7
+; GFX9-GISEL-NEXT:    v_or_b32_e32 v6, v0, v2
+; GFX9-GISEL-NEXT:    v_or_b32_e32 v7, v1, v3
+; GFX9-GISEL-NEXT:    v_lshlrev_b64 v[2:3], 1, v[12:13]
+; GFX9-GISEL-NEXT:    v_lshrrev_b32_e32 v12, 31, v9
+; GFX9-GISEL-NEXT:    v_lshlrev_b64 v[0:1], 1, v[14:15]
+; GFX9-GISEL-NEXT:    v_or_b32_e32 v2, v2, v12
+; G...
[truncated]

arsenm
arsenm commented Feb 5, 2024
View reviewed changes
Copy link
Contributor Author

@arsenm arsenm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably should add tests with power of 2 constant tests since the pass specifically avoids handling them. I'm a bit uncomfortable relying on that type of optimization though

yxsamliu
yxsamliu approved these changes Feb 5, 2024
View reviewed changes
Copy link
Collaborator

@yxsamliu yxsamliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks. power of 2 tests could be added later

@arsenm arsenm merged commit a5d206d into llvm:main Feb 5, 2024
4 of 5 checks passed
@arsenm arsenm deleted the amdgpu-div-i128 branch February 5, 2024 13:39
llvmbot pushed a commit to llvmbot/llvm-project that referenced this pull request Feb 5, 2024
@arsenm @llvmbot
AMDGPU: Set max supported div/rem size to 64 (llvm#80669)
2ee5c9d 
This enables IR expansion for i128 divisions. The vector case is still
broken because ExpandLargeDivRem doesn't try to handle them.

Fixes: SWDEV-426193
(cherry picked from commit a5d206d)
agozillon pushed a commit to agozillon/llvm-project that referenced this pull request Feb 5, 2024
@arsenm @agozillon
AMDGPU: Set max supported div/rem size to 64 (llvm#80669)
95efcfb 
This enables IR expansion for i128 divisions. The vector case is still
broken because ExpandLargeDivRem doesn't try to handle them.

Fixes: SWDEV-426193
llvmbot pushed a commit to llvmbot/llvm-project that referenced this pull request Feb 5, 2024
@arsenm @tstellar
AMDGPU: Set max supported div/rem size to 64 (llvm#80669)
c072ff2 
This enables IR expansion for i128 divisions. The vector case is still
broken because ExpandLargeDivRem doesn't try to handle them.

Fixes: SWDEV-426193
(cherry picked from commit a5d206d)
searlmc1 pushed a commit to ROCm/llvm-project that referenced this pull request Feb 12, 2024
@arsenm @searlmc1
AMDGPU: Set max supported div/rem size to 64 (llvm#80669)
59cf9c8 
This enables IR expansion for i128 divisions. The vector case is still
broken because ExpandLargeDivRem doesn't try to handle them.

Fixes: SWDEV-426193
Change-Id: Ie1bf0f3ae1d199a120a75dc663c39642308ecec9
tstellar pushed a commit to tstellar/llvm-project that referenced this pull request Feb 14, 2024
@arsenm @tstellar
AMDGPU: Set max supported div/rem size to 64 (llvm#80669)
47ccb52 
This enables IR expansion for i128 divisions. The vector case is still
broken because ExpandLargeDivRem doesn't try to handle them.

Fixes: SWDEV-426193
(cherry picked from commit a5d206d)
tstellar pushed a commit to tstellar/llvm-project that referenced this pull request Feb 14, 2024
@arsenm @tstellar
AMDGPU: Set max supported div/rem size to 64 (llvm#80669)
5c99f5c 
This enables IR expansion for i128 divisions. The vector case is still
broken because ExpandLargeDivRem doesn't try to handle them.

Fixes: SWDEV-426193
(cherry picked from commit a5d206d)
tstellar pushed a commit to tstellar/llvm-project that referenced this pull request Feb 14, 2024
@arsenm @tstellar
AMDGPU: Set max supported div/rem size to 64 (llvm#80669)
38ea7b5 
This enables IR expansion for i128 divisions. The vector case is still
broken because ExpandLargeDivRem doesn't try to handle them.

Fixes: SWDEV-426193
(cherry picked from commit a5d206d)
tstellar pushed a commit to tstellar/llvm-project that referenced this pull request Feb 14, 2024
@arsenm @tstellar
AMDGPU: Set max supported div/rem size to 64 (llvm#80669)
fe425eb 
This enables IR expansion for i128 divisions. The vector case is still
broken because ExpandLargeDivRem doesn't try to handle them.

Fixes: SWDEV-426193
(cherry picked from commit a5d206d)
@pointhex pointhex mentioned this pull request May 7, 2024
Closed
@aemerson aemerson mentioned this pull request May 9, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yxsamliu yxsamliu yxsamliu approved these changes

@Pierre-vh Pierre-vh Awaiting requested review from Pierre-vh

Assignees
No one assigned
Labels
backend:AMDGPU
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@arsenm @llvmbot @yxsamliu