[0017] Investigate catastrophic failures more cloesly

[llvm-beanz]

Which proposal does this relate to?
0017 Conforming Literals

Describe the issue or outstanding question.
During experimentation with existing shaders we discovered several cases with catastrophic rendering defects. We should investigate them to determine if we need to do additional mitigations.

[llvm-beanz]

Investigating this a bit further this is not at all what I expected. The problematic behavior change occurs with code like this:

export float Fn(int inInt, inFloat) {
  int Val = (inInt & 0xffff0000) >> 16);
  return Val* inFloat
}

Compiler Explorer

With HLSL's literal int type 0xffff0000 is signed, which means the bit shifts are arithmetic shifts filling in the sign bit.

define float @"\01?Fn@@YAMHM@Z"(i32 %inInt, float %inFloat) #0 {
  %1 = ashr i32 %inInt, 16
  %2 = sitofp i32 %1 to float
  %3 = fmul fast float %2, %inFloat
  ret float %3
}

With C's rules 0xffff0000 is unsigned, and the logical operators produce an unsigned result with 0's shifted in at the most significant bits.

define float @"\01?Fn@@YAMHM@Z"(i32 %inInt, float %inFloat) #0 {
  %1 = lshr i32 %inInt, 16
  %2 = sitofp i32 %1 to float
  %3 = fmul fast float %2, %inFloat
  ret float %3
}