Assertion failure in checkPointerTypesForAssignment due to non-canonical RHS type

[arunkumarbhattar]

Description:
I encountered an assertion failure in the Clang compiler (Checked C branch) while compiling a Checked-C program. The assertion is triggered in the checkPointerTypesForAssignment function in SemaExpr.cpp, specifically at line 9155, where it checks if the RHSType is canonical.

Assertion `RHSType.isCanonical() && "RHS not canonicalized!"' failed.

The error message suggests submitting a bug report with the crash backtrace, preprocessed source, and associated run script. Here are the relevant details:

Crash backtrace:
clang-12: /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Sema/SemaExpr.cpp:9155: clang::Sema::AssignConvertType checkPointerTypesForAssignment(clang::Sema&, clang::QualType, clang::QualType): Assertion `RHSType.isCanonical() && "RHS not canonicalized!"' failed.
PLEASE submit a bug report to https://github.com/Microsoft/checkedc-clang/issues and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0. Program arguments: /home/arun/Desktop/nonmacro/checkedc-llvm-project/llvm/cmake-build-debug/bin/clang-12 -cc1 -triple x86_64-unknown-linux-gnu -emit-obj -mrelax-all --mrelax-relocations -disable-free -main-file-name crash.c -mrelocation-model static -mframe-pointer=all -fmath-errno -fno-rounding-math -mconstructor-aliases -munwind-tables -target-cpu x86-64 -tune-cpu generic -fno-split-dwarf-inlining -debugger-tuning=gdb -resource-dir /home/arun/Desktop/nonmacro/checkedc-llvm-project/llvm/cmake-build-debug/lib/clang/12.0.0 -internal-isystem /usr/local/include -internal-isystem /home/arun/Desktop/nonmacro/checkedc-llvm-project/llvm/cmake-build-debug/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdebug-compilation-dir /home/arun/Desktop/testC/crashSetting -ferror-limit 19 -fgnuc-version=4.2.1 -fcolor-diagnostics -faddrsig -o /tmp/crash-326fee.o -x c crash.c

1. crash.c:72:73: current parser token ')'
2. crash.c:65:1: parsing function body 'mineVariableServiceGetVariable'
3. crash.c:65:1: in compound statement ('{}')
   #0 0x000055bdd6eb6768 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /home/arun/Desktop/nonmacro/checkedc-llvm-project/llvm/lib/Support/Unix/Signals.inc:565:22
   Merge-back to respective upstreams? #1 0x000055bdd6eb682f PrintStackTraceSignalHandler(void*) /home/arun/Desktop/nonmacro/checkedc-llvm-project/llvm/lib/Support/Unix/Signals.inc:632:1
   Update setup and build instructions. #2 0x000055bdd6eb43f1 llvm::sys::RunSignalHandlers() /home/arun/Desktop/nonmacro/checkedc-llvm-project/llvm/lib/Support/Signals.cpp:71:20
   test serialization/deserialization of clang IR with new checked types and new bounds expressions #3 0x000055bdd6eb6098 SignalHandler(int) /home/arun/Desktop/nonmacro/checkedc-llvm-project/llvm/lib/Support/Unix/Signals.inc:407:1
   test AST printing of clang IR with new checked C types and bounds expressions #4 0x00007fa2cb242520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x42520)
   test traversal of ASTs with new checked types and bounds expressions #5 0x00007fa2cb296a7c __pthread_kill_implementation ./nptl/./nptl/pthread_kill.c:44:76
   test tree transformations of ASTs with new checked types and bounds expressions #6 0x00007fa2cb296a7c __pthread_kill_internal ./nptl/./nptl/pthread_kill.c:78:10
   process bounds expressions for parameters with all the parameters available #7 0x00007fa2cb296a7c pthread_kill ./nptl/./nptl/pthread_kill.c:89:10
   Extend clang IR with bounds expressions and parse bounds expressions for parameters. #8 0x00007fa2cb242476 raise ./signal/../sysdeps/posix/raise.c:27:6
   Only allow Checked C extension flag for C in clang #9 0x00007fa2cb2287f3 abort ./stdlib/./stdlib/abort.c:81:7
   Plan out testing clang internal features with new Checked C features #10 0x00007fa2cb22871b _nl_load_domain ./intl/./intl/loadmsgcat.c:1177:9
   Extend IR with bounds expressions #11 0x00007fa2cb239e96 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x39e96)
   Parse bounds declarations for function parameters #12 0x000055bddac8fd5c checkPointerTypesForAssignment(clang::Sema&, clang::QualType, clang::QualType) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Sema/SemaExpr.cpp:9159:15
   Parse bounds declarations for variable declarations #13 0x000055bddac91667 clang::Sema::CheckAssignmentConstraints(clang::QualType, clang::ActionResult<clang::Expr*, true>&, clang::CastKind&, bool) (.localalias) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Sema/SemaExpr.cpp:9663:68
   Type check non-count bounds expressions #14 0x000055bddac92f57 clang::Sema::CheckSingleAssignmentConstraints(clang::QualType, clang::ActionResult<clang::Expr*, true>&, bool, bool, bool, clang::QualType) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Sema/SemaExpr.cpp:10083:31
   Parse bounds declarations for function return values #15 0x000055bddafca841 clang::InitializationSequence::Perform(clang::Sema&, clang::InitializedEntity const&, clang::InitializationKind const&, llvm::MutableArrayRefclang::Expr*, clang::QualType*) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Sema/SemaInit.cpp:8566:43
   Only allow the Checked C extension flag for C. #16 0x000055bddafd10da clang::Sema::PerformCopyInitialization(clang::InitializedEntity const&, clang::SourceLocation, clang::ActionResult<clang::Expr*, true>, bool, bool) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Sema/SemaInit.cpp:9924:34
   Parse bounds expressions in function parameter list scopes. #17 0x000055bddac81261 clang::Sema::GatherArgumentsForCall(clang::SourceLocation, clang::FunctionDecl*, clang::FunctionProtoType const*, unsigned int, llvm::ArrayRefclang::Expr*, llvm::SmallVectorImplclang::Expr*&, clang::Sema::VariadicCallType, bool, bool) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Sema/SemaExpr.cpp:6110:50
   Parse bounds declarations for variable declarations. #18 0x000055bddac80d9e clang::Sema::ConvertArgumentsForCall(clang::CallExpr*, clang::Expr*, clang::FunctionDecl*, clang::FunctionProtoType const*, llvm::ArrayRefclang::Expr*, clang::SourceLocation, bool) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Sema/SemaExpr.cpp:6049:35
   Parse bounds declarations for struct members #19 0x000055bddac85228 clang::Sema::BuildResolvedCallExpr(clang::Expr*, clang::NamedDecl*, clang::SourceLocation, llvm::ArrayRefclang::Expr*, clang::SourceLocation, clang::Expr*, bool, clang::CallExpr::ADLCallKind) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Sema/SemaExpr.cpp:6947:32
   Represent bounds information in function types #20 0x000055bddac84074 clang::Sema::BuildCallExpr(clang::Scope*, clang::Expr*, clang::SourceLocation, llvm::MutableArrayRefclang::Expr*, clang::SourceLocation, clang::Expr*, bool, bool) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Sema/SemaExpr.cpp:6738:31
   Parse bounds declarations for function return values. #21 0x000055bddac82c00 clang::Sema::ActOnCallExpr(clang::Scope*, clang::Expr*, clang::SourceLocation, llvm::MutableArrayRefclang::Expr*, clang::SourceLocation, clang::Expr*) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Sema/SemaExpr.cpp:6514:20
   Fix crash during typo correction in C mode. #22 0x000055bdda1999cf clang::Parser::ParsePostfixExpressionSuffix(clang::ActionResult<clang::Expr*, true>) (.localalias) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Parse/ParseExpr.cpp:2082:36
   Parsing of member bounds declarations. #23 0x000055bdda198384 clang::Parser::ParseCastExpression(clang::Parser::CastParseKind, bool, bool&, clang::Parser::TypeCastState, bool, bool*) (.localalias) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Parse/ParseExpr.cpp:1821:37
   Allow member bounds to use enumeration constants #24 0x000055bdda1937c0 clang::Parser::ParseCastExpression(clang::Parser::CastParseKind, bool, clang::Parser::TypeCastState, bool, bool*) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Parse/ParseExpr.cpp:683:39
   Update status and add code of conduct. #25 0x000055bdda1929d6 clang::Parser::ParseRHSOfBinaryExpression(clang::ActionResult<clang::Expr*, true>, clang::prec::Level) (.localalias) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Parse/ParseExpr.cpp:548:32
   Type check count bounds expressions #26 0x000055bdda190f57 clang::Parser::ParseAssignmentExpression(clang::Parser::TypeCastState) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Parse/ParseExpr.cpp:178:58
   Remove declaring unchecked arrays using the 'unchecked' keyword #27 0x000055bdda190c57 clang::Parser::ParseExpression(clang::Parser::TypeCastState) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Parse/ParseExpr.cpp:124:43
   Check bounds declarations for pointers to constant-sized data for a subset of expressions #28 0x000055bdda20f360 clang::Parser::ParseExprStatement(clang::Parser::ParsedStmtContext) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Parse/ParseStmt.cpp:491:34
   Parse and represent 'ptr' interop annotation for unchecked pointer types #29 0x000055bdda20e596 clang::Parser::ParseStatementOrDeclarationAfterAttributes(llvm::SmallVector<clang::Stmt*, 32u>&, clang::Parser::ParsedStmtContext, clang::SourceLocation*, clang::Parser::ParsedAttributesWithRange&) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Parse/ParseStmt.cpp:254:30
   Handle redeclarations of functions and variables with bounds #30 0x000055bdda20dc33 clang::Parser::ParseStatementOrDeclaration(llvm::SmallVector<clang::Stmt*, 32u>&, clang::Parser::ParsedStmtContext, clang::SourceLocation*) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Parse/ParseStmt.cpp:106:62
   Implement implicit conversions for ptr interop annotations #31 0x000055bdda211d22 clang::Parser::ParseCompoundStatementBody(bool, clang::CheckedScopeSpecifier, clang::SourceLocation, clang::SourceLocation, clang::SourceLocation) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Parse/ParseStmt.cpp:1199:38
   Dimensions of multi-dimensional arrays must have consistent checked properties #32 0x000055bdda2166e9 clang::Parser::ParseFunctionStatementBody(clang::Decl*, clang::Parser::ParseScope&, clang::CheckedScopeSpecifier) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Parse/ParseStmt.cpp:2362:47
   Initial version of the checked-c conversion tool. Current features: #33 0x000055bdda126376 clang::Parser::ParseFunctionDefinition(clang::ParsingDeclarator&, clang::Parser::ParsedTemplateInfo const&, clang::Parser::LateParsedAttrList*) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Parse/Parser.cpp:1438:36
   Rewrite tool #34 0x000055bdda1462b5 clang::Parser::ParseDeclGroup(clang::ParsingDeclSpec&, clang::DeclaratorContext, clang::SourceLocation*, clang::Parser::ForRangeInit*) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Parse/ParseDecl.cpp:2066:36
   Create documentation for the rewriter tool #35 0x000055bdda124fdc clang::Parser::ParseDeclOrFunctionDefInternal(clang::Parser::ParsedAttributesWithRange&, clang::ParsingDeclSpec&, clang::AccessSpecifier) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Parse/Parser.cpp:1173:24
   Typecheck count bounds expressions. #36 0x000055bdda1250d1 clang::Parser::ParseDeclarationOrFunctionDefinition(clang::Parser::ParsedAttributesWithRange&, clang::ParsingDeclSpec*, clang::AccessSpecifier) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Parse/Parser.cpp:1189:42
   Add support for relative bounds expressions #37 0x000055bdda124420 clang::Parser::ParseExternalDeclaration(clang::Parser::ParsedAttributesWithRange&, clang::ParsingDeclSpec*) (.localalias) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Parse/Parser.cpp:991:48
   Check that subexpressions of bounds expressions are non-modifying expressions. #38 0x000055bdda123496 clang::Parser::ParseTopLevelDecl(clang::OpaquePtrclang::DeclGroupRef&, bool) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Parse/Parser.cpp:737:36
   Implement storage class restrictions on bounds declarations #39 0x000055bdda11e823 clang::ParseAST(clang::Sema&, bool, bool) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Parse/ParseAST.cpp:158:37
   Implement restrictions on variables at external scope #40 0x000055bdd7be4bf7 clang::ASTFrontendAction::ExecuteAction() /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Frontend/FrontendAction.cpp:1056:11
   Checked additonal type requirements on expressions with count bounds expressions #41 0x000055bdd85dfab6 clang::CodeGenAction::ExecuteAction() /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/CodeGen/CodeGenAction.cpp:1083:5
   Check type requirements for bounds declarations. #42 0x000055bdd7be44a6 clang::FrontendAction::Execute() /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Frontend/FrontendAction.cpp:953:38
   Initial support for linking constraints #43 0x000055bdd7b38c30 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/Frontend/CompilerInstance.cpp:949:42
   Bug fixes and modest extensions #44 0x000055bdd7d85172 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp:278:38
   Type check bounds expression arguments #45 0x000055bdd3cdeb1a cc1_main(llvm::ArrayRef<char const*>, char const*, void*) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/tools/driver/cc1_main.cpp:240:40
   No more test baselines #46 0x000055bdd3cd2997 ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&) /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/tools/driver/driver.cpp:330:20
   Statistics printing and typedef changes #47 0x000055bdd3cd318c main /home/arun/Desktop/nonmacro/checkedc-llvm-project/clang/tools/driver/driver.cpp:407:26
   Replace logic in NewTyp::mkTypForConstrainedType with a recursive AST visitor #48 0x00007fa2cb229d90 __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
   Re-visit re-writing variable declarations with a typedefed type #49 0x00007fa2cb229e40 call_init ./csu/../csu/libc-start.c:128:20
   Revamp kinds for bounds expressions #50 0x00007fa2cb229e40 __libc_start_main ./csu/../csu/libc-start.c:379:5
   Update setup and build instructions. #51 0x000055bdd3cd0fb5 _start (/home/arun/Desktop/nonmacro/checkedc-llvm-project/llvm/cmake-build-debug/bin/clang-12+0x148dfb5)
   clang-12: error: unable to execute command: Aborted (core dumped)
   clang-12: error: clang frontend command failed due to signal (use -v to see invocation)
   clang version 12.0.0 ([email protected]:secure-sw-dev/checkedc-llvm-project.git ebf531a)
   Target: x86_64-unknown-linux-gnu
   Thread model: posix
   InstalledDir: /home/arun/Desktop/nonmacro/checkedc-llvm-project/llvm/cmake-build-debug/bin
   clang-12: note: diagnostic msg:

---

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-12: note: diagnostic msg: /tmp/crash-158445.c
clang-12: note: diagnostic msg: /tmp/crash-158445.sh
clang-12: note: diagnostic msg:

---

Source code:
(base) arun@arun-GL65-Leopard-10SEK:~/Desktop/testC/crashSetting$ cat crash.c
#include <stdlib.h>

typedef char CHAR16;

#define IN
#define OUT
#define OPTIONAL

int
FindAccessVariable (
IN _Nt_array_ptr VariableName
)
{

return -1;
}

EFI_STATUS
mineVariableServiceSetVariable (
IN _Nt_array_ptr VariableName
)
{
FindAccessVariable (VariableName);
return 1;
}

EFI_STATUS
mineVariableServiceGetVariable (
IN CHAR16 *VariableName : itype(_Nt_array_ptr)
)
{
return FindAccessVariable (VariableName);
}

int main()
{
return 0;

}

Steps to reproduce:
clang crash.c

It would be helpful to investigate the cause of this assertion failure and determine if there is an issue in the compiler's handling of non-canonical types, or if there is an issue in the input source code itself that leads to the assertion being triggered.

Possible solutions could involve ensuring that both the LHS and RHS types are canonicalized before comparing them in checkPointerTypesForAssignment.