Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[cling] Create new CompoundStmt instead of replacing children #7488

Merged
merged 3 commits into from
Apr 23, 2021

Conversation

hahnjo
Copy link
Member

@hahnjo hahnjo commented Mar 12, 2021

For the update of LLVM 9, Cling required another patch to Clang for replacing the children of a CompoundStmt. Instead solve this by creating a new CompoundStmt with the right Stmts attached.

@hahnjo hahnjo self-assigned this Mar 12, 2021
@phsft-bot
Copy link
Collaborator

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

@hahnjo
Copy link
Member Author

hahnjo commented Mar 12, 2021

The idea for this change comes from my own experiments with upgrading LLVM during my time as student worker (and I can't reliably tell if I wrote the code during paid hours or in my free time, hence the funny list of authorships). It would allow to drop one possibly contentious patch for Clang, so I wanted to get the change out for feedback.

One thing I currently don't understand are two test failures in gtest-math-mathcore-test-CladDerivatorTests and gtest-hist-hist-test-TFormulaGradientTests - but only with the second commit that removes the then unneeded patch. I can make the tests pass again by adding back only the WasReplaced bit:

diff --git a/interpreter/llvm/src/tools/clang/include/clang/AST/Stmt.h b/interpreter/llvm/src/tools/clang/include/clang/AST/Stmt.h
index 403b88ac3a..5e1581f098 100644
--- a/interpreter/llvm/src/tools/clang/include/clang/AST/Stmt.h
+++ b/interpreter/llvm/src/tools/clang/include/clang/AST/Stmt.h
@@ -131,7 +131,8 @@ protected:
 
     unsigned : NumStmtBits;
 
-    unsigned NumStmts : 32 - NumStmtBits;
+    unsigned WasReplaced : 1;
+    unsigned NumStmts : 32 - (NumStmtBits + 1);
 
     /// The location of the opening "{".
     SourceLocation LBraceLoc;
@@ -1328,6 +1329,7 @@ public:
   explicit CompoundStmt(SourceLocation Loc)
       : Stmt(CompoundStmtClass), RBraceLoc(Loc) {
     CompoundStmtBits.NumStmts = 0;
+    CompoundStmtBits.WasReplaced = 0;
     CompoundStmtBits.LBraceLoc = Loc;
   }
 
diff --git a/interpreter/llvm/src/tools/clang/lib/AST/Stmt.cpp b/interpreter/llvm/src/tools/clang/lib/AST/Stmt.cpp
index 0a4d403106..cc8c6888b8 100644
--- a/interpreter/llvm/src/tools/clang/lib/AST/Stmt.cpp
+++ b/interpreter/llvm/src/tools/clang/lib/AST/Stmt.cpp
@@ -293,6 +293,7 @@ CompoundStmt::CompoundStmt(ArrayRef<Stmt *> Stmts, SourceLocation LB,
                            SourceLocation RB)
     : Stmt(CompoundStmtClass), RBraceLoc(RB) {
   CompoundStmtBits.NumStmts = Stmts.size();
+  CompoundStmtBits.WasReplaced = 0;
   setStmts(Stmts);
   CompoundStmtBits.LBraceLoc = LB;
 }
@@ -316,6 +317,7 @@ CompoundStmt *CompoundStmt::CreateEmpty(const ASTContext &C,
   void *Mem =
       C.Allocate(totalSizeToAlloc<Stmt *>(NumStmts), alignof(CompoundStmt));
   CompoundStmt *New = new (Mem) CompoundStmt(EmptyShell());
+  New->CompoundStmtBits.WasReplaced = 0;
   New->CompoundStmtBits.NumStmts = NumStmts;
   return New;
 }

which smells fishy (but I don't want to go down that rabbit hole if the patch is not deemed suitable for other reasons)...

@phsft-bot
Copy link
Collaborator

Build failed on mac11.0/cxx17.
Running on macphsft20.dyndns.cern.ch:/Users/sftnight/build/workspace/root-pullrequests-build
See console output.

Errors:

  • [2021-03-12T11:08:06.354Z] FAILED: interpreter/llvm/src/lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/IfConversion.cpp.o
  • [2021-03-12T11:08:06.648Z] clang: error: unable to execute command: Segmentation fault: 11
  • [2021-03-12T11:08:06.648Z] clang: error: clang frontend command failed due to signal (use -v to see invocation)

hahnjo and others added 3 commits April 21, 2021 16:28
As Node is still a CompoundStmt, this will call the same function
recursively and again visit all previously visited and replaced
children, which doesn't seem necessary.
For the update of LLVM 9, Cling required another patch to Clang for
replacing the children of a CompoundStmt. Instead solve this by
creating a new CompoundStmt with the right Stmts attached.

Co-authored-by: Jonas Hahnfeld <[email protected]>
Co-authored-by: Jonas Hahnfeld <[email protected]>
@phsft-bot
Copy link
Collaborator

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

@hahnjo
Copy link
Member Author

hahnjo commented Apr 21, 2021

I think that we don't actually need the recursive visits in EvaluateTSynthesizer, which removes the need for the temporary AST node. However, I can still reproduce the failing clad tests... Edit: And applying the "fix" from #7488 (comment) still works - WHY?

@vgvassilev
Copy link
Member

vgvassilev commented Apr 22, 2021

@hahnjo, thanks for working on this! Could you share some more information about the test failures?

Just like you, I am quite puzzled why that fixes any tests... The only reason that I can think of is we did not properly reverted vgvassilev/clang@fce2607 which does not seem the case...

Can you remove the fixup patch to see current failures?

PS: Maybe somehow clad picks up old headers/libraries?

@hahnjo
Copy link
Member Author

hahnjo commented Apr 22, 2021

@vgvassilev sure: gtest-math-mathcore-test-CladDerivatorTests crashes with an illegal instruction somewhere in the JITed code, while gtest-hist-hist-test-TFormulaGradientTests shows:

105: Running main() from /home/jhahnfel/ROOT/build/googletest-prefix/src/googletest/googletest/src/gtest_main.cc
105: [==========] Running 7 tests from 1 test suite.
105: [----------] Global test environment set-up.
105: [----------] 7 tests from TFormulaGradientPar
105: [ RUN      ] TFormulaGradientPar.Sanity
105: /home/jhahnfel/ROOT/src/hist/hist/test/TFormulaGradientTests.cxx:34: Failure
105: Expected equality of these values:
105:   x[0] * std::cos(30)
105:     Which is: 0.15425146
105:   result[0]
105:     Which is: 0
105: [  FAILED  ] TFormulaGradientPar.Sanity (134 ms)
105: [ RUN      ] TFormulaGradientPar.ResultUpsize
105: /home/jhahnfel/ROOT/src/hist/hist/test/TFormulaGradientTests.cxx:52: Failure
105: Expected equality of these values:
105:   std::cos(30)
105:     Which is: 0.15425146
105:   result[1]
105:     Which is: 0
105: [  FAILED  ] TFormulaGradientPar.ResultUpsize (6 ms)
105: [ RUN      ] TFormulaGradientPar.ResultDownsize
105: /home/jhahnfel/ROOT/src/hist/hist/test/TFormulaGradientTests.cxx:69: Failure
105: Expected equality of these values:
105:   std::cos(60)
105:     Which is: -0.95241296
105:   result[0]
105:     Which is: 0
105: [  FAILED  ] TFormulaGradientPar.ResultDownsize (6 ms)
105: [ RUN      ] TFormulaGradientPar.GausCrossCheck
105: [       OK ] TFormulaGradientPar.GausCrossCheck (6 ms)
105: [ RUN      ] TFormulaGradientPar.BreitWignerCrossCheck
105: [       OK ] TFormulaGradientPar.BreitWignerCrossCheck (6 ms)
105: [ RUN      ] TFormulaGradientPar.BreitWignerCrossCheckAccuracyDemo
105: [       OK ] TFormulaGradientPar.BreitWignerCrossCheckAccuracyDemo (0 ms)
105: [ RUN      ] TFormulaGradientPar.GetGradFormula
105: [       OK ] TFormulaGradientPar.GetGradFormula (1 ms)
105: [----------] 7 tests from TFormulaGradientPar (159 ms total)
105:
105: [----------] Global test environment tear-down
105: [==========] 7 tests from 1 test suite ran. (159 ms total)
105: [  PASSED  ] 4 tests.
105: [  FAILED  ] 3 tests, listed below:
105: [  FAILED  ] TFormulaGradientPar.Sanity
105: [  FAILED  ] TFormulaGradientPar.ResultUpsize
105: [  FAILED  ] TFormulaGradientPar.ResultDownsize
105:
105:  3 FAILED TESTS

Can you remove the fixup patch to see current failures?

That's the funny thing: the PR doesn't contain the fixup, but Jenkins is happy anyway. It could be that it doesn't include clad (which would be bad) or that some of my changes either introduce or uncover bad loads of uninitialized memory or use-after-free issues. I'm currently building an instrumented version of ROOT with AddressSanitizer...

@vgvassilev
Copy link
Member

@hahnjo, is your build incremental? Like, you built the master, then rebuild the PR code? If so, clad won't be rebuilt as any of the other ExternalProject_Adds in cmake...

@hahnjo
Copy link
Member Author

hahnjo commented Apr 22, 2021

@hahnjo, is your build incremental? Like, you built the master, then rebuild the PR code? If so, clad won't be rebuilt as any of the other ExternalProject_Adds in cmake...

Yes, I'm building incrementally, but even removing interpreter/cling/tools/plugins/clad/clad-prefix/ and forcing a rebuild of Clad doesn't solve the issue. I think ASan found an issue in Clad, but now I need to rebuild with debug symbols in order to locate it...

@vgvassilev
Copy link
Member

Ok, good, progress! I'd propose to move forward with this change and address the clad issue separately.

@hahnjo
Copy link
Member Author

hahnjo commented Apr 22, 2021

Eventually yes (feel free to review the current changes), but I don't like having failing tests. Green Jenkins being one thing, but I also want all to pass locally 😃

Copy link
Member

@vgvassilev vgvassilev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@hahnjo
Copy link
Member Author

hahnjo commented Apr 22, 2021

Hm, the "finding" from ASan was #7968 aka Clad built without ASan and setting some memory that the sanitizer afterwards doesn't know about. Both tests, gtest-math-mathcore-test-CladDerivatorTests and gtest-hist-hist-test-TFormulaGradientTests, still fail in the same way as before, but ASan stays silent so it either can't detect the problem or there's none from the class of errors it checks. Back to the beginning...

@vgvassilev
Copy link
Member

Could you delete your build folder, reconfigure and rebuild?

@hahnjo
Copy link
Member Author

hahnjo commented Apr 22, 2021

Okay, I just found where I made the mistake: While deleting interpreter/cling/tools/plugins/clad/clad-prefix/ rebuilds Clad, it does NOT rebuild libCling.so which includes the statically built plugin (as far as I understand). It's now passing all tests with an incremental build when taking this into consideration. I'll do a clean build over the night and re-think tomorrow morning, but I think the failures were indeed caused by the build system.

@hahnjo hahnjo marked this pull request as ready for review April 23, 2021 06:32
@hahnjo
Copy link
Member Author

hahnjo commented Apr 23, 2021

Yep, a clean build passed all tests. And to prove the point, I then switched back to master (without additional measures) and now the tests are failing there (because Clad wasn't rebuilt for changed-back headers). So everything checks out.

@hahnjo hahnjo merged commit 2d304cf into root-project:master Apr 23, 2021
@hahnjo hahnjo deleted the cling-replaceStmts branch April 23, 2021 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants