Update ROOT and Cling to LLVM18 #15696

devajithvs · 2024-05-31T14:37:27Z

This Pull request:

Rebase

A few relevant LLVM commits that were referenced for the rebase:

Notes

Cling and Root commits for the rebase are kept separate for now (can squash similar commits together later).
Also added a temporary commit root-project/llvm-project@4066693 in LLVM to prevent building of mlgo-utils, which will not compile when all the test folders are removed. We can either keep it or add an exception to this file in the Root GitHub workflow to not remove the test folders under mlgo-utils.

TO DISCUSS/PROPERLY FIX

The commit 64f7cf6 is memleaking; it is a temporary fix for now to prevent some failing tests, like gtest-roofit-histfactory-test-testHistFactory.
- Fixed with: [cling] Fix lifetime of ClingMMapper #16314
This commit: de5d141 is a temporarily fix reintroducing a memory leak.
- Fixed with: 2c2d4dc
This commit: d79963e fixes an issue with failing tests like tutorial-hist-cumulative when JITLink is turned on. This works for now, but will need to investigate further on why resource trackers were not destructed in the right order.
- Not a critical issue: Can also look into this later after the merge.
Discussion on a proper solution for the macOS modulemap issue
- Fixed with: [rootcling] Add option to allow system byproducts #16315

Changes or fixes:

Checklist:

Fix mac13 build
tested changes locally
updated the docs (if necessary)

This PR fixes #

github-actions · 2024-05-31T20:47:55Z

Test Results

13 files 13 suites 3d 2h 28m 40s ⏱️
3 029 tests 3 029 ✅ 0 💤 0 ❌
33 856 runs 33 856 ✅ 0 💤 0 ❌

Results for commit a52d34d.

♻️ This comment has been updated with latest results.

stephanlachnit · 2024-07-18T21:09:50Z

I know this is totally not relevant for you, but I am still wondering: does this allow me to use system clang or are there still downstream patches required?

hahnjo · 2024-07-19T06:23:26Z

I know this is totally not relevant for you, but I am still wondering: does this allow me to use system clang or are there still downstream patches required?

Hi Stephan, no we still require downstream patches. Their number gets reduced on every upgrade, you can track the status for LLVM 18 here: https://github.com/root-project/llvm-project/commits/ROOT-llvm18. More importantly, and not yet reflected in ROOT-llvm18, we will now also require a patch to the core LLVM JIT infrastructure (until upgrading past LLVM 19): llvm/llvm-project#95532

stephanlachnit · 2024-07-19T06:45:55Z

Thanks for the update!

bellenot · 2024-08-09T06:13:03Z

Here is the stack trace on Windows:

  Generating etc/allDict.cxx.pch

  Generating PCH for bindings\tpython core core\clingutils core\imt core\rint core\thread graf2d\asimage graf2d\fitsio graf2d\gpad graf2d\gpadv7 graf2d\graf graf2d\postscript graf2d\primitivesv7 graf2d\win32gdk gr
  af3d\g3d graf3d\gl gui\fitpanel gui\fitpanelv7 gui\gui hist\hist hist\histdrawv7 hist\histpainter hist\histv7 hist\spectrum hist\spectrumpainter hist\unfold io\io math\genetic math\genvector math\mathcore math\m
  athmore math\matrix math\minuit math\minuit2 math\physics math\smatrix math\vecops net\net roofit\RDataFrameHelpers roofit\histfactory roofit\hs3 roofit\jsoninterface roofit\roofit roofit\roofitcore roofit\roofi
  tmore roofit\roostats tmva\pymva tmva\sofie tmva\tmva tmva\tmvagui tree\dataframe tree\ntuple tree\ntupleutil tree\tree tree\treeplayer tree\treeviewer tree\webviewer

  missing serialization code for annotation token
  UNREACHABLE executed at C:\root-dev\git\dev.llvm18\interpreter\llvm-project\clang\lib\Serialization\ASTWriter.cpp:4514!
  PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
  Stack dump:
  0.    Program arguments: .\\bin\\rootcling -rootbuild -generate-pch -f allDict.cxx -noDictSelection -D__CLING__ -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -DROOT_PCH -I.\\include -I.\\etc -I.\\etc\\dictpch -
  I.\\etc\\cling -IC:\\Python311\\Lib\\site-packages\\numpy\\core\\include -IC:\\Python311\\include -IC:\\libs\\x64\\GSL\\2.7\\include -IC:\\libs\\x64\\libxml2\\2.7.8\\include -IC:\\libs\\x64\\sqlite3\\3.45.2 -IC:
  /root-dev/git/dev.llvm18/graf2d/win32gdk/gdk/src etc\\dictpch\\allHeaders.h -IC:/root-dev/git/dev.llvm18/graf2d/win32gdk/gdk/src -IC:/root-dev/git/dev.llvm18/graf2d/win32gdk/gdk/src/gdk -IC:/root-dev/git/dev.llv
  m18/graf2d/win32gdk/gdk/src/glib -IC:/root-dev/build/x64/dev.llvm18/include -IC:/root-dev/git/dev.llvm18/core etc\\dictpch\\allLinkDefs.h
  Exception Code: 0x80000003
   #0 0x00007ffc2c2306eb HandleAbort C:\root-dev\git\dev.llvm18\interpreter\llvm-project\llvm\lib\Support\Windows\Signals.inc:424:0
   #1 0x00007ffcabe91881 (C:\windows\System32\ucrtbase.dll+0x71881)
   #2 0x00007ffcabe92851 (C:\windows\System32\ucrtbase.dll+0x72851)
   #3 0x00007ffc2c1dad4c llvm::llvm_unreachable_internal(char const *, char const *, unsigned int) C:\root-dev\git\dev.llvm18\interpreter\llvm-project\llvm\lib\Support\ErrorHandling.cpp:212:0
   #4 0x00007ffc25258963 clang::ASTWriter::AddToken(class clang::Token const &, class llvm::SmallVectorImpl<unsigned __int64> &) C:\root-dev\git\dev.llvm18\interpreter\llvm-project\clang\lib\Serialization\ASTWrite
  r.cpp:4514:0
   #5 0x00007ffc2525399f clang::ASTWriter::WriteLateParsedTemplates(class clang::Sema &) C:\root-dev\git\dev.llvm18\interpreter\llvm-project\clang\lib\Serialization\ASTWriter.cpp:4340:0
   #6 0x00007ffc252568d1 clang::ASTWriter::WriteASTCore(class clang::Sema &, class llvm::StringRef, class clang::Module *) C:\root-dev\git\dev.llvm18\interpreter\llvm-project\clang\lib\Serialization\ASTWriter.cpp:
  5117:0
   #7 0x00007ffc2525852a clang::ASTWriter::WriteAST(class clang::Sema &, class llvm::StringRef, class clang::Module *, class llvm::StringRef, bool) C:\root-dev\git\dev.llvm18\interpreter\llvm-project\clang\lib\Ser
  ialization\ASTWriter.cpp:4649:0
   #8 0x00007ffc237996f8 WriteAST C:\root-dev\git\dev.llvm18\core\dictgen\src\rootcling_impl.cxx:2021:0
   #9 0x00007ffc23799959 GenerateAllDict C:\root-dev\git\dev.llvm18\core\dictgen\src\rootcling_impl.cxx:2042:0
  #10 0x00007ffc237ac399 RootClingMain(int, char **, bool) C:\root-dev\git\dev.llvm18\core\dictgen\src\rootcling_impl.cxx:5010:0
  #11 0x00007ffc2378c455 ROOT_rootcling_Driver C:\root-dev\git\dev.llvm18\core\dictgen\src\rootcling_impl.cxx:6238:0
  #12 0x00007ff68e2f2605 main C:\root-dev\git\dev.llvm18\main\src\rootcling.cxx:44:0
  #13 0x00007ff68e2f2924 invoke_main D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78:0
  #14 0x00007ff68e2f2924 __scrt_common_main_seh D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288:0
  #15 0x00007ffcacad7374 (C:\windows\System32\KERNEL32.DLL+0x17374)
  #16 0x00007ffcae65cc91 (C:\windows\SYSTEM32\ntdll.dll+0x4cc91)
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(254,5): error MSB8066: Custom build for 'C:\root-dev\build\x64\dev.llvm18\CMakeFiles\6485b15e5a52a9a93f
294ac9c6ea6f8f\allDict.cxx.pch.rule;C:\root-dev\build\x64\dev.llvm18\CMakeFiles\c2921ee0b70b9f5a32ad0eb8246e0cd2\onepcm.rule;C:\root-dev\git\dev.llvm18\CMakeLists.txt' exited with code -1. [C:\root-dev\build\x64\d
ev.llvm18\onepcm.vcxproj]

hahnjo · 2024-08-20T17:29:33Z

FWIW the current state compiles fine on RISC-V and is at least able to run df103_NanoAODHiggsAnalysis.py 😉

vgvassilev · 2024-08-26T14:22:40Z

@smuzaffar, @devajithvs and @hahnjo have everything working reasonably with the llvm18 upgrade. Can you check this PR against cmssw?

vgvassilev · 2024-08-27T10:15:59Z

@smuzaffar, @devajithvs and @hahnjo have everything working reasonably with the llvm18 upgrade. Can you check this PR against cmssw?

@iarspider, could you give us a hand?

dpiparo · 2024-08-27T13:18:00Z

@smuzaffar, @devajithvs and @hahnjo have everything working reasonably with the llvm18 upgrade. Can you check this PR against cmssw?

@iarspider, could you give us a hand?

I allow myself to add @aandvalenzuela to perhaps get some help.

Changes similar to: llvm/llvm-project@8b1771b

Matching the change in: - llvm/llvm-project#78463

- Include `StringExtras.h` This is now needed due to: llvm/llvm-project@3ff3af3

In LLVM 18 with incremental extensions enabled, the Preprocessor will never report tok::eof, but tok::annot_repl_input_end instead.

Prevents failures such as: ``` Processing roottest/root/meta/runnamespace.C... Error in cling::MetaProcessor: cannot read from binary input: 'runnamespace.C' ``` This is needed for macros that start with `{` after the rebase

C++20 builds were failing due to the commit: llvm-project/commit/574ee1c02ef73b66c5957cf93888234b0471695f with an error `imported non C++20 importable modules`

This makes them run after the <Platform> library: It provides the __cxa_atexit function, which must not be found in the process.

LLVM now defines multiple Dylibs, and __cxa_atexit is provided in the <Platform> library.

We started seeing an assertion failure in ~FinalizedAlloc when we moved the generators to ProcessSymbols JITDylib: [cling] Move generators to ProcessSymbols JITDylib This ikely changed the destruction order that caused the assertion to trigger.

This fixes the Cling test Driver/E.C.

This applies the upstream changes in commit llvm/llvm-project@9992b38 to a copy of the modulemap from MacOSX14.2.sdk.

This fixes the failing python enum tests like: - `roottest-python-cpp-cpp` - `roottest-python-cmdLineUtils-ROOT_8197` Also remove `R__BYTESWAP` which should not be needed anymore. The leak was fixed with: llvm/llvm-project#78311 This caused our tests to fail as they relied on the previous behavior.

Upstream LLVM now unconditionally uses indirect / PC relative personality encodings since commit llvm/llvm-project@ca65969 This leads to references of DW.ref.__gxx_personality_v0 that cannot be resolved with RuntimeDyld, that we are still using for the moment. As a fix, auto-claim symbols as we are doing for PPC, see upstream commit: llvm/llvm-project@72ea1a7

There are still failures with RuntimeDyld on AArch64 which seem to be solved with JITLink, so make the switch now.

devajithvs self-assigned this May 31, 2024

hahnjo added in:Cling clean build Ask CI to do non-incremental build on PR labels May 31, 2024

devajithvs force-pushed the dev.llvm18 branch from 9a38f1e to e7383af Compare July 5, 2024 15:55

greenc-FNAL mentioned this pull request Jul 29, 2024

Request _C++20_ module compatibility with ROOT 6.30/00 #13536

Closed

devajithvs force-pushed the dev.llvm18 branch from e7383af to d50e594 Compare August 7, 2024 11:07

hahnjo force-pushed the dev.llvm18 branch from d50e594 to 3347f10 Compare August 8, 2024 08:43

devajithvs force-pushed the dev.llvm18 branch 4 times, most recently from 81d5c1b to d79963e Compare August 21, 2024 12:10

hahnjo force-pushed the dev.llvm18 branch from ee859c6 to 031202d Compare August 23, 2024 12:17

devajithvs force-pushed the dev.llvm18 branch from 031202d to b9b6290 Compare August 27, 2024 11:13

aandvalenzuela mentioned this pull request Aug 27, 2024

[Testing] Update ROOT and Cling to LLVM18 cms-sw/root#209

Closed

devajithvs force-pushed the dev.llvm18 branch from b9b6290 to 6456e70 Compare August 28, 2024 17:02

devajithvs marked this pull request as ready for review August 29, 2024 05:11

devajithvs requested review from bellenot, vgvassilev, dpiparo and pcanal as code owners August 29, 2024 05:11

devajithvs requested a review from hahnjo August 29, 2024 08:50

devajithvs and others added 28 commits September 10, 2024 16:07

[cling] Update OPTION() to match rebased OptTable

d4c16b6

[cling] JITTargetAddress --> ExecutorAddr

7cf2f6c

Changes similar to: llvm/llvm-project@8b1771b

[core] JITTargetAddress --> ExecutorAddr

c0551b7

Changes similar to: llvm/llvm-project@8b1771b

[core] Use FileEntryRef and DirectoryEntryRef

b83614c

[core] Use isPureVirtual

f656008

Matching the change in: - llvm/llvm-project#78463

[core] Additional parameters in AnnotateAttr::CreateImplicit

6ff579f

[core] Include missing headers

97244c0

- Include `StringExtras.h` This is now needed due to: llvm/llvm-project@3ff3af3

[core] Rename enums to match upstream LLVM 18

57468c8

[interpreter] Bump LLVM version

bcac194

[TCling] Handle enumeration value ‘StructuralValue’

6dbb74f

[cling] Adapt tok::eof -> tok::annot_repl_input_end

a3c84ee

In LLVM 18 with incremental extensions enabled, the Preprocessor will never report tok::eof, but tok::annot_repl_input_end instead.

[cling] Disable line numbers in error messages and warnings

fb5505c

[metcling] Povide a transaction before using GetDecl()

37cb102

[cling] Do not report an error if the fileType is a tapi_file

fc2002f

Prevents failures such as: ``` Processing roottest/root/meta/runnamespace.C... Error in cling::MetaProcessor: cannot read from binary input: 'runnamespace.C' ``` This is needed for macros that start with `{` after the rebase

[cling] Set BuiltinHeadersInSystemModules = 1

c28b205

[cling] Fix C++20 builds

8962e22

C++20 builds were failing due to the commit: llvm-project/commit/574ee1c02ef73b66c5957cf93888234b0471695f with an error `imported non C++20 importable modules`

[cling] Move generators to ProcessSymbols JITDylib

3eb76a4

This makes them run after the <Platform> library: It provides the __cxa_atexit function, which must not be found in the process.

[cling] Replace symbols in platform Dylib

6ab0c92

LLVM now defines multiple Dylibs, and __cxa_atexit is provided in the <Platform> library.

Make parallel jobs 4 for Windows

9d2327b

[cling] Enable conforming preprocessor with MSVC

70a1cdb

[cling] Fix option parsing, again

4950e9f

This fixes the Cling test Driver/E.C.

[cling] Update expected output of cling/test/Lookup/data.C

bdd8080

[runtime_cxxmodules] Fix macOS 13 modulemap for LLVM 18

6055bab

This applies the upstream changes in commit llvm/llvm-project@9992b38 to a copy of the modulemap from MacOSX14.2.sdk.

[core] Fix ASan build with LLVM 18

bc96e54

[cling] Enable JITLink by default for AArch64 on Linux

a52d34d

There are still failures with RuntimeDyld on AArch64 which seem to be solved with JITLink, so make the switch now.

devajithvs force-pushed the dev.llvm18 branch from 3873f4d to a52d34d Compare September 10, 2024 14:09

dpiparo merged commit 7f00674 into root-project:master Sep 11, 2024
18 of 19 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update ROOT and Cling to LLVM18 #15696

Update ROOT and Cling to LLVM18 #15696

devajithvs commented May 31, 2024 •

edited

Loading

github-actions bot commented May 31, 2024 •

edited

Loading

stephanlachnit commented Jul 18, 2024

hahnjo commented Jul 19, 2024

stephanlachnit commented Jul 19, 2024

bellenot commented Aug 9, 2024

hahnjo commented Aug 20, 2024

vgvassilev commented Aug 26, 2024

vgvassilev commented Aug 27, 2024

dpiparo commented Aug 27, 2024

Update ROOT and Cling to LLVM18 #15696

Update ROOT and Cling to LLVM18 #15696

Conversation

devajithvs commented May 31, 2024 • edited Loading

This Pull request:

Rebase

Notes

TO DISCUSS/PROPERLY FIX

Changes or fixes:

Checklist:

github-actions bot commented May 31, 2024 • edited Loading

Test Results

stephanlachnit commented Jul 18, 2024

hahnjo commented Jul 19, 2024

stephanlachnit commented Jul 19, 2024

bellenot commented Aug 9, 2024

hahnjo commented Aug 20, 2024

vgvassilev commented Aug 26, 2024

vgvassilev commented Aug 27, 2024

dpiparo commented Aug 27, 2024

devajithvs commented May 31, 2024 •

edited

Loading

github-actions bot commented May 31, 2024 •

edited

Loading