Phase-2 Global Trigger Emulator #41808

jheikkil · 2023-05-30T12:21:31Z

PR description:

This PR includes the first version of the Phase-2 Global Trigger emulator that implements a bit-wise compatible emulation of the GT firmware. The PR includes the HLT-TDR simplified L1 menu (16 seeds), documented in this twiki: https://twiki.cern.ch/twiki/bin/viewauth/CMS/PhaseIIL1TriggerMenuTools#How_to_use_the_GT_Emulator

As the Phase-2 Global Trigger emulator relies on various upstream emulators, this PR can be merged once:

another PR is completed (Retrained L1 Tau Trigger Integration #41492), and
minor changes on the input collections has been validated before/during the review process (related to taus and muons).

PR validation:

The expected rates have been presented at the Annual Review (https://indico.cern.ch/event/1257943/#7-algorithms-and-physics-perfo).

-Jaana on behalf of the Phase-2 Global Trigger team

cmsbuild · 2023-05-30T12:28:01Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41808/35693

ERROR: Build errors found during clang-tidy run.

L1Trigger/Phase2L1GT/plugins/L1GTProducer.cc:301:17: error: no viable conversion from 'const PackedJet' (aka 'const array<unsigned long, 2>') to 'l1gt::Jet' [clang-diagnostic-error]
      l1gt::Jet gtJet = collection[i].getHWJetGT();
                ^       ~~~~~~~~~~~~~~~~~~~~~~~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02787/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-29-2300/src/DataFormats/L1TParticleFlow/interface/gt_datatypes.h:80:10: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'const PackedJet' (aka 'const array<unsigned long, 2>') to 'const Jet &' for 1st argument
--
L1Trigger/Phase2L1GT/plugins/L1GTProducer.cc:370:39: error: no member named 'getHWTauGT' in 'l1t::PFTau' [clang-diagnostic-error]
      l1gt::Tau gtTau = collection[i].getHWTauGT();
                        ~~~~~~~~~~~~~ ^
L1Trigger/Phase2L1GT/plugins/L1GTProducer.cc:386:35: warning: std::move of the variable 'gtObj' of the trivially-copyable type 'P2GTCandidate' has no effect [performance-move-const-arg]
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:129: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

cmsbuild · 2023-05-30T15:23:15Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41808/35695

ERROR: Build errors found during clang-tidy run.

L1Trigger/Phase2L1GT/plugins/L1GTBoardWriter.cc:60:31: note: 'channel' declared here
--
L1Trigger/Phase2L1GT/plugins/L1GTBoardWriter.cc:61:106: error: variable 'channel' cannot be implicitly captured in a lambda with no capture-default specified [clang-diagnostic-error]
              channelMap.insert({l1t::demo::LinkId{"Algos", channel}, {l1t::demo::ChannelSpec{1, 0, 0}, {channel}}});
                                                                                                         ^
L1Trigger/Phase2L1GT/plugins/L1GTBoardWriter.cc:60:31: note: 'channel' declared here
--
L1Trigger/Phase2L1GT/plugins/L1GTBoardWriter.cc:63:20: error: variable 'channelMap' cannot be implicitly captured in a lambda with no capture-default specified [clang-diagnostic-error]
            return channelMap;
                   ^
L1Trigger/Phase2L1GT/plugins/L1GTBoardWriter.cc:59:54: note: 'channelMap' declared here
--
L1Trigger/Phase2L1GT/plugins/L1GTEvaluationProducer.cc:111:53: error: no member named 'EMP' in 'l1t::demo::FileFormat' [clang-diagnostic-error]
                           : l1t::demo::FileFormat::EMP,
                             ~~~~~~~~~~~~~~~~~~~~~~~^
Suppressed 1235 warnings (1235 in non-user code).
--
L1Trigger/Phase2L1GT/plugins/L1GTProducer.cc:370:39: error: no member named 'getHWTauGT' in 'l1t::PFTau' [clang-diagnostic-error]
      l1gt::Tau gtTau = collection[i].getHWTauGT();
                        ~~~~~~~~~~~~~ ^
Suppressed 1141 warnings (1141 in non-user code).
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:129: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

cmsbuild · 2023-05-30T16:01:26Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41808/35696

ERROR: Build errors found during clang-tidy run.

L1Trigger/Phase2L1GT/plugins/L1GTProducer.cc:370:39: error: no member named 'getHWTauGT' in 'l1t::PFTau' [clang-diagnostic-error]
      l1gt::Tau gtTau = collection[i].getHWTauGT();
                        ~~~~~~~~~~~~~ ^
Suppressed 1141 warnings (1141 in non-user code).
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:129: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

cmsbuild · 2023-06-02T10:23:53Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41808/35753

ERROR: Build errors found during clang-tidy run.

L1Trigger/Phase2L1GT/plugins/L1GTProducer.cc:370:39: error: no member named 'getHWTauGT' in 'l1t::PFTau' [clang-diagnostic-error]
      l1gt::Tau gtTau = collection[i].getHWTauGT();
                        ~~~~~~~~~~~~~ ^
Suppressed 1141 warnings (1141 in non-user code).
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:129: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

aloeliger · 2023-06-07T14:46:42Z

This requires #41492

cmsbuild · 2023-07-21T21:27:01Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-59f559/33830/summary.html
COMMIT: 2c267b2
CMSSW: CMSSW_13_3_X_2023-07-21-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41808/33830/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 15 lines from the logs
Reco comparison results: 52 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3149863
DQMHistoTests: Total failures: 225
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3149616
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 207 log files, 159 edm output root files, 48 DQM output files
TriggerResults: no differences found

aloeliger · 2023-07-22T13:29:29Z

+l1

@cms-sw/upgrade-l2 At this stage I think HLT and core software are okay with the idiosyncrasies of this emulator and we can plan to upgrade it as we go along to patch it into phase 2. Is there anything else you would liked addressed from your side?

srimanob · 2023-07-24T09:42:11Z

+Upgrade

cmsbuild · 2023-07-24T09:42:44Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

rappoccio · 2023-07-25T13:31:28Z

+1

aloeliger · 2023-07-26T15:07:33Z

@makortel @fwyzard At this point I want to ask both of you if you have any advice on the question of a menu parser. I talked about this a bit here: #41808 (comment) and here: #41808 (comment).

The complication is that for phase-2, whenever one wants to run the GT now, one has to load a series of modules corresponding to different paths or conditions, create (or load) paths for each of them, and have them attached to the process alongside two other GT components. I.e. to run the GT in a configuration, one must simultaneously specify what the menu is, there is no separation anymore on the questions of "Is the GT emulator running?" and "What is the GT emulator menu?". There is no XML file it reads from. It's all python modules now.

So there are two questions I want to try and sort out:

This is onerous to write by hand every time for someone writing a config by hand (i.e. you don't want to write out 512 paths and modules attached to a process every time). Presumably the L1 menu will pull from a lot of pre-defined places, so is there a good or best tool to use for defining a file to load from, loading/creating the modules and paths and attaching them to process, and then making sure they are in the schedule? The GT team has provided a tool that functions in this capacity, collectAlgorithmPaths, here: https://github.com/cms-sw/cmssw/pull/41808/files#diff-e31eafd3c6618dda2a3823af5957890f581204bf99538a0ef3dc42597b0a9b01R11-R23, where each file that defines a menu element, also adds it to a variable algorithms, and then by loading each file you fill this variable with desired modules, and it gives you a series of paths to be run. Is this an acceptable solution?
A primary use case is of course going to involve the cmsDriver. The same question exists as to how we want to specify a menu to be loaded (i.e. getting and loading the modules, attaching to the process, etc.) in a default way, and how one would do it for a non-default case (i.e. testing out a new menu or running a different L1 menu) when using the cmsDriver. I had to tackle this for some of our prototype use, and I came up with this: https://github.com/cms-l1t-offline/cmssw/blob/315e0b6cbcc55bec5b5eecbe955f25f68db9daef/Configuration/Applications/python/ConfigBuilder.py#L1523-L1579, which I didn't think looked like a long term solution (not the least of which because it just hard-codes the menu). Is there a better idea for using the new menu functionality at the cmsDriver? Could or should this sort of solution be adapted for cmsDriver?

(@qvyz Just tagging you on this, because we had discussed it a little)

srimanob · 2023-07-28T17:33:54Z

Hi @aloeliger
Do you think it is better to collect points and open the git issue to discuss? So we can keep issue as opening until solved.
Thx.

aloeliger · 2023-07-31T08:46:32Z

Hi @aloeliger Do you think it is better to collect points and open the git issue to discuss? So we can keep issue as opening until solved. Thx.

Probably. I'll open an issue.

fwyzard · 2024-10-07T13:17:45Z

L1Trigger/Phase2L1GT/python/l1tGTAlgoBlockProducer_cff.py

+def collectAlgorithmPaths(process) -> "tuple[cms.Path]":
+    str_paths = set()
+    for algorithm in algorithms:
+        algo_paths = re.sub(r'[()]'," " , algorithm.expression.value()).split()
+        for algo in algo_paths:
+            if algo in process.pathNames() :
+                str_paths.add(algo)
+    paths = set()
+
+    for str_path in str_paths:
+        paths.add(getattr(process, str_path))
+
+    return tuple(paths)


Unfortunately this approach can potentially return the paths in a different order on each invocation.
This does happen in production, and results in jobs with a different process hash.
The framework then considers them different configurations and does not fully merge the resulting files; for example, they appear to belong to different runs (even though the run number is the same).

@fwyzard I see, so in principle if the order is deterministic there shouldn't be a problem, right? So wouldn't a simple sorted(str_paths) then do the trick?

Yes, that should fix the reproducibility problem.

@fwyzard Would you mind either checking whether this indeed solves the issue or pointing me to how I can reproduce it myself before submitting a PR?

I'm not sure.

You could try running the cmsDriver command used in production a few times, potentially on different machines (like lxplus8 and lxplus9) and check

that sometimes you get different ordering with the current code

that you always get the same order with the fix

Would it be feasible to insert them in the process using the same order ?
Or it's not meaningful anyway ?

I suppose it could be ordered by seed number in the menu, but I don't think that is meaningful enough in this context that we truly need to order them that way as paths as well.

Would it be feasible to insert them in the process using the same order ? Or it's not meaningful anyway ?

Given that an algorithm can be a combination of multiple paths and in principle a path can be used in multiple algorithms (although it's unclear whether we should deem the latter case bad practice), I'm not entirely sure how meaningful any path order actually is.

I have opened #46303. @fwyzard or @rovere would it be possible to request that one of you retests the issues with this PR pulled in? I'm not quite sure what you have done to recreate this issue.

@rovere simply used the samples from the official Spring24 production; a test would be to re-produce the datasets.

cmsbuild added this to the CMSSW_13_2_X milestone May 30, 2023

cmsbuild added l1-pending pending-signatures tests-pending orp-pending new-package-pending code-checks-pending labels May 30, 2023

jheikkil changed the title ~~Implement Phase 2 GT~~ Phase-2 Global Trigger Emulator May 30, 2023

cmsbuild added code-checks-rejected and removed code-checks-pending labels May 30, 2023

HaarigerHarald force-pushed the GT_emulator_PR2 branch from f572c36 to 10a7b95 Compare May 30, 2023 15:16

cmsbuild added code-checks-pending and removed code-checks-rejected labels May 30, 2023

cmsbuild added code-checks-rejected and removed code-checks-pending labels May 30, 2023

HaarigerHarald force-pushed the GT_emulator_PR2 branch from 10a7b95 to 00ac291 Compare May 30, 2023 15:53

cmsbuild added code-checks-pending and removed code-checks-rejected labels May 30, 2023

cmsbuild added code-checks-rejected and removed code-checks-pending labels May 30, 2023

HaarigerHarald force-pushed the GT_emulator_PR2 branch from 00ac291 to d2ea389 Compare June 2, 2023 10:18

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jun 2, 2023

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jun 2, 2023

cmsbuild added the tests-started label Jul 21, 2023

cmsbuild added tests-approved and removed tests-started labels Jul 21, 2023

cmsbuild added l1-approved and removed l1-pending labels Jul 22, 2023

cmsbuild added fully-signed upgrade-approved and removed pending-signatures upgrade-pending labels Jul 24, 2023

cmsbuild added orp-approved and removed orp-pending labels Jul 25, 2023

cmsbuild merged commit e6660e3 into cms-sw:master Jul 25, 2023

This was referenced Jul 25, 2023

EMTF NNv16 Implementation #42177

Merged

Update TF to 2.12 (backport from IB/CMSSW_13_2_X/g13 branch) cms-sw/cmsdist#8565

Merged

HCALDQM - Add occupancy-per-LS based ML monitoring plots #42212

Merged

This was referenced Jul 27, 2023

Phase-2 Global Trigger Emulator (backport of PR41808) cms-l1t-offline/cmssw#1155

Closed

Phase-2 Global Trigger Emulator (backport of PR41808) cms-l1t-offline/cmssw#1156

Closed

aloeliger mentioned this pull request Jul 31, 2023

Configuration of the phase 2 L1T global trigger menu. #42421

Open

This was referenced Jan 31, 2024

[ASAN] Relval failing with asan libaray load issue #43823

Closed

Declare InvariantMassError as transient data product #43825

Merged

fwyzard reviewed Oct 7, 2024

View reviewed changes

aloeliger mentioned this pull request Oct 8, 2024

Sort Phase 2 L1T GT trigger paths #46303

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Phase-2 Global Trigger Emulator #41808

Phase-2 Global Trigger Emulator #41808

jheikkil commented May 30, 2023 •

edited

Loading

cmsbuild commented May 30, 2023

cmsbuild commented May 30, 2023

cmsbuild commented May 30, 2023

cmsbuild commented Jun 2, 2023

aloeliger commented Jun 7, 2023

cmsbuild commented Jul 21, 2023

aloeliger commented Jul 22, 2023

srimanob commented Jul 24, 2023

cmsbuild commented Jul 24, 2023

rappoccio commented Jul 25, 2023

aloeliger commented Jul 26, 2023

srimanob commented Jul 28, 2023

aloeliger commented Jul 31, 2023

fwyzard Oct 7, 2024

HaarigerHarald Oct 7, 2024

fwyzard Oct 7, 2024

HaarigerHarald Oct 7, 2024

fwyzard Oct 7, 2024

fwyzard Oct 8, 2024

aloeliger Oct 8, 2024 •

edited

Loading

HaarigerHarald Oct 8, 2024

aloeliger Oct 8, 2024

fwyzard Oct 8, 2024

Phase-2 Global Trigger Emulator #41808

Phase-2 Global Trigger Emulator #41808

Conversation

jheikkil commented May 30, 2023 • edited Loading

cmsbuild commented May 30, 2023

cmsbuild commented May 30, 2023

cmsbuild commented May 30, 2023

cmsbuild commented Jun 2, 2023

aloeliger commented Jun 7, 2023

cmsbuild commented Jul 21, 2023

Comparison Summary

aloeliger commented Jul 22, 2023

srimanob commented Jul 24, 2023

cmsbuild commented Jul 24, 2023

rappoccio commented Jul 25, 2023

aloeliger commented Jul 26, 2023

srimanob commented Jul 28, 2023

aloeliger commented Jul 31, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aloeliger Oct 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jheikkil commented May 30, 2023 •

edited

Loading

aloeliger Oct 8, 2024 •

edited

Loading