EMTF Emulator Update to Add Displaced Muon pT & Dxy Assignment #32813

eyigitba · 2021-02-04T11:37:53Z

PR description:

This PR is a resubmission of #32641 which was reported to have thread safety issues in cms-sw/cmsdist/issues/6618. The thread safety issues are fixed by removing the 'static' calls in PtAssignmentEngineDxy.

This PR updates EMTF emulator by adding a new class (PtAssignmentEngineDxy) that handles vertex unconstrained pT and Dxy assignment. The changes should not affect anything for prompt muons. The main changes are:

Added PtAssignmentEngineDxy and necessary changes to the EMTF algorithm to support it.
Added setters and getters for vertex unconstrained pT and Dxy to EMTFTrack

The backport of this PR to 11_2_X (#32642 ) is already updated/

PR validation:

Validated with:

scram b runtests
runTheMatrix.py -l limited -i all --ibeos

Notifications: @jiafulow

…ting EMTFTrack object to have the relevant setters and getters.

…ines.

…than the constructor in PtAssignmentEngineDxy

…the new data path.

cmsbuild · 2021-02-04T11:43:34Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-32813/21000

This PR adds an extra 20KB to repository
Found files with invalid states:
- L1Trigger/L1TMuonEndCap/data/emtfpp_tf_graphs/model_graph.displ.5.pb:
  - Added: 24c6939
  - Deleted: 5b7189f
- L1Trigger/L1TMuonEndCap/data/emtfpp_tf_graphs/model_graph.displ.4.pb:
  - Added: 8ae8aaf
  - Deleted: 8ad2a1c

cmsbuild · 2021-02-04T11:43:55Z

A new Pull Request was created by @eyigitba for master.

It involves the following packages:

DataFormats/L1TMuon
L1Trigger/L1TMuonEndCap

@cmsbuild, @rekovic, @jmduarte can you please review it and eventually sign? Thanks.
@dinyar, @Martin-Grunewald, @rovere, @thomreis this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

silviodonato · 2021-02-09T13:15:54Z

please test

cmsbuild · 2021-02-09T14:36:09Z

-1

Failed Tests: RelVals RelVals-INPUT AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f83712/12788/summary.html
COMMIT: 75ca712
CMSSW: CMSSW_11_3_X_2021-02-08-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

136.731

----- Begin Fatal Exception 09-Feb-2021 14:46:05 CET-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Processing  Event run: 274199 lumi: 21 event: 39389360 stream: 0
   [1] Running path 'HLTAnalyzerEndpath'
   [2] Prefetching for module L1TRawToDigi/'hltGtStage2Digis'
   [3] Prefetching for module RawDataCollectorByLabel/'rawDataCollector'
   [4] Prefetching for module L1TDigiToRaw/'packGmtStage2'
   [5] Calling method for module L1TMuonEndCapTrackProducer/'simEmtfDigis'
Exception Message:
edm::FileInPath unable to find file L1Trigger/L1TMuon/data/emtf_luts/model_graph.displ.5.pb anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/CMSSW_11_3_X_2021-02-08-2300/poison:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/CMSSW_11_3_X_2021-02-08-2300/src:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/CMSSW_11_3_X_2021-02-08-2300/external/slc7_amd64_gcc900/data:/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_X_2021-02-08-2300/poison:/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_X_2021-02-08-2300/src:/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_X_2021-02-08-2300/external/slc7_amd64_gcc900/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/runTheMatrix-results/136.731_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2
----- End Fatal Exception -------------------------------------------------

136.793

----- Begin Fatal Exception 09-Feb-2021 14:46:06 CET-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Processing  Event run: 301998 lumi: 9 event: 8792128 stream: 0
   [1] Running path 'HLTAnalyzerEndpath'
   [2] Prefetching for module L1TRawToDigi/'hltGtStage2Digis'
   [3] Prefetching for module RawDataCollectorByLabel/'rawDataCollector'
   [4] Prefetching for module L1TDigiToRaw/'packGmtStage2'
   [5] Calling method for module L1TMuonEndCapTrackProducer/'simEmtfDigis'
Exception Message:
edm::FileInPath unable to find file L1Trigger/L1TMuon/data/emtf_luts/model_graph.displ.5.pb anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/CMSSW_11_3_X_2021-02-08-2300/poison:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/CMSSW_11_3_X_2021-02-08-2300/src:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/CMSSW_11_3_X_2021-02-08-2300/external/slc7_amd64_gcc900/data:/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_X_2021-02-08-2300/poison:/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_X_2021-02-08-2300/src:/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_X_2021-02-08-2300/external/slc7_amd64_gcc900/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/runTheMatrix-results/136.793_RunDoubleEG2017C+RunDoubleEG2017C+HLTDR2_2017+RECODR2_2017reHLT_skimDoubleEG_Prompt+HARVEST2017
----- End Fatal Exception -------------------------------------------------

136.874

----- Begin Fatal Exception 09-Feb-2021 14:46:05 CET-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Processing  Event run: 319450 lumi: 76 event: 105676750 stream: 0
   [1] Running path 'HLTAnalyzerEndpath'
   [2] Prefetching for module L1TRawToDigi/'hltGtStage2Digis'
   [3] Prefetching for module RawDataCollectorByLabel/'rawDataCollector'
   [4] Prefetching for module L1TDigiToRaw/'packGmtStage2'
   [5] Calling method for module L1TMuonEndCapTrackProducer/'simEmtfDigis'
Exception Message:
edm::FileInPath unable to find file L1Trigger/L1TMuon/data/emtf_luts/model_graph.displ.5.pb anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/CMSSW_11_3_X_2021-02-08-2300/poison:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/CMSSW_11_3_X_2021-02-08-2300/src:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/CMSSW_11_3_X_2021-02-08-2300/external/slc7_amd64_gcc900/data:/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_X_2021-02-08-2300/poison:/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_X_2021-02-08-2300/src:/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_X_2021-02-08-2300/external/slc7_amd64_gcc900/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/runTheMatrix-results/136.874_RunEGamma2018C+RunEGamma2018C+HLTDR2_2018+RECODR2_2018reHLT_skimEGamma_Offline_L1TEgDQM+HARVEST2018_L1TEgDQM
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

136.722136.722_RunDoubleEG2016B+RunDoubleEG2016B+HLTDR2_2016+RECODR2_2016reHLT_skimDoubleEG_HIPM+HARVESTDR2/step2_RunDoubleEG2016B+RunDoubleEG2016B+HLTDR2_2016+RECODR2_2016reHLT_skimDoubleEG_HIPM+HARVESTDR2.log
136.723136.723_RunDoubleMuon2016B+RunDoubleMuon2016B+HLTDR2_2016+RECODR2_2016reHLT_HIPM+HARVESTDR2/step2_RunDoubleMuon2016B+RunDoubleMuon2016B+HLTDR2_2016+RECODR2_2016reHLT_HIPM+HARVESTDR2.log
136.721136.721_RunHLTPhy2016B+RunHLTPhy2016B+HLTDR2_2016+RECODR2_2016reHLT_HIPM+HARVESTDR2/step2_RunHLTPhy2016B+RunHLTPhy2016B+HLTDR2_2016+RECODR2_2016reHLT_HIPM+HARVESTDR2.log

Expand to see more relval errors ...

AddOn Tests

hlt_data_Fake2

----- Begin Fatal Exception 09-Feb-2021 14:42:16 CET-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Processing  Event run: 272762 lumi: 51 event: 41468301 stream: 0
   [1] Running path 'L1TAnalyzerEndpath'
   [2] Prefetching for module L1TGlobalSummary/'L1TGlobalSummary'
   [3] Prefetching for module L1TGlobalProducer/'simGtStage2Digis'
   [4] Prefetching for module L1TMuonProducer/'simGmtStage2Digis'
   [5] Calling method for module L1TMuonEndCapTrackProducer/'simEmtfDigis'
Exception Message:
edm::FileInPath unable to find file L1Trigger/L1TMuon/data/emtf_luts/model_graph.displ.5.pb anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/CMSSW_11_3_X_2021-02-08-2300/poison:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/CMSSW_11_3_X_2021-02-08-2300/src:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/CMSSW_11_3_X_2021-02-08-2300/external/slc7_amd64_gcc900/data:/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_X_2021-02-08-2300/poison:/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_X_2021-02-08-2300/src:/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_X_2021-02-08-2300/external/slc7_amd64_gcc900/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-addon/addOnTests/hlt_data_Fake2
----- End Fatal Exception -------------------------------------------------

hlt_data_PRef

----- Begin Fatal Exception 09-Feb-2021 14:42:20 CET-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Processing  Event run: 323775 lumi: 99 event: 111190435 stream: 1
   [1] Running path 'L1TAnalyzerEndpath'
   [2] Prefetching for module L1TGlobalSummary/'L1TGlobalSummary'
   [3] Prefetching for module L1TGlobalProducer/'simGtStage2Digis'
   [4] Prefetching for module L1TMuonProducer/'simGmtStage2Digis'
   [5] Calling method for module L1TMuonEndCapTrackProducer/'simEmtfDigis'
Exception Message:
edm::FileInPath unable to find file L1Trigger/L1TMuon/data/emtf_luts/model_graph.displ.5.pb anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/CMSSW_11_3_X_2021-02-08-2300/poison:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/CMSSW_11_3_X_2021-02-08-2300/src:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/CMSSW_11_3_X_2021-02-08-2300/external/slc7_amd64_gcc900/data:/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_X_2021-02-08-2300/poison:/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_X_2021-02-08-2300/src:/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_X_2021-02-08-2300/external/slc7_amd64_gcc900/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-addon/addOnTests/hlt_data_PRef
----- End Fatal Exception -------------------------------------------------

hlt_data_HIon

----- Begin Fatal Exception 09-Feb-2021 14:42:21 CET-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Processing  Event run: 325112 lumi: 6 event: 29040 stream: 1
   [1] Running path 'L1TAnalyzerEndpath'
   [2] Prefetching for module L1TGlobalSummary/'L1TGlobalSummary'
   [3] Prefetching for module L1TGlobalProducer/'simGtStage2Digis'
   [4] Prefetching for module L1TMuonProducer/'simGmtStage2Digis'
   [5] Calling method for module L1TMuonEndCapTrackProducer/'simEmtfDigis'
Exception Message:
edm::FileInPath unable to find file L1Trigger/L1TMuon/data/emtf_luts/model_graph.displ.5.pb anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/CMSSW_11_3_X_2021-02-08-2300/poison:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/CMSSW_11_3_X_2021-02-08-2300/src:/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12788/CMSSW_11_3_X_2021-02-08-2300/external/slc7_amd64_gcc900/data:/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_X_2021-02-08-2300/poison:/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_X_2021-02-08-2300/src:/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_11_3_X_2021-02-08-2300/external/slc7_amd64_gcc900/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-addon/addOnTests/hlt_data_HIon
----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

eyigitba · 2021-02-09T14:38:27Z

This PR once again depends on the data file at: cms-data/L1Trigger-L1TMuon#19. Should I resubmit that PR as well?

silviodonato · 2021-02-09T15:20:21Z

test parameters:

pull_request = Data files for EMTF NN for displaced muons cms-data/L1Trigger-L1TMuon#19

silviodonato · 2021-02-09T15:20:28Z

please test

silviodonato · 2021-02-09T15:20:48Z

kind reminder @cms-sw/l1-l2 (@duartej @rekovic )

duartej · 2021-02-09T15:55:50Z

kind reminder @cms-sw/l1-l2 (@duartej @rekovic )

I think you mean @jmduarte :)

cmsbuild · 2021-02-09T19:45:57Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f83712/12798/summary.html
COMMIT: 75ca712
CMSSW: CMSSW_11_3_X_2021-02-09-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32813/12798/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2751765
DQMHistoTests: Total failures: 123
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2751620
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 36 files compared)
Checked 156 log files, 37 edm output root files, 37 DQM output files

rekovic · 2021-02-10T13:53:43Z

+1

cmsbuild · 2021-02-10T13:54:06Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

silviodonato · 2021-02-11T08:30:21Z

+1

makortel · 2021-02-12T14:02:17Z

Seems that this PR could be causing segfaults in ROOT master IBs, see #32894 for more details. (it is very much unclear to me why the crashes are specific to ROOT master)

eyigitba · 2021-02-12T14:08:17Z

Hi @makortel , thanks for letting me know. I have no idea why this might be. I'll investigate.

eyigitba added 9 commits February 4, 2021 11:15

Adding NNv5 for displaced pT and d0 assignment to EMTF emulator. Upda…

8ae8aaf

…ting EMTFTrack object to have the relevant setters and getters.

Added the NN data file

24c6939

Fixed styling. Moved pbFileNameDxy option to config. Removed unused l…

4517c1f

…ines.

Moved configuration of graphDxy and sessionDxy to configure() rather …

28a9987

…than the constructor in PtAssignmentEngineDxy

Removed the old/unused NN protobuf file.

8ad2a1c

Added code checks and code format changes.

a6fe759

Removed data directory as requested by Vladimir. Changed the path to …

5b7189f

…the new data path.

Changed data file path to match the PR to external data repository

a90e14d

Fixing thread safety issues caused by static calls.

75ca712

cmsbuild added this to the CMSSW_11_3_X milestone Feb 4, 2021

cmsbuild added code-checks-pending l1-pending orp-pending pending-signatures tests-pending labels Feb 4, 2021

eyigitba mentioned this pull request Feb 4, 2021

EMTF Emulator Update to Add Displaced Muon pT & Dxy Assignment #32641

Merged

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 4, 2021

silviodonato mentioned this pull request Feb 9, 2021

[11_2_X] EMTF Emulator Update to Add Displaced Muon pT & Dxy Assignment #32642

Merged

cmsbuild added tests-started and removed tests-pending labels Feb 9, 2021

cmsbuild added tests-rejected and removed tests-started labels Feb 9, 2021

cmsbuild added the requires-external label Feb 9, 2021

cmsbuild added tests-started and removed tests-rejected labels Feb 9, 2021

cmsbuild added tests-approved and removed tests-started labels Feb 9, 2021

cmsbuild added fully-signed l1-approved and removed l1-pending pending-signatures labels Feb 10, 2021

silviodonato mentioned this pull request Feb 11, 2021

Update tag for L1Trigger-L1TMuon to V01-03-00 cms-sw/cmsdist#6643

Merged

cmsbuild added orp-approved and removed orp-pending labels Feb 11, 2021

cmsbuild merged commit f08df55 into cms-sw:master Feb 11, 2021

eyigitba mentioned this pull request Feb 11, 2021

EMTF Emulator Update to Add Displaced Muon pT & Dxy Assignment cms-l1t-offline/cmssw#885

Merged

makortel mentioned this pull request Feb 12, 2021

Crashes in TensorFlow via L1TMuonEndCapTrackProducer #32894

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EMTF Emulator Update to Add Displaced Muon pT & Dxy Assignment #32813

EMTF Emulator Update to Add Displaced Muon pT & Dxy Assignment #32813

eyigitba commented Feb 4, 2021 •

edited

Loading

cmsbuild commented Feb 4, 2021

cmsbuild commented Feb 4, 2021

silviodonato commented Feb 9, 2021

cmsbuild commented Feb 9, 2021

eyigitba commented Feb 9, 2021

silviodonato commented Feb 9, 2021

silviodonato commented Feb 9, 2021

silviodonato commented Feb 9, 2021

duartej commented Feb 9, 2021

cmsbuild commented Feb 9, 2021

rekovic commented Feb 10, 2021

cmsbuild commented Feb 10, 2021

silviodonato commented Feb 11, 2021

makortel commented Feb 12, 2021

eyigitba commented Feb 12, 2021

EMTF Emulator Update to Add Displaced Muon pT & Dxy Assignment #32813

EMTF Emulator Update to Add Displaced Muon pT & Dxy Assignment #32813

Conversation

eyigitba commented Feb 4, 2021 • edited Loading

PR description:

PR validation:

cmsbuild commented Feb 4, 2021

cmsbuild commented Feb 4, 2021

silviodonato commented Feb 9, 2021

cmsbuild commented Feb 9, 2021

RelVals

RelVals-INPUT

AddOn Tests

eyigitba commented Feb 9, 2021

silviodonato commented Feb 9, 2021

silviodonato commented Feb 9, 2021

silviodonato commented Feb 9, 2021

duartej commented Feb 9, 2021

cmsbuild commented Feb 9, 2021

Comparison Summary

rekovic commented Feb 10, 2021

cmsbuild commented Feb 10, 2021

silviodonato commented Feb 11, 2021

makortel commented Feb 12, 2021

eyigitba commented Feb 12, 2021

eyigitba commented Feb 4, 2021 •

edited

Loading