Improve various Patatrack Kernels #35598

VinInn · 2021-10-09T12:17:43Z

Simplify and improve the logic of various Kernels.
Some maths have been sync with CPU version.
Fixed and improved the statistics collection and printing (off by default).

Technical.
No regression observed: math has changed so some regression cannot be excluded (even in CPU wf).

VinInn · 2021-10-09T12:19:16Z

enable gpu

VinInn · 2021-10-09T12:19:35Z

@cmsbuild , please test

cmsbuild · 2021-10-09T12:25:26Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35598/25860

This PR adds an extra 48KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35598/25860/code-format.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35598/25860/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

VinInn · 2021-10-09T12:36:00Z

@cmsbuild , please test

cmsbuild · 2021-10-09T12:43:37Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35598/25861

This PR adds an extra 48KB to repository

cmsbuild · 2021-10-09T12:44:03Z

A new Pull Request was created by @VinInn (Vincenzo Innocente) for master.

It involves the following packages:

CalibTracker/SiPixelESProducers (alca)
CondFormats/SiPixelObjects (db, alca)
RecoLocalTracker/SiPixelClusterizer (reconstruction)
RecoLocalTracker/SiPixelRecHits (reconstruction)
RecoPixelVertexing/PixelTriplets (reconstruction)

@malbouis, @yuanchao, @ggovi, @slava77, @jpata, @tvami, @francescobrivio can you please review it and eventually sign? Thanks.
@mtosi, @felicepantaleo, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @OzAmram, @ferencek, @dkotlins, @tocheng, @gpetruc, @mmusich, @threus, @dgulhan, @seemasharmafnal, @tvami this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

tvami · 2021-10-09T14:04:52Z

CalibTracker/SiPixelESProducers/src/SiPixelGainCalibrationForHLTService.cc

@@ -38,7 +38,7 @@ void SiPixelGainCalibrationForHLTService::calibrate(
    if (isDeadColumn | isNoisyColumn)
      electron[i++] = 0;
    else {
-      float vcal = di->adc() * gain - pedestal * gain;
+      float vcal = float(di->adc()) * gain - pedestal * gain;
      //    float vcal = (di->adc()  - DBpedestal) * DBgain;


please remove the commented out code, or explain why it might be needed, thanks Vincenzo!

It was there before.
In principle maybe faster and more precise, BUT I assume the calibration is done with the above so better to keep that.
Anyhow trivial to remove.

tvami · 2021-10-09T14:05:07Z

CalibTracker/SiPixelESProducers/src/SiPixelGainCalibrationService.cc

@@ -28,7 +28,7 @@ void SiPixelGainCalibrationServiceBase::calibrate(
    else {
      float DBgain = getGain(detID, col, row);
      float DBpedestal = getPedestal(detID, col, row) * DBgain;
-      float vcal = di->adc() * DBgain - DBpedestal;
+      float vcal = float(di->adc()) * DBgain - DBpedestal;
      //    float vcal = (di->adc()  - DBpedestal) * DBgain;


Same comment as above regarding commented code

cmsbuild · 2021-10-09T16:04:45Z

-1

Failed Tests: RelVals-GPU
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-80c871/19518/summary.html
COMMIT: df4460f
CMSSW: CMSSW_12_1_X_2021-10-09-1100/slc7_amd64_gcc900
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/35598/19518/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-GPU

10824.50210824.502_TTbar_13+2018_Patatrack_PixelOnlyGPU+TTbar_13TeV_TuneCUETP8M1_GenSim+Digi+RecoFakeHLT+HARVESTFakeHLT/step3_TTbar_13+2018_Patatrack_PixelOnlyGPU+TTbar_13TeV_TuneCUETP8M1_GenSim+Digi+RecoFakeHLT+HARVESTFakeHLT.log

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 40
DQMHistoTests: Total histograms compared: 2798082
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2798060
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 39 files compared)
Checked 169 log files, 37 edm output root files, 40 DQM output files
TriggerResults: no differences found

VinInn · 2021-10-09T17:39:23Z

Tried to reproduce runTheMatrix crashes by itself though

[innocent@patatrack02 matrix]$ runTheMatrix.py -l 10824.502
processing relval_standard
processing relval_highstats
processing relval_pileup
processing relval_generator
processing relval_extendedgen
processing relval_production
processing relval_ged
ignoring relval_upgrade from default matrix
ignoring relval_cleanedupgrade from default matrix
ignoring relval_gpu from default matrix
processing relval_2017
processing relval_2026
ignoring relval_identity from default matrix
processing relval_machine
processing relval_premix
Traceback (most recent call last):
  File "/cvmfs/cms-ib.cern.ch/nweek-02701/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-10-04-2300/bin/slc7_amd64_gcc900/runTheMatrix.py", line 596, in <module>
    ret = runSelected(opt)
  File "/cvmfs/cms-ib.cern.ch/nweek-02701/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-10-04-2300/bin/slc7_amd64_gcc900/runTheMatrix.py", line 31, in runSelected
    if len(undefSet)>0: raise ValueError('Undefined workflows: '+', '.join(map(str,list(undefSet))))
ValueError: Undefined workflows: 10824.502
[[innocent@patatrack02 matrix]$ echo $CMSSW_BASE/
/home/innocent/FixClus/CMSSW_12_1_X_2021-10-05-1100/

VinInn · 2021-10-09T17:43:44Z

same in CMSSW_12_1_X_2021-10-09-1100

VinInn · 2021-10-09T17:51:05Z

ok. reproduced on MC using a hlt menu of my own...
Will investigate

VinInn · 2021-10-25T15:15:55Z

@cmsbuild , please test

VinInn · 2021-10-25T15:17:26Z

oops messed up the merge

VinInn · 2021-10-25T15:23:21Z

superseded by #35835

cmsbuild · 2021-10-25T15:23:40Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35598/26199

This PR adds an extra 24KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Alignment/MuonAlignment/src/MuonAlignment.cc modified in PR(s): Migrate to the new ref. based writeOneIOV() in Alignment package #35711
- File CalibTracker/SiPixelESProducers/src/SiPixelGainCalibrationForHLTService.cc modified in PR(s): Improve Various Patatrack kernels #35835
- File CalibTracker/SiPixelESProducers/src/SiPixelGainCalibrationService.cc modified in PR(s): Improve Various Patatrack kernels #35835
- File CondFormats/SiPixelObjects/interface/SiPixelGainCalibrationForHLT.h modified in PR(s): Improve Various Patatrack kernels #35835
- File CondFormats/SiPixelObjects/interface/SiPixelGainForHLTonGPU.h modified in PR(s): Improve Various Patatrack kernels #35835
- File Configuration/AlCa/python/autoCond.py modified in PR(s): Migration to DD4hep as default for all Run-3 workflows #35761
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): add basic data run3 reco and apply to splash events as a wf 138.3; fix tools for simple lumi list #35776, Use pdigi_hi_nogen for HI MC wfs #35028, Add DQM Matrix to RelVals #35678, Migration to DD4hep as default for all Run-3 workflows #35761
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Add run3 tracking low pu era #33532, Phase 2 HLT menu targetting 7.5×10³⁴ cm⁻²s⁻¹ #35342, Migration to DD4hep as default for all Run-3 workflows #35761
- File Configuration/PyReleaseValidation/scripts/runTheMatrix.py modified in PR(s): Migration to DD4hep as default for all Run-3 workflows #35761
- File DQM/BeamMonitor/plugins/BeamMonitor.cc modified in PR(s): Modernize plugins in DQM/BeamMonitor #35501
- File DQM/BeamMonitor/plugins/FakeBeamMonitor.cc modified in PR(s): Modernize plugins in DQM/BeamMonitor #35501
- File DQMOffline/Configuration/python/autoDQM.py modified in PR(s): Phase2 L1T DQM #35744
- File DataFormats/PatCandidates/src/classes_def_objects.xml modified in PR(s): Add track-based isolation, recHit energy sum to IsolatedTrack class #31399
- File EventFilter/Utilities/src/FedRawDataInputSource.cc modified in PR(s): DAQ input throttling (12_1_X) #35833
- File FWCore/ParameterSet/python/Config.py modified in PR(s): [RFC] Add a mechanism to set the chosen case for SwitchProducer #35510
- File Geometry/HGCalCommonData/interface/HGCalDDDConstants.h modified in PR(s): Phase2-hgx293 Try to resolve some issues for scintillator part of HGCal Geometry #35796
- File Geometry/HGCalCommonData/src/HGCalDDDConstants.cc modified in PR(s): Phase2-hgx292 Bug fix for real partial wafer simulations in versions V15 and V16 #35765
- File Geometry/HGCalGeometry/src/HGCalGeometry.cc modified in PR(s): Phase2-hgx293 Try to resolve some issues for scintillator part of HGCal Geometry #35796
- File Geometry/HGCalGeometry/src/HGCalGeometryLoader.cc modified in PR(s): Phase2-hgx293 Try to resolve some issues for scintillator part of HGCal Geometry #35796
- File Geometry/HGCalGeometry/test/HGCalGeometryRotTest.cc modified in PR(s): Phase2-hgx292 Bug fix for real partial wafer simulations in versions V15 and V16 #35765
- File HLTrigger/Configuration/python/customizeHLTforCMSSW.py modified in PR(s): [WIP] add fillPSetDescription to CkfTrackCandidateMakerBase and propagate it to CkfTrackCandidateMaker #35385
- File HeterogeneousCore/MPICore/test/BuildFile.xml modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File RecoLocalCalo/HGCalRecProducers/plugins/HGCalCLUEAlgo.h modified in PR(s): Updates to the SiPM-on-tile and Si sensor digitization (end-of-life/realistic startup) #35763
- File RecoLocalTracker/SiPixelClusterizer/plugins/SiPixelRawToClusterGPUKernel.cu modified in PR(s): Improve Various Patatrack kernels #35835
- File RecoLocalTracker/SiPixelClusterizer/plugins/gpuCalibPixel.h modified in PR(s): Improve Various Patatrack kernels #35835
- File RecoLocalTracker/SiPixelClusterizer/plugins/gpuClusterChargeCut.h modified in PR(s): Improve Various Patatrack kernels #35835, Fix for crash in gpuClusterChargeCut.h after "warning too many clusters ..." #35829
- File RecoLocalTracker/SiPixelClusterizer/plugins/gpuClustering.h modified in PR(s): Improve Various Patatrack kernels #35835, Use cooperative groups to populate Associations (Histograms) in Pixel Patatrack #35713
- File RecoLocalTracker/SiPixelRecHits/interface/pixelCPEforGPU.h modified in PR(s): Improve Various Patatrack kernels #35835
- File RecoPixelVertexing/PixelTriplets/plugins/CAHitNtupletGeneratorKernels.cc modified in PR(s): Improve Various Patatrack kernels #35835, Use cooperative groups to populate Associations (Histograms) in Pixel Patatrack #35713
- File RecoPixelVertexing/PixelTriplets/plugins/CAHitNtupletGeneratorKernels.cu modified in PR(s): Use cooperative groups to populate Associations (Histograms) in Pixel Patatrack #35713
- File RecoPixelVertexing/PixelTriplets/plugins/CAHitNtupletGeneratorKernelsImpl.h modified in PR(s): Improve Various Patatrack kernels #35835, Use cooperative groups to populate Associations (Histograms) in Pixel Patatrack #35713
- File RecoPixelVertexing/PixelTriplets/plugins/GPUCACell.h modified in PR(s): Improve Various Patatrack kernels #35835
- File RecoTracker/IterativeTracking/python/DetachedQuadStep_cff.py modified in PR(s): [WIP] add fillPSetDescription to CkfTrackCandidateMakerBase and propagate it to CkfTrackCandidateMaker #35385
- File RecoTracker/IterativeTracking/python/DetachedTripletStep_cff.py modified in PR(s): [WIP] add fillPSetDescription to CkfTrackCandidateMakerBase and propagate it to CkfTrackCandidateMaker #35385
- File RecoTracker/IterativeTracking/python/HighPtTripletStep_cff.py modified in PR(s): [WIP] add fillPSetDescription to CkfTrackCandidateMakerBase and propagate it to CkfTrackCandidateMaker #35385
- File RecoTracker/IterativeTracking/python/JetCoreRegionalStep_cff.py modified in PR(s): [WIP] add fillPSetDescription to CkfTrackCandidateMakerBase and propagate it to CkfTrackCandidateMaker #35385
- File RecoTracker/IterativeTracking/python/LowPtBarrelTripletStep_cff.py modified in PR(s): [WIP] add fillPSetDescription to CkfTrackCandidateMakerBase and propagate it to CkfTrackCandidateMaker #35385
- File RecoTracker/IterativeTracking/python/LowPtQuadStep_cff.py modified in PR(s): [WIP] add fillPSetDescription to CkfTrackCandidateMakerBase and propagate it to CkfTrackCandidateMaker #35385
- File RecoTracker/IterativeTracking/python/LowPtTripletStep_cff.py modified in PR(s): [WIP] add fillPSetDescription to CkfTrackCandidateMakerBase and propagate it to CkfTrackCandidateMaker #35385
- File RecoTracker/IterativeTracking/python/MixedTripletStep_cff.py modified in PR(s): [WIP] add fillPSetDescription to CkfTrackCandidateMakerBase and propagate it to CkfTrackCandidateMaker #35385
- File RecoTracker/IterativeTracking/python/PixelLessStep_cff.py modified in PR(s): [WIP] add fillPSetDescription to CkfTrackCandidateMakerBase and propagate it to CkfTrackCandidateMaker #35385
- File RecoTracker/IterativeTracking/python/PixelPairStep_cff.py modified in PR(s): [WIP] add fillPSetDescription to CkfTrackCandidateMakerBase and propagate it to CkfTrackCandidateMaker #35385
- File RecoTracker/IterativeTracking/python/TobTecStep_cff.py modified in PR(s): [WIP] add fillPSetDescription to CkfTrackCandidateMakerBase and propagate it to CkfTrackCandidateMaker #35385
- File RecoTracker/TrackProducer/plugins/TrackExtraRekeyer.cc modified in PR(s): Add producers/configs to refit muon tracks with standard TrackRefitter #34640
- File RecoTracker/TrackProducer/plugins/TrackProducerFromPatMuons.cc modified in PR(s): Add producers/configs to refit muon tracks with standard TrackRefitter #34640
- File RecoTracker/TrackProducer/test/refitFromAOD.py modified in PR(s): Add producers/configs to refit muon tracks with standard TrackRefitter #34640
- File RecoTracker/TrackProducer/test/refitFromMINIAOD.py modified in PR(s): Add producers/configs to refit muon tracks with standard TrackRefitter #34640
- File SimG4CMS/Calo/src/HGCalNumberingScheme.cc modified in PR(s): Phase2-hgx292 Bug fix for real partial wafer simulations in versions V15 and V16 #35765
- File SimG4Core/Application/src/RunManagerMT.cc modified in PR(s): Remaining migration of SIM to EventSetup #35732
- File SimG4Core/Application/src/RunManagerMTWorker.cc modified in PR(s): Remaining migration of SIM to EventSetup #35732
- File Utilities/XrdAdaptor/src/XrdRequestManager.cc modified in PR(s): For async calls always store XrdCl::FileSystem with the response-handler. #35481, Backport XrootD 5.3.1 PR to master from DEVEL #34991

cmsbuild · 2021-10-25T15:24:10Z

Pull request #35598 was updated. @SiewYan, @gouskos, @bbilin, @wajidalikhan, @pmandrik, @ianna, @Saptaparna, @Martin-Grunewald, @ggovi, @alberto-sanchez, @santocch, @cecilecaillol, @perrotta, @civanch, @yuanchao, @makortel, @ahmad3213, @missirol, @agrohsje, @fwyzard, @GurpreetSinghChahal, @smorovic, @davidlange6, @smuzaffar, @Dr15Jones, @cvuosalo, @emanueleusai, @mdhildreth, @AdrianoDee, @jfernan2, @kskovpen, @slava77, @jpata, @qliphy, @fabiocos, @pbo0, @francescobrivio, @malbouis, @mkirsano, @jordan-martins, @rekovic, @emeschi, @alja, @srimanob, @fgolf, @mariadalfonso, @tvami, @rvenditti can you please check and sign again.

tvami · 2021-10-25T15:25:40Z

Hi @VinInn I think you need to rebase now to the latest IBs

tvami · 2021-10-25T15:26:54Z

ok, closing is also an option :)

VinInn · 2021-10-25T15:28:06Z

In any case the thread here was long and confusing due the too many spurious errors

cmsbuild · 2021-10-25T19:02:15Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-80c871/19919/summary.html
COMMIT: 58fd079
CMSSW: CMSSW_12_1_X_2021-10-25-1100/slc7_amd64_gcc900
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/35598/19919/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-80c871/11634.506_TTbar_14TeV+2021_Patatrack_PixelOnlyTripletsGPU+TTbar_14TeV_TuneCP5_GenSim+Digi+Reco+HARVEST

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19782
DQMHistoTests: Total failures: 15
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 19767
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: no differences found

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 41
DQMHistoTests: Total histograms compared: 2797338
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2797316
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 40 files compared)
Checked 173 log files, 37 edm output root files, 41 DQM output files
TriggerResults: no differences found

cmsbuild added this to the CMSSW_12_1_X milestone Oct 9, 2021

cmsbuild added alca-pending code-checks-pending db-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Oct 9, 2021

cmsbuild added tests-started and removed tests-pending labels Oct 9, 2021

cmsbuild added code-checks-rejected and removed code-checks-pending labels Oct 9, 2021

cmsbuild added code-checks-pending and removed code-checks-rejected labels Oct 9, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 9, 2021

tvami reviewed Oct 9, 2021

View reviewed changes

cmsbuild added tests-rejected and removed tests-started labels Oct 9, 2021

cmsbuild added dqm-pending generators-pending geometry-pending hlt-pending l1-pending operations-pending pdmv-pending simulation-pending tests-pending upgrade-pending visualization-pending xpog-pending labels Oct 25, 2021

cmsbuild added tests-started and removed tests-pending labels Oct 25, 2021

cmsbuild mentioned this pull request Oct 25, 2021

DAQ input throttling (12_1_X) #35833

Merged

VinInn mentioned this pull request Oct 25, 2021

Improve Various Patatrack kernels #35835

Merged

VinInn closed this Oct 25, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 25, 2021

cmsbuild added tests-approved and removed tests-started labels Oct 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve various Patatrack Kernels #35598

Improve various Patatrack Kernels #35598

VinInn commented Oct 9, 2021

VinInn commented Oct 9, 2021

VinInn commented Oct 9, 2021

cmsbuild commented Oct 9, 2021

VinInn commented Oct 9, 2021

cmsbuild commented Oct 9, 2021

cmsbuild commented Oct 9, 2021

tvami Oct 9, 2021

VinInn Oct 9, 2021

tvami Oct 9, 2021

cmsbuild commented Oct 9, 2021

VinInn commented Oct 9, 2021 •

edited

Loading

VinInn commented Oct 9, 2021 •

edited

Loading

VinInn commented Oct 9, 2021

VinInn commented Oct 25, 2021

VinInn commented Oct 25, 2021

VinInn commented Oct 25, 2021

cmsbuild commented Oct 25, 2021

cmsbuild commented Oct 25, 2021

tvami commented Oct 25, 2021

tvami commented Oct 25, 2021

VinInn commented Oct 25, 2021

cmsbuild commented Oct 25, 2021

Improve various Patatrack Kernels #35598

Improve various Patatrack Kernels #35598

Conversation

VinInn commented Oct 9, 2021

VinInn commented Oct 9, 2021

VinInn commented Oct 9, 2021

cmsbuild commented Oct 9, 2021

VinInn commented Oct 9, 2021

cmsbuild commented Oct 9, 2021

cmsbuild commented Oct 9, 2021

tvami Oct 9, 2021

Choose a reason for hiding this comment

VinInn Oct 9, 2021

Choose a reason for hiding this comment

tvami Oct 9, 2021

Choose a reason for hiding this comment

cmsbuild commented Oct 9, 2021

RelVals-GPU

Comparison Summary

VinInn commented Oct 9, 2021 • edited Loading

VinInn commented Oct 9, 2021 • edited Loading

VinInn commented Oct 9, 2021

VinInn commented Oct 25, 2021

VinInn commented Oct 25, 2021

VinInn commented Oct 25, 2021

cmsbuild commented Oct 25, 2021

cmsbuild commented Oct 25, 2021

tvami commented Oct 25, 2021

tvami commented Oct 25, 2021

VinInn commented Oct 25, 2021

cmsbuild commented Oct 25, 2021

GPU Comparison Summary

Comparison Summary

VinInn commented Oct 9, 2021 •

edited

Loading

VinInn commented Oct 9, 2021 •

edited

Loading