Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix MTV validation of initialStepPreSplitting tracks and add B-hadron MTV variation to pixel track validation sequence #199

Conversation

makortel
Copy link

This PR is a combination of #198 and #193 (to address the merge conflics).

@VinInn
Copy link

VinInn commented Nov 27, 2018

got

An exception of category 'InvalidReference' occurred while
   [0] Processing  Event run: 1 lumi: 150 event: 7453 stream: 2
   [1] Running path 'prevalidation_step'
   [2] Prefetching for module MultiTrackValidator/'trackValidatorTrackingOnly'
   [3] Prefetching for module VertexAssociatorByPositionAndTracksProducer/'VertexAssociatorByPositionAndTracks'
   [4] Calling method for module TrackAssociatorEDProducer/'trackingParticleRecoTrackAsssociation'
Exception Message:
ClusterTPAssociation has TrackingParticles with ProductID 32577:0, but got TrackingParticleRef/Handle/ProductID with ID 3:823. This is typically caused by a configuration error.
-

I remind that I only added

from RecoPixelVertexing.PixelTriplets.caHitQuadrupletHeterogeneousEDProducer_cfi import caHitQuadrupletHeterogeneousEDProducer as _caHitQuadrupletHeterogeneousEDProducer
process.initialStepHitQuadrupletsPreSplitting = _caHitQuadrupletHeterogeneousEDProducer.clone()
process.initialStepHitQuadrupletsPreSplitting.trackingRegions = "initialStepTrackingRegions"

to a standard reco dqm config (with Run2_2018,gpu)

@VinInn
Copy link

VinInn commented Nov 27, 2018

the standard tkreco dqm gives

----- Begin Fatal Exception 27-Nov-2018 09:30:29 CET-----------------------
An exception of category 'NotFound' occurred while
   [0] Calling beginStream for module ClusterTPAssociationHeterogeneous/'tpClusterProducerHeterogeneousPreSplitting'
Exception Message:
Service Request unable to find requested service with compiler type name ' 11CUDAService'.
----- End Fatal Exception -------------------------------------------------

something wrong, maybe in the merge
I will try with a virgin area w/o my PR...

I confirm that starting from CMSSW_10_4_0_pre2_Patatrack and merging this PR I get the error above
when running a cpu workflow (reco+dqm)

@VinInn
Copy link

VinInn commented Nov 27, 2018

ok,
adding

process.load('HeterogeneousCore.CUDAServices.CUDAService_cfi')

fix the cpu wf. maybe it has always been the case, not sure

@VinInn
Copy link

VinInn commented Nov 27, 2018

Ok, after merging and remerging #197 and this PR it seems running...
will post the result when ready....

@VinInn
Copy link

VinInn commented Nov 27, 2018

it makes much more sense now!
http://innocent.home.cern.ch/innocent/RelVal/ttbarPU50_gpuSeeds/plots_seeding_seeds_initialStepPreSplitting/effandfakePtEtaPhi.pdf

ps original is the old code...
have to redo it...

@makortel
Copy link
Author

@VinInn Great!

@VinInn
Copy link

VinInn commented Nov 27, 2018

Fixed the reference as well... (same link)
the one with the buggy reference is moved to
http://innocent.home.cern.ch/innocent/RelVal/ttbarPU50_gpuSeedsBuggy/plots_seeding_seeds_initialStepPreSplitting/effandfakePtEtaPhi.pdf

@makortel
Copy link
Author

Interesting that GPU workflow gives so much higher efficiency.

@VinInn
Copy link

VinInn commented Nov 27, 2018

and zero fakes/duplicates...
I know exactly why (cuts too tight in FPIX)

@fwyzard fwyzard added this to the CMSSW_10_4_0_pre3_Patatrack milestone Nov 28, 2018
@fwyzard
Copy link

fwyzard commented Nov 28, 2018

Validation summary

Reference release CMSSW_10_4_0_pre2 at c6061f4
Development branch CMSSW_10_4_X_Patatrack at fb73c7a
Testing PRs:

makeTrackValidationPlots.py plots

/RelValTTbar_13/CMSSW_10_4_0_pre2-PU25ns_103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_4_0_pre2-103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

DQM GUI plots

/RelValTTbar_13/CMSSW_10_4_0_pre2-PU25ns_103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_4_0_pre2-103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

logs and nvprof/nvvp profiles

/RelValTTbar_13/CMSSW_10_4_0_pre2-PU25ns_103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_4_0_pre2-103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

Logs

The full log is available at https://fwyzard.web.cern.ch/fwyzard/patatrack/pulls/df54e84387e5e3ee75936e5b88fa354e72ed6e4e/log .

@fwyzard
Copy link

fwyzard commented Nov 28, 2018

@makortel with #193 I get a summary that looks like this:

image

while with this PR I get something like this:

image

Did we loose the Pixel tracks (B-hadron TPs) part ?

@makortel
Copy link
Author

Did we loose the Pixel tracks (B-hadron TPs) part ?

A copy-paste mistake when resolving the conflict between #198 and #193, hopefully fixed with the last commit. Thanks for spotting!

@fwyzard
Copy link

fwyzard commented Nov 28, 2018

Validation summary

Reference release CMSSW_10_4_0_pre2 at c6061f4
Development branch CMSSW_10_4_X_Patatrack at fb73c7a
Testing PRs:

makeTrackValidationPlots.py plots

/RelValTTbar_13/CMSSW_10_4_0_pre2-PU25ns_103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_4_0_pre2-103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

DQM GUI plots

/RelValTTbar_13/CMSSW_10_4_0_pre2-PU25ns_103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_4_0_pre2-103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

logs and nvprof/nvvp profiles

/RelValTTbar_13/CMSSW_10_4_0_pre2-PU25ns_103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_4_0_pre2-103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

Logs

The full log is available at https://fwyzard.web.cern.ch/fwyzard/patatrack/pulls/27ca17c9e39f6fb387c81308756173c23ebcd265/log .

@fwyzard
Copy link

fwyzard commented Nov 28, 2018

In the meantime, I'm looking at the 2423 initcheck errors...

Looking at the first one

========= CUDA-MEMCHECK
========= Host API memory access error at host access to 0x761720000 of size 112660 bytes
=========     Uninitialized access at 0x761731720 on access by cudaMemcopy source.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/lib64/libcuda.so.1 (cuMemcpyAsync + 0x1b9) [0x24b4e9]
=========     Host Frame:/data/cmssw/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_0_pre2_Patatrack/external/slc7_amd64_gcc700/lib/libcudart.so.10.0 [0x3738d]
=========     Host Frame:/data/cmssw/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_0_pre2_Patatrack/external/slc7_amd64_gcc700/lib/libcudart.so.10.0 [0x14643]
=========     Host Frame:/data/cmssw/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_4_0_pre2_Patatrack/external/slc7_amd64_gcc700/lib/libcudart.so.10.0 (cudaMemcpyAsync + 0x176) [0x52a66]
=========     Host Frame:/data/user/fwyzard/patatrack/validation/run.1Fx6fmCJys/testing/lib/slc7_amd64_gcc700/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so (_ZN15pixelgpudetails28SiPixelRawToClusterGPUKernel17makeClustersAsyncEPK23SiPixelFedCablingMapGPUPKhPK22SiPixelGainForHLTonGPURKNS0_15WordFedAppenderEjjbbbbbRN4cuda8stream_tILb0EEE + 0x1d87) [0x6e827]
...

with objdump gives

$ objdump -d -S -l .../lib/slc7_amd64_gcc700/pluginRecoLocalTrackerSiPixelClusterizerPlugins.so | grep -B15 -A5 '6e827:'
   6e7ef:       48 8b bd b8 fb ff ff    mov    -0x448(%rbp),%rdi
   6e7f6:       ba 03 00 00 00          mov    $0x3,%edx
   6e7fb:       48 89 fe                mov    %rdi,%rsi
   6e7fe:       ff d0                   callq  *%rax
.../src/RecoLocalTracker/SiPixelClusterizer/plugins/SiPixelRawToClusterGPUKernel.cu:625
        cudaCheck(cudaMemcpyAsync(digis_clusters_h.adc.get(), digis_d.adc(), wordCounter*sizeof(uint16_t), cudaMemcpyDefault, stream.id()));
   6e800:       48 8b 95 a0 fb ff ff    mov    -0x460(%rbp),%rdx
   6e807:       48 8b b3 90 01 00 00    mov    0x190(%rbx),%rsi
   6e80e:       b9 04 00 00 00          mov    $0x4,%ecx
   6e813:       48 8b bb f0 00 00 00    mov    0xf0(%rbx),%rdi
   6e81a:       4d 8b 44 24 08          mov    0x8(%r12),%r8
   6e81f:       48 01 d2                add    %rdx,%rdx
   6e822:       e8 29 79 fc ff          callq  36150 <cudaMemcpyAsync@plt>
_Z10cudaCheck_PKciS0_9cudaError():
.../src/HeterogeneousCore/CUDAUtilities/interface/cudaCheck.h:28
   6e827:       85 c0                   test   %eax,%eax
   6e829:       0f 85 fd 14 00 00       jne    6fd2c <_ZN15pixelgpudetails28SiPixelRawToClusterGPUKernel17makeClustersAsyncEPK23SiPixelFedCablingMapGPUPKhPK22SiPixelGainForHLTonGPURKNS0_15WordFedAppenderEjjbbbbbRN4cuda8stream_tILb0EEE+0x328c>
_ZN15pixelgpudetails28SiPixelRawToClusterGPUKernel17makeClustersAsyncEPK23SiPixelFedCablingMapGPUPKhPK22SiPixelGainForHLTonGPURKNS0_15WordFedAppenderEjjbbbbbRN4cuda8stream_tILb0EEE():
...src/RecoLocalTracker/SiPixelClusterizer/plugins/SiPixelRawToClusterGPUKernel.cu:634
       std::cout
         << "CUDA countModules kernel launch with " << blocks

which points to

/RecoLocalTracker/SiPixelClusterizer/plugins/SiPixelRawToClusterGPUKernel.cu:625
        cudaCheck(cudaMemcpyAsync(digis_clusters_h.adc.get(), digis_d.adc(), wordCounter*sizeof(uint16_t), cudaMemcpyDefault, stream.id()));

@VinInn VinInn mentioned this pull request Nov 29, 2018
@fwyzard
Copy link

fwyzard commented Nov 29, 2018

The Pixel tracks (B-hadron TPs) part is back :-)

@fwyzard fwyzard merged commit 6110cf4 into cms-patatrack:CMSSW_10_4_X_Patatrack Nov 29, 2018
fwyzard pushed a commit that referenced this pull request Nov 29, 2018
… MTV variation to pixel track validation sequence (#199)

  - add B-hadron MTV variation to pixel track validation sequence
  - fix MTV validation of initialStepPreSplitting tracks
fwyzard pushed a commit that referenced this pull request Oct 8, 2020
… MTV variation to pixel track validation sequence (#199)

  - add B-hadron MTV variation to pixel track validation sequence
  - fix MTV validation of initialStepPreSplitting tracks
fwyzard pushed a commit that referenced this pull request Oct 20, 2020
… MTV variation to pixel track validation sequence (#199)

  - add B-hadron MTV variation to pixel track validation sequence
  - fix MTV validation of initialStepPreSplitting tracks
fwyzard pushed a commit that referenced this pull request Oct 20, 2020
… MTV variation to pixel track validation sequence (#199)

  - add B-hadron MTV variation to pixel track validation sequence
  - fix MTV validation of initialStepPreSplitting tracks
fwyzard pushed a commit that referenced this pull request Oct 23, 2020
… MTV variation to pixel track validation sequence (#199)

  - add B-hadron MTV variation to pixel track validation sequence
  - fix MTV validation of initialStepPreSplitting tracks
fwyzard pushed a commit that referenced this pull request Nov 6, 2020
… MTV variation to pixel track validation sequence (#199)

  - add B-hadron MTV variation to pixel track validation sequence
  - fix MTV validation of initialStepPreSplitting tracks
fwyzard pushed a commit that referenced this pull request Nov 6, 2020
… MTV variation to pixel track validation sequence (#199)

  - add B-hadron MTV variation to pixel track validation sequence
  - fix MTV validation of initialStepPreSplitting tracks
fwyzard pushed a commit that referenced this pull request Nov 16, 2020
… MTV variation to pixel track validation sequence (#199)

  - add B-hadron MTV variation to pixel track validation sequence
  - fix MTV validation of initialStepPreSplitting tracks
fwyzard added a commit that referenced this pull request Nov 27, 2020
… MTV variation to pixel track validation sequence (#199)

  - add B-hadron MTV variation to pixel track validation sequence
  - fix MTV validation of initialStepPreSplitting tracks
fwyzard added a commit that referenced this pull request Nov 28, 2020
… MTV variation to pixel track validation sequence (#199)

  - add B-hadron MTV variation to pixel track validation sequence
  - fix MTV validation of initialStepPreSplitting tracks
fwyzard pushed a commit that referenced this pull request Dec 26, 2020
… MTV variation to pixel track validation sequence (#199)

  - add B-hadron MTV variation to pixel track validation sequence
  - fix MTV validation of initialStepPreSplitting tracks
fwyzard added a commit that referenced this pull request Dec 26, 2020
… MTV variation to pixel track validation sequence (#199)

  - add B-hadron MTV variation to pixel track validation sequence
  - fix MTV validation of initialStepPreSplitting tracks
fwyzard pushed a commit that referenced this pull request Dec 29, 2020
… MTV variation to pixel track validation sequence (#199)

  - add B-hadron MTV variation to pixel track validation sequence
  - fix MTV validation of initialStepPreSplitting tracks
fwyzard pushed a commit that referenced this pull request Jan 15, 2021
… MTV variation to pixel track validation sequence (#199)

  - add B-hadron MTV variation to pixel track validation sequence
  - fix MTV validation of initialStepPreSplitting tracks
fwyzard pushed a commit that referenced this pull request Mar 23, 2021
… MTV variation to pixel track validation sequence (#199)

  - add B-hadron MTV variation to pixel track validation sequence
  - fix MTV validation of initialStepPreSplitting tracks
fwyzard pushed a commit that referenced this pull request Apr 1, 2021
… MTV variation to pixel track validation sequence (#199)

  - add B-hadron MTV variation to pixel track validation sequence
  - fix MTV validation of initialStepPreSplitting tracks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants