Non reproducibility in wf 136.793

[perrotta]

It looks like that there is some non-reproducibility in (at least) workflow 136.793

It is witnessed by the following lines in the log, which appear quite often in different number and with different values in the bot tests when compared between baseline and baseline+PR

curv error not pos-def
[  1.78225e+18  1.2382e+19 -4.3207e+19-1.49326e+20 1.63409e+20
    1.2382e+19-7.05161e+30-3.66399e+30-1.25526e+31-9.31912e+31
   -4.3207e+19-3.66399e+30 1.10223e+32 3.81575e+32-4.78835e+31
  -1.49326e+20-1.25526e+31 3.81575e+32 1.32094e+33-1.64027e+32
   1.63409e+20-9.31912e+31-4.78835e+31-1.64027e+32-1.23157e+33 ]
pos/mom/mf  (-91.8426,13.6769,-182.754)   (-7.26726,11.4853,-50.0727)   (0.0342356,-0.00509825,3.75128)

A couple of examples in #43025 (comment) and #43283 (e.g.)

[perrotta]

assign reconstruction

[cmsbuild]

New categories assigned: reconstruction

@jfernan2,@mandrenguyen you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

A new Issue was created by @perrotta Andrea Perrotta.

@rappoccio, @antoniovilela, @sextonkennedy, @Dr15Jones, @smuzaffar, @makortel can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[perrotta]

@cms-sw/tracking-pog-l2

[makortel]

From the log of tests of #43025 (comment) it seems the printout mentioned in the issue description comes from ConversionTrackCandidateProducer:conversionTrackCandidates

%MSG-w BasicTrajectoryState:   ConversionTrackCandidateProducer:conversionTrackCandidates  15-Nov-2023 10:32:41 CET Run: 301998 Event: 9374081
curv error not pos-def
[  1.78225e+18  1.2382e+19 -4.3207e+19-1.49326e+20 1.63409e+20
    1.2382e+19-7.05161e+30-3.66399e+30-1.25526e+31-9.31912e+31
   -4.3207e+19-3.66399e+30 1.10223e+32 3.81575e+32-4.78835e+31
  -1.49326e+20-1.25526e+31 3.81575e+32 1.32094e+33-1.64027e+32
   1.63409e+20-9.31912e+31-4.78835e+31-1.64027e+32-1.23157e+33 ]
pos/mom/mf  (-91.8426,13.6769,-182.754)   (-7.26726,11.4853,-50.0727)   (0.0342356,-0.00509825,3.75128) 
%MSG

[perrotta]

With some little investigation made while in train, it seems to me that:

• this issue was not present in CMSSW_13_3_X_2023-11-02-1100 [1]
• this issue was present in CMSSW_13_3_X_2023-11-03-1100
• Observation: [mkFit] Tools supporting better layer-related computations -- technical changes to support further development #43145 was merged in CMSSW_13_3_X_2023-11-03-1100. Not necessarily the culprit, but perhaps worth investigating further...

[1] at least I'm not seeing a large number of lines added/removed from the log in a PR that was tested on that IB

[dan131riley]

I'm running valgrind memcheck on 136.793 and 11834.21 (I was already doing 11834.21 for #42700, I've updated to CMSSW_14_0_X_2023-11-16-1100 to cover this too)

[makortel]

Log differences in #43310 (comment) show similar non-reproducibility in workflow 136.874 as well. The full message in question (in PR's tests) is

%MSG-w BasicTrajectoryState:   ConversionTrackCandidateProducer:conversionTrackCandidates  17-Nov-2023 01:41:50 CET Run: 319450 Event: 105991987
BasicTrajectoryState: attempt to access errors when none available  accessing local error..
freestate pointer: parameters
x =       35.2409     -52.2478      81.7855
p =   1.74807e+08  1.21867e+08 -9.77032e+08
no error defined.

local error valid/values :0
[  -9.9999e+14 -2.3304e-05  0.00125994   0.02167531.15042e-310
   -2.3304e-05 7.03905e-06-0.0005213461.15036e-3104.94066e-324
    0.00125994-0.000521346  0.001443776.95253e-3101.15048e-310
     0.02167531.15036e-3106.95253e-3101.15042e-3101.15042e-310
  1.15042e-3104.94066e-3241.15048e-3101.15042e-3101.15042e-310 ]
%MSG

[makortel]

Is anyone investigating these non-reproducibilities?

[makortel]

Is anybody looking into these?

[cmsbuild]

cms-bot internal usage

[jfernan2]

type tracking