Fake differences in wf 11642.911

[silviodonato]

Many PR tests are reporting fake differences in the new DD4HEP wf 11642.911.

See
#32278 (comment)
#32263 (comment)
#32279 (comment)
#32276 (comment)
#32093 (comment)
#32272 (comment)
#32273 (comment)
#32270 (comment)

For instance in #32258
we ran two times the same PR (commit 71e9ea9) in the same release (CMSSW_11_2_X_2020-11-24-1100). The result was different

Reco comparison results: 2325 differences found in the comparisons

and then

Reco comparison results: 7 differences found in the comparisons

[cmsbuild]

A new Issue was created by @silviodonato Silvio Donato.

@Dr15Jones, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[silviodonato]

assign geometry

[cmsbuild]

New categories assigned: geometry

@Dr15Jones,@cvuosalo,@mdhildreth,@makortel,@ianna,@civanch you have been requested to review this Pull request/Issue and eventually sign? Thanks

[silviodonato]

The PR #32096 added both 11642.911 and 11624.911 in the short matrix. If we continue to see these fake difference, I would remove 11642.911 from the short matrix (ie. DD4HEP remains in the short matrix with 11624.911; 11642.911 will be only in the extended matrix)

[bsunanda]

@silviodonato Any difference in dd4hep workflow should be ignored as of now. Once dd4hep workflows are studied in detail differences in those workflows should be taken seriously.

[civanch]

@silviodonato , @bsunanda , we can expect, that after some time differences in these WFs will be meaningful but it is not the case now. Even more differences indicate, that PR is working.
Because the difference is should happens only in PRs, which relevant to geometry, I would think we may work in these circumstances for some time.

[mmusich]

same thing observed in #32280 (comment) and that has nothing to do with Geometry.

[cvuosalo]

Why is workflow 11642.911 creating differences but 11624.911 does not? They seem to be basically the same, 11624 is ttbar and 11642 is zmm, but otherwise they both perform gen-sim. They should always use the same random seed to give the same events, unless the geometry is changed in the PR. I don't understand why 11642 is generating new events every time it is run in a PR test even when the geometry does not change.

[makortel]

I suspect we're witnessing a memory problem of some kind (in addition to the threading problem, or perhaps the problem is the same and threading just makes it worse).

[cvuosalo]

@makortel I thought the PR tests run single-threaded.

[makortel]

They are, but memory problems can nevertheless show up as random failures or differences.

[silviodonato]

Workflow removed (#32288). We will try again once #32249 is solved