Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DQM bin-by-bin comparisons broken in last PRs? #42050

Closed
mmusich opened this issue Jun 22, 2023 · 12 comments
Closed

DQM bin-by-bin comparisons broken in last PRs? #42050

mmusich opened this issue Jun 22, 2023 · 12 comments

Comments

@mmusich
Copy link
Contributor

mmusich commented Jun 22, 2023

Title says it all, I wanted to check the outcome of the comparisons in a few PR-s but despite the counts of differences being non-zero they look empty.
Examples:

In other PRs it seems to work:

@cmsbuild
Copy link
Contributor

A new Issue was created by @mmusich Marco Musich.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@makortel
Copy link
Contributor

assign dqm,core

@cmsbuild
Copy link
Contributor

New categories assigned: dqm,core

@tjavaid,@micsucmed,@nothingface0,@rvenditti,@Dr15Jones,@smuzaffar,@emanueleusai,@syuvivida,@makortel,@pmandrik you have been requested to review this Pull request/Issue and eventually sign? Thanks

@mmusich
Copy link
Contributor Author

mmusich commented Jun 26, 2023

@cms-sw/dqm-l2 any update (or acknowledgement on this)?

@emanueleusai
Copy link
Member

Hello,

We’re looking into it. In the meantime the link to “comparisons failed” contains essentially the same plots and they do not appear empty, see e.g.:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_13_2_X_2023-06-22-2300+e01bcc/57647/140.58_RunHI2018/Tracking_TrackParameters_generalTracks_SeedMon_pixelLessStep_TrackBuilding.html

@perrotta
Copy link
Contributor

So, in CMSSW_13_2_X_2023-06-18-2300 the bin by bin comparison worked correctly, while in CMSSW_13_2_X_2023-06-20-1100 it did not (according to the issue description).
The only PR merged in the meanwhile that smells as a possible culprit is #41986: could it be the origin of it?

@emanueleusai
Copy link
Member

Hmm that is a very good lead. On the other hand, last year we had a similar problem and it was caused by a corrupted database. We had to delete it and recreate it.

@mmusich
Copy link
Contributor Author

mmusich commented Jun 26, 2023

In the meantime the link to “comparisons failed” contains essentially the same plots and they do not appear empty, see e.g.:

This doesn't seem to be completely correct.
At least for example in the "comparisons failed for the last round of tests of PR #41815 I see

Screenshot from 2023-06-26 10-36-22

while in the DQM bin-by-bin comparisons of the same PR I see more workflows with changes:
Screenshot from 2023-06-26 10-36-38

perhaps something to be investigated in another issue?

@emanueleusai
Copy link
Member

Yes the two comparison methods have always shown differences. It might have to do with the different thresholds the two methods use to define what’s different/failed, but is was never really investigated (not even by previous DQM people afaik).
I support opening a separate issue for this.

I usually check both and I found out that actual differences in dqm plots show up in both. I believe “Bin-to-bin” tends to be more noisy (or sensitive) and pick up stuff that is not a real discrepancy.

@emanueleusai
Copy link
Member

Btw it looks like the indexing db is broken. Last time it was fixed by deleting and recreating the index file but we were strongly advised against doing that by SW experts, so we are looking for other solutions .

@syuvivida
Copy link
Contributor

Hello,
The indexing of the development DQMGUI server vocms0731 is now back. One will start seeing output of comparison plots from the DQM bin-by-bin tools from the PR request.

@mmusich
Copy link
Contributor Author

mmusich commented Jul 25, 2023

The indexing of the development DQMGUI server vocms0731 is now back. One will start seeing output of comparison plots from the DQM bin-by-bin tools from the PR request.

OK, thanks I will close this issue for the time being.

@mmusich mmusich closed this as completed Jul 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants