Monitoring JSON changes in DAQ (80X) #12138

smorovic · 2015-10-28T11:49:41Z

Changes:

migrating to JSON based format for module legend used by f3mon
adding HLTErrorEvents field which will used to distinguish between losses caused by crashes in CMSSW and other reasons, for example hardware failures.

port of #12136 submitted for 75X and #12137 submitted for 76X

Config for LO NNPDF30 PDF needed for commit cms-sw#685

…M DB with additional information that distinguishes errors from crashes in HLT farm and other, purely DAQ related causes)

cmsbuild · 2015-10-28T11:51:11Z

A new Pull Request was created by @smorovic (Srecko Morovic) for CMSSW_8_0_X.

Monitoring JSON changes in DAQ (80X)

It involves the following packages:

DQMServices/Components
EventFilter/Utilities
HLTrigger/JSONMonitoring

@perrotta, @cmsbuild, @cvuosalo, @fwyzard, @emeschi, @Martin-Grunewald, @deguio, @slava77, @mommsen, @vanbesien, @danduggan can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @barvic this is something you requested to watch as well.
You can sign-off by replying to this message having '+1' in the first line of your reply.
You can reject by replying to this message having '-1' in the first line of your reply.
If you are a L2 or a release manager you can ask for tests by saying 'please test' in the first line of a comment.
@Degano you are the release manager for this.
You can merge this pull request by typing 'merge' in the first line of your comment.

mommsen · 2015-10-28T11:52:02Z

+1

Martin-Grunewald · 2015-10-28T11:55:02Z

please test

cmsbuild · 2015-10-28T11:56:08Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/9259/console

fwyzard · 2015-10-28T13:02:33Z

+1

deguio · 2015-10-28T17:39:57Z

+1

cmsbuild · 2015-10-28T18:14:59Z

-1

Tested at: 7a58df2
I found errors in the following unit tests:

---> test testPythonParameterSet had ERRORS

you can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12138/9259/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
9d76187
50895f5
deaf58f
b5589bb
9e5b112
e42a1c3
7bfe79f
368d54e
d7d8b2b
35de49e
015289e
304970d
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12138/9259/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12138/9259/git-merge-result

Dr15Jones · 2015-10-30T12:39:33Z

EventFilter/Utilities/src/FastMonitoringService.cc

    for(int i = 0; i < encModule_.current_; i++)
-      ost<<i<<"="<<((const edm::ModuleDescription *)(encModule_.decode(i)))->moduleLabel()<<" ";
-    return ost.str();
+       legendaVector.append(Json::Value(((const edm::ModuleDescription *)(encModule_.decode(i)))->moduleLabel()));


Please avoid use of C style casts. The C++ casts are much easier to notIce and avoid accidental casting away can const.

As the name suggests, this monitors the FFF ramdisk. Collects file delays and numbers of events. (cherry picked from commit 7b48600)

(cherry picked from commit 2ff8733)

cmsbuild · 2015-10-30T13:50:37Z

-1
Tested at: 8ab7ae7
When I ran the RelVals I found an error in the following worklfows:
50202.0 step3

runTheMatrix-results/50202.0_TTbar_13+TTbar_13+DIGIUP15_PU50+RECOUP15_PU50+HARVESTUP15_PU50/step3_TTbar_13+TTbar_13+DIGIUP15_PU50+RECOUP15_PU50+HARVESTUP15_PU50.log

1003.0 step2

runTheMatrix-results/1003.0_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM/step2_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM.log
----- Begin Fatal Exception 30-Oct-2015 13:36:46 CET-----------------------
An exception of category 'FatalRootError' occurred while
   [0] Calling EventProcessor::runToCompletion (which does almost everything after beginJob and before endJob)
   Additional Info:
      [a] Fatal Root Error: @SUB=TFile::Flush
error flushing file step2_inDQM.root (Disk quota exceeded)
----- End Fatal Exception -------------------------------------------------

you can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12138/9374/summary.html

Ramdisk 80x

…11-01-2300 missing dependency added which was dropped due to DQM/HLTEvF cleanup

…iple_initialization Initialize Sherpa only once for multiple lumi sections

Herwig: Add config for LO NNPDF30

upgrade AMPT to v2.25t5

* void* --> const void*

smorovic · 2015-11-03T09:27:02Z

@cmsbuild please test

cmsbuild · 2015-11-03T09:31:34Z

Pull request #12138 was updated. @perrotta, @cmsbuild, @cvuosalo, @fwyzard, @emeschi, @Martin-Grunewald, @deguio, @slava77, @mommsen, @vanbesien, @danduggan can you please check and sign again.

slava77 · 2015-11-03T11:35:51Z

@smorovic
there are merge conflicts in this PR. So, it can not proceed without the merge issues resolved.
Please rebase this to a more recent 80X IB (anything starting from CMSSW_8_0_X_2015-10-30-2300) and re-test there.

… definition

* void* --> const void*

…o jsonlegends-80X

cmsbuild · 2015-11-03T12:01:26Z

Pull request #12138 was updated. @vciulli, @covarell, @bendavid, @danduggan, @cvuosalo, @emeschi, @thuer, @cmsbuild, @deguio, @slava77, @mommsen, @vanbesien, @govoni can you please check and sign again.

smorovic · 2015-11-03T12:02:48Z

rebase got messed up. Will open another PR and branch for this.

kurtejung and others added 6 commits October 8, 2015 14:57

upgrade AMPT to v2.25t5

c314fb0

Herwig: Add config for LO NNPDF30

e3f34f8

Config for LO NNPDF30 PDF needed for commit cms-sw#685

alpha_S in LO set to 0.130

3fd6a72

new (json) format for module and path legend json file

eb15d12

outputmodules entry in json legend

29bf970

*adding HLTErrorEvents field to output JSON (to be used for filling S…

7a58df2

…M DB with additional information that distinguishes errors from crashes in HLT farm and other, purely DAQ related causes)

cmsbuild added this to the Next CMSSW_8_0_X milestone Oct 28, 2015

cmsbuild added reconstruction-pending dqm-pending hlt-pending daq-pending pending-signatures tests-pending orp-pending comparison-pending labels Oct 28, 2015

cmsbuild added daq-approved tests-started and removed daq-pending tests-pending labels Oct 28, 2015

cmsbuild added hlt-approved and removed hlt-pending labels Oct 28, 2015

cmsbuild added dqm-approved and removed dqm-pending labels Oct 28, 2015

Dr15Jones reviewed Oct 30, 2015
View reviewed changes

cmsdqm and others added 2 commits October 30, 2015 14:10

Implement "ramdisk" online client.

a40cc08

As the name suggests, this monitors the FFF ramdisk. Collects file delays and numbers of events. (cherry picked from commit 7b48600)

Fix code formatting DQMServices/StreamerIO

c9b9afb

(cherry picked from commit 2ff8733)

cmsbuild added tests-rejected and removed tests-started labels Oct 30, 2015

davidlange6 and others added 6 commits November 2, 2015 04:51

Merge pull request cms-sw#12209 from dmitrijus/ramdisk_76x

96658e0

Ramdisk 80x

Merge pull request cms-sw#12229 from smuzaffar/from-CMSSW_8_0_X_2015-…

01a95a7

…11-01-2300 missing dependency added which was dropped due to DQM/HLTEvF cleanup

Merge pull request cms-sw#12199 from thuer/CMSSW_8_0_X_sherpa_no_mult…

3ac9285

…iple_initialization Initialize Sherpa only once for multiple lumi sections

Merge pull request cms-sw#12025 from mharrend/patch-3

b1e354d

Herwig: Add config for LO NNPDF30

Merge pull request cms-sw#11932 from kurtejung/AMPT_upgrades

4be17af

upgrade AMPT to v2.25t5

*C++ style casts in FastMonitoringService

74d4bf2

* void* --> const void*

cmsbuild added daq-pending tests-pending and removed daq-approved tests-rejected labels Nov 3, 2015

smorovic added 3 commits November 3, 2015 12:58

*const correctness and using static const int instead of preprocessor…

6909b65

… definition

*C++ style casts in FastMonitoringService

c92781b

* void* --> const void*

Merge branch 'jsonlegends-80X' of ssh://github.com/smorovic/cmssw int…

97d6be2

…o jsonlegends-80X

cmsbuild added generators-pending and removed hlt-pending labels Nov 3, 2015

smorovic closed this Nov 3, 2015

smorovic mentioned this pull request Nov 3, 2015

Monitoring service changes for DAQ: rebase (80X) #12245

Merged

smorovic deleted the jsonlegends-80X branch November 13, 2015 10:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Monitoring JSON changes in DAQ (80X) #12138

Monitoring JSON changes in DAQ (80X) #12138

smorovic commented Oct 28, 2015

cmsbuild commented Oct 28, 2015

mommsen commented Oct 28, 2015

Martin-Grunewald commented Oct 28, 2015

cmsbuild commented Oct 28, 2015

fwyzard commented Oct 28, 2015

deguio commented Oct 28, 2015

cmsbuild commented Oct 28, 2015

Dr15Jones Oct 30, 2015

cmsbuild commented Oct 30, 2015

smorovic commented Nov 3, 2015

cmsbuild commented Nov 3, 2015

slava77 commented Nov 3, 2015

cmsbuild commented Nov 3, 2015

smorovic commented Nov 3, 2015

Monitoring JSON changes in DAQ (80X) #12138

Monitoring JSON changes in DAQ (80X) #12138

Conversation

smorovic commented Oct 28, 2015

cmsbuild commented Oct 28, 2015

mommsen commented Oct 28, 2015

Martin-Grunewald commented Oct 28, 2015

cmsbuild commented Oct 28, 2015

fwyzard commented Oct 28, 2015

deguio commented Oct 28, 2015

cmsbuild commented Oct 28, 2015

Dr15Jones Oct 30, 2015

Choose a reason for hiding this comment

cmsbuild commented Oct 30, 2015

smorovic commented Nov 3, 2015

cmsbuild commented Nov 3, 2015

slava77 commented Nov 3, 2015

cmsbuild commented Nov 3, 2015

smorovic commented Nov 3, 2015