Implement TH2Poly in DQM Services for HGCal DQM #41932

ywkao · 2023-06-12T15:51:27Z

PR description:

This PR introduces a new type of DQM MonitorElement, TH2Poly, for HGCal DQM in the future. This feature allows a display of polygonal histograms on the CMS DQM GUI. As a demonstration, a wafer map can be displayed like the screenshot here [1].

TH2Poly is a 2D histogram class inherited from TH2. Polygonal bins, defined by TGraph, can be loaded using the AddBin() method. After setting up the polygonal bins, a TH2Poly object can store information through Fill() or SetBinContent().

A workflow for creating polygonal histograms looks like this:
DQM Service -> DQM EDAnalyzer -> CMS DQM GUI

An implementation of TH2Poly in DQM Service and MonitorElement is necessary to display the polygonal histograms. It involves updates on two repositories: cmssw and dqmgui_prod. A pull request is created in the dqmgui_prod repository [2] with a relevant issue reported in this link [3], which is about setting up a CMS DQM GUI with the new feature.

PR validation:

The workflow and the implementation have been tested: (a) From this feature branch, monitor elements of TH2Poly can be stored in a DQM root file [4]. (b) The DQM root file can be uploaded to a CMS DQM GUI, which is built following the steps noted in this issue [3]. Polygonal maps can be displayed on the DQM GUI, as demonstrated in [1].

[1] https://ykao.web.cern.ch/ykao/raw_data_handling/hgcal_dqm_gui/screenshot_demo_th2poly_wafermap.png
[2] cms-DQM/dqmgui_prod#14
[3] cms-DQM/dqmgui_prod#13
[4] A DQM root file containing demo polygonal maps: /afs/cern.ch/work/y/ykao/public/example_HGCAL_DQM/DQM_V0001_HGCAL_R000123469.root

cmsbuild · 2023-06-12T15:59:08Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41932/35885

This PR adds an extra 44KB to repository

cmsbuild · 2023-06-12T15:59:32Z

A new Pull Request was created by @ywkao for master.

It involves the following packages:

DQMServices/Core (dqm)
DataFormats/Histograms (dqm, core)

@smuzaffar, @Dr15Jones, @makortel, @nothingface0, @emanueleusai, @cmsbuild, @pmandrik, @syuvivida, @tjavaid, @micsucmed, @rvenditti can you please review it and eventually sign? Thanks.
@missirol, @barvic, @rovere this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

makortel

I think in addition the following places need to be amended accordingly

cmssw/DQMServices/FwkIO/plugins/DQMRootOutputModule.cc

Lines 250 to 253 in e18c96d

    
           static TreeHelperBase* makeHelper(unsigned int iTypeIndex, TTree* iTree, std::string* iFullNameBufferPtr) { 
        
             switch (iTypeIndex) { 
        
               case kIntIndex: 
        
                 return new IntTreeHelper(iTree, iFullNameBufferPtr);

cmssw/DQMServices/FwkIO/plugins/DQMRootSource.cc

Lines 465 to 468 in e18c96d

    
             m_treeReaders[kTProfileIndex].reset(new TreeObjectReader<TProfile>(MonitorElementData::Kind::TPROFILE, m_rescope)); 
        
             m_treeReaders[kTProfile2DIndex].reset( 
        
                 new TreeObjectReader<TProfile2D>(MonitorElementData::Kind::TPROFILE2D, m_rescope)); 
        
           }

How about adding some tests to make sure all the I/O components work properly? (I don't remember if there are already any such tests that could be easily amended)

makortel · 2023-06-12T19:41:55Z

DQMServices/Core/interface/DQMStore.h

+                                 double lowY,
+                                 double highY,
+                                 FUNC onbooking = NOOP()) {
+        return bookME(name, MonitorElementData::Kind::TH2F, [=]() {


Shouldn't this be

Suggested change

return bookME(name, MonitorElementData::Kind::TH2F, [=]() {

return bookME(name, MonitorElementData::Kind::TH2Poly, [=]() {

?

Indeed, this should be corrected. I will update it in a commit accordingly.

makortel · 2023-06-12T19:43:04Z

DQMServices/Core/src/MonitorElement.cc

+  /// set polygon bin (TH2Poly)
+  void MonitorElement::addBin(TGraph *graph) {
+    auto access = this->accessMut();
+    static_cast<TH2Poly *>(accessRootObject(access, __PRETTY_FUNCTION__, 2))->AddBin(graph);


What if the ROOT object is not TH2Poly? The other similar functions throw an exception via the incompatible() function.

Indeed, incompatible() function is necessary. This block of code will be modified as follows:

- static_cast<TH2Poly *>(accessRootObject(access, __PRETTY_FUNCTION__, 2))->AddBin(graph); + if (kind() == Kind::TH2Poly) { + static_cast<TH2Poly *>(accessRootObject(access, __PRETTY_FUNCTION__, 2))->AddBin(graph); + } else { + incompatible(__PRETTY_FUNCTION__); + }

makortel · 2023-06-12T19:44:31Z

DataFormats/Histograms/interface/MonitorElementCollection.h

@@ -140,7 +140,8 @@ struct MonitorElementData {
    TH2I = 0x23,
    TH3F = 0x30,
    TPROFILE = 0x40,
-    TPROFILE2D = 0x41
+    TPROFILE2D = 0x41,
+    TH2Poly = 0x60


Why such a large gap between the previous value (0x41)?

I guess I kept thinking of hexagons for a wafer map and then put a number starting with six. We can assign 0x24 for TH2Poly if it is feasible.

+ TH2Poly = 0x24, TH3F = 0x30, TPROFILE = 0x40, - TPROFILE2D = 0x41, - TH2Poly = 0x60 + TPROFILE2D = 0x41

emanueleusai · 2023-06-13T03:13:04Z

type hgcal

emanueleusai · 2023-06-13T03:13:13Z

please test

cmsbuild · 2023-06-13T07:05:42Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-03d614/33112/summary.html
COMMIT: 5fa5e4e
CMSSW: CMSSW_13_2_X_2023-06-12-2300/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41932/33112/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 15 lines to the logs
Reco comparison results: 59 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3190910
DQMHistoTests: Total failures: 2477
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3188411
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 207 log files, 159 edm output root files, 48 DQM output files
TriggerResults: no differences found

cmsbuild · 2023-06-13T16:49:38Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41932/35914

This PR adds an extra 40KB to repository

cmsbuild · 2023-06-13T16:50:06Z

Pull request #41932 was updated. @smuzaffar, @Dr15Jones, @makortel, @nothingface0, @emanueleusai, @cmsbuild, @pmandrik, @syuvivida, @tjavaid, @micsucmed, @rvenditti can you please check and sign again.

ywkao · 2023-06-13T17:15:37Z

I think in addition the following places need to be amended accordingly

cmssw/DQMServices/FwkIO/plugins/DQMRootOutputModule.cc

Lines 250 to 253 in e18c96d

static TreeHelperBase* makeHelper(unsigned int iTypeIndex, TTree* iTree, std::string* iFullNameBufferPtr) {

switch (iTypeIndex) {

case kIntIndex:

return new IntTreeHelper(iTree, iFullNameBufferPtr);

cmssw/DQMServices/FwkIO/plugins/DQMRootSource.cc

Lines 465 to 468 in e18c96d

m_treeReaders[kTProfileIndex].reset(new TreeObjectReader<TProfile>(MonitorElementData::Kind::TPROFILE, m_rescope));

m_treeReaders[kTProfile2DIndex].reset(

new TreeObjectReader<TProfile2D>(MonitorElementData::Kind::TPROFILE2D, m_rescope));

}

How about adding some tests to make sure all the I/O components work properly? (I don't remember if there are already any such tests that could be easily amended)

Okay, I will modify the lines and test it. Would PR #37665 be a good reference for the implementation of TH2Poly?

Also, I would appreciate any instructions from experts on how to test all the I/O components.

makortel · 2023-06-13T20:54:56Z

Would PR #37665 be a good reference for the implementation of TH2Poly?

That PR indeeds looks like a good example.

IzaakWN · 2024-12-09T13:15:32Z

Hi @mmusich, @makortel, is there anything else needed from this PR? It would be very convenient for testing PR #45665 along with other HGCal tools if this PR can still make it into the next (pre-)release (CMSSW_15_0_0_pre1?). (Apologies for pushing.)

mmusich · 2024-12-09T13:19:14Z

is there anything else needed from this PR?

it doesn't depend on me :)
@cms-sw/core-l2 and @cms-sw/dqm-l2 should review the PR and sign it if they are happy.
It looks like you might want to address the GUI PR cms-DQM/dqmgui_prod#14 as well, before pushing this in production.

makortel · 2024-12-09T15:10:15Z

DQMServices/Components/plugins/DQMStoreStats.cc

+      //  curr->update(it->getNbinsX() * it->getNbinsY(),
+      //               getEmptyMetric(it->getTH2Poly()->GetArray(), it->getNbinsX() + 2, it->getNbinsY() + 2, 0),
+      //               it->getNbinsX() * it->getNbinsY() * sizeof(int));
+      //  break;


What about this code? Is something needed here? (I have no idea what exactly is being done in this function in general)

I am not sure all the details either. I referred to PR#37665-files, the implementation of DQM support of TH1I and TH2I.

@mmusich could you help evaluate the necessity of having TH2Poly::GetArray() functionality? Maybe we need help from ROOT experts.

makortel · 2024-12-09T15:19:32Z

DataFormats/Histograms/interface/MEtoEDMFormat.h

+        //       double value2 = newMEtoEDMObject[i].object.GetBinContent(ibin);
+        //       double total = value1 + value2;
+        //       MEtoEdmObject[j].object.SetBinContent(ibin, total);
+        //   }


Does uncommenting of this block not compile? If that is the case (i.e. that motivates the specialization of the entire mergeProduct() function), I think we need to think a bit for a solution that wouldn't require duplicating the entire logic of the mergeProduct() function.

Yes, uncommenting this block of code can now compile without an error if a dummy structure in DataFormats/Histograms/test/metoedmformat_t.cc is properly handled:

+ void Add(const Dummy* iOther, float weight=1.) { m_i += iOther->m_i; } + int GetNcells() const { return 9; } + float GetBinContent(int bin) const { return -1.; } + void SetBinContent(int bin, float value) const { return; } + const TClass* IsA() const { return 0; }

We can also remove the ad-hoc mergeProduct() for TH2Poly and have minimal modification on L151:

- MEtoEdmObject[j].object.Add(&newMEtoEDMObject[i].object); + MEtoEdmObject[j].object.Add(&newMEtoEDMObject[i].object, 1.);

After the modification of L151, I found that removing this ad-hoc TH2Poly addition code can also pass compilation and unit tests in DQMServices/Demo/test/runtests.sh. To ensure the complete functionality, I plan to:

Add polygonal bins test cases in DQMServices/Demo/test/TestDQMEDAnalyzer.cc.

Produce a few DQM root files for verification.

antoniovagnerini · 2024-12-09T19:15:07Z

is there anything else needed from this PR?

it doesn't depend on me :) @cms-sw/core-l2 and @cms-sw/dqm-l2 should review the PR and sign it if they are happy. It looks like you might want to address the GUI PR cms-DQM/dqmgui_prod#14 as well, before pushing this in production.

hi, since this PR touches the DQMStore which is the basic class that the clients use to manipulate/fill Histograms, we need to test it in our playback system DQM machines. The latter is unfortunately down for the programmed YETS, so I am afraid we will have to postpone this batch of test to the new year.

smuzaffar · 2024-12-18T07:51:04Z

code-checks

cmsbuild · 2024-12-18T07:52:51Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41932/43100

There are other open Pull requests which might conflict with changes you have proposed:
- File DQMServices/Components/plugins/DQMStoreStats.cc modified in PR(s): [LLVM Analyzer][DQM] Fix dead assignments #46247
- File DQMServices/Core/src/DQMNet.cc modified in PR(s): [LLVM Analyzer][DQM] Fix dead assignments #46247
- File DQMServices/FwkIO/plugins/DQMRootOutputModule.cc modified in PR(s): [RFC] descriptions.addDefault -> descriptions.addWithDefaultLabel #44372
- File DQMServices/FwkIO/plugins/DQMRootSource.cc modified in PR(s): [RFC] descriptions.addDefault -> descriptions.addWithDefaultLabel #44372

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-checks:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41932/43100/code-checks.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41932/43100/code-checks.patch | patch -p1
You can also run scram build code-checks to apply code checks directly
code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41932/43100/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41932/43100/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2024-12-19T07:38:50Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41932/43111

There are other open Pull requests which might conflict with changes you have proposed:
- File DQMServices/Components/plugins/DQMStoreStats.cc modified in PR(s): [LLVM Analyzer][DQM] Fix dead assignments #46247
- File DQMServices/Core/src/DQMNet.cc modified in PR(s): [LLVM Analyzer][DQM] Fix dead assignments #46247
- File DQMServices/FwkIO/plugins/DQMRootOutputModule.cc modified in PR(s): [RFC] descriptions.addDefault -> descriptions.addWithDefaultLabel #44372
- File DQMServices/FwkIO/plugins/DQMRootSource.cc modified in PR(s): [RFC] descriptions.addDefault -> descriptions.addWithDefaultLabel #44372

cmsbuild · 2024-12-19T07:39:12Z

Pull request #41932 was updated. @Dr15Jones, @antoniovagnerini, @cmsbuild, @makortel, @rseidita, @smuzaffar can you please check and sign again.

antoniovagnerini · 2024-12-20T09:48:34Z

please test

cmsbuild · 2024-12-20T12:14:07Z

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-03d614/43585/summary.html
COMMIT: 07470c1
CMSSW: CMSSW_15_0_X_2024-12-19-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41932/43585/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 196 lines from the logs
Reco comparison results: 11 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3818210
DQMHistoTests: Total failures: 444
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3817746
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: found differences in 1 / 47 workflows

cmsbuild added this to the CMSSW_13_2_X milestone Jun 12, 2023

cmsbuild added dqm-pending core-pending pending-signatures tests-pending orp-pending code-checks-pending labels Jun 12, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 12, 2023

makortel reviewed Jun 12, 2023

View reviewed changes

cmsbuild added tests-started hgcal and removed tests-pending labels Jun 13, 2023

cmsbuild added tests-approved tests-pending code-checks-pending and removed tests-started tests-approved code-checks-approved labels Jun 13, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 13, 2023

cmsbuild added tests-approved and removed tests-started labels Dec 6, 2024

nothingface0 mentioned this pull request Dec 6, 2024

DQM monitor element of Th2poly cms-DQM/dqmgui_prod#14

Open

makortel reviewed Dec 9, 2024

View reviewed changes

This was referenced Dec 13, 2024

[cmssw_deploy] conficts during application of cherry-picked PRs cms-DQM/fff_dqmtools#29

Closed

Script to do release comparison cms-DQM/fff_dqmtools#27

Open

cmsbuild added code-checks-pending and removed code-checks-approved labels Dec 18, 2024

cmsbuild added code-checks-rejected and removed code-checks-pending labels Dec 18, 2024

apply code checks and format recommendation

07470c1

cmsbuild added tests-pending code-checks-pending and removed tests-approved code-checks-rejected labels Dec 19, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Dec 19, 2024

cmsbuild added tests-started and removed tests-pending labels Dec 20, 2024

cmsbuild added tests-approved and removed tests-started labels Dec 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement TH2Poly in DQM Services for HGCal DQM #41932

Implement TH2Poly in DQM Services for HGCal DQM #41932

ywkao commented Jun 12, 2023

cmsbuild commented Jun 12, 2023

cmsbuild commented Jun 12, 2023 •

edited

Loading

makortel left a comment

makortel Jun 12, 2023

ywkao Jun 13, 2023

makortel Jun 12, 2023

ywkao Jun 13, 2023 •

edited

Loading

makortel Jun 12, 2023

ywkao Jun 13, 2023

emanueleusai commented Jun 13, 2023

emanueleusai commented Jun 13, 2023

cmsbuild commented Jun 13, 2023

cmsbuild commented Jun 13, 2023

cmsbuild commented Jun 13, 2023

ywkao commented Jun 13, 2023

makortel commented Jun 13, 2023

IzaakWN commented Dec 9, 2024

mmusich commented Dec 9, 2024

makortel Dec 9, 2024

ywkao Dec 16, 2024

makortel Dec 9, 2024

ywkao Dec 16, 2024

ywkao Dec 17, 2024

antoniovagnerini commented Dec 9, 2024

smuzaffar commented Dec 18, 2024

cmsbuild commented Dec 18, 2024

cmsbuild commented Dec 19, 2024

cmsbuild commented Dec 19, 2024

antoniovagnerini commented Dec 20, 2024

cmsbuild commented Dec 20, 2024

	static TreeHelperBase* makeHelper(unsigned int iTypeIndex, TTree* iTree, std::string* iFullNameBufferPtr) {
	switch (iTypeIndex) {
	case kIntIndex:
	return new IntTreeHelper(iTree, iFullNameBufferPtr);

	m_treeReaders[kTProfileIndex].reset(new TreeObjectReader<TProfile>(MonitorElementData::Kind::TPROFILE, m_rescope));
	m_treeReaders[kTProfile2DIndex].reset(
	new TreeObjectReader<TProfile2D>(MonitorElementData::Kind::TPROFILE2D, m_rescope));
	}

	return bookME(name, MonitorElementData::Kind::TH2F, [=]() {
	return bookME(name, MonitorElementData::Kind::TH2Poly, [=]() {

Implement TH2Poly in DQM Services for HGCal DQM #41932

Are you sure you want to change the base?

Implement TH2Poly in DQM Services for HGCal DQM #41932

Conversation

ywkao commented Jun 12, 2023

PR description:

PR validation:

cmsbuild commented Jun 12, 2023

cmsbuild commented Jun 12, 2023 • edited Loading

makortel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ywkao Jun 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emanueleusai commented Jun 13, 2023

emanueleusai commented Jun 13, 2023

cmsbuild commented Jun 13, 2023

Comparison Summary

cmsbuild commented Jun 13, 2023

cmsbuild commented Jun 13, 2023

ywkao commented Jun 13, 2023

makortel commented Jun 13, 2023

IzaakWN commented Dec 9, 2024

mmusich commented Dec 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antoniovagnerini commented Dec 9, 2024

smuzaffar commented Dec 18, 2024

cmsbuild commented Dec 18, 2024

cmsbuild commented Dec 19, 2024

cmsbuild commented Dec 19, 2024

antoniovagnerini commented Dec 20, 2024

cmsbuild commented Dec 20, 2024

Comparison Summary

cmsbuild commented Jun 12, 2023 •

edited

Loading

ywkao Jun 13, 2023 •

edited

Loading