L1T DemonstratorTools: support transparent compression of pattern files

[gpetruc]

PR description:

This PR updates the DemonstratorTools package to support transparent compression of files pattern files if they end up with .gz (gzip) or .xz (lzma), using boost iostreams.

This feature is already implemented in other tools outside CMSSW that work with pattern files, in the emp software framework and correlator trigger tools so having the feature in CMSSW is useful.

For the moment lzma is not recommended since the other emp software tools don't support it as they're built on a version of boost iostreams without lzma, but since that may change in the future and adding lzma is only 5 extra lines of code I thought it was easier to just add it already in CMSSW.

The PR also updates the costructor of the BoardDataWriter to make the extension of the file configurable, and updates the pieces of C++ code using that class in CMSSW to add this extra parameter.

• @tswilliams If you prefer, I can add an additonal constructor that doesn't take the file extension to keep backwards compatibility.
• The behaviour of the GTT dumpers in DemonstratorTools is unchanged, the output remains an uncompressed text file.
• The configuration of the Correlator pattern dumpers in Phase2L1ParticleFlow is updated to use compressed files and also to switch to the new EMP format introduced in L1T demonstrator tools: Support for EMP v0.7.x buffer file format #41768 above
• I have also a version of the Phase2L1ParticleFlow which introduces fillDescriptions() for the Correlator Layer 2 e/gamma L1TCtL2EgProducer (@cerminar), I can either include it in this PR or leave it for the later bigger PR that adds fillDescriptions() also for the Correlator Layer 1 producers. You can see the diff in gpetruc/cmssw@DemonstratorTools_compression_master...L1TCtL2EgProducer_fillDescriptions_master

Note that pattern file production is not run in any production workflow

The PR was tested with the usual code checks + runTheMatrix -l 23234.0 (to check nothing got broken elsewhere) and L1Trigger/Phase2L1ParticleFlow/test/make_l1ct_binaryFiles_cfg.py (which runs the correlator emulator + pattern file producers)

A PR to cms-l1t-offline is also under way

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-42115/36093

• This PR adds an extra 40KB to repository

[cmsbuild]

A new Pull Request was created by @gpetruc (Giovanni Petrucciani) for master.

It involves the following packages:

• L1Trigger/DemonstratorTools (l1)
• L1Trigger/Phase2L1ParticleFlow (upgrade, l1)

@epalencia, @AdrianoDee, @srimanob, @cmsbuild, @aloeliger can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @missirol this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

[gpetruc]

@cmsbuild, please test

[tswilliams]

Hi @gpetruc

@tswilliams If you prefer, I can add an additonal constructor that doesn't take the file extension to keep backwards compatibility.

I think it's fine without an additional backwards-compatible constructor. But on that point, I'll just mention @EmyrClement here so he's aware that he might have to update the HGC code to add the extra constructor argument in future.

After looking through the code, my only other comment is that it would be good to add a check on fileExt in the BoardDataWriter, so that it throws an exception if the extension isn't txt, txt.gz or txt.xz - to make it clear to people straight away if they make a mistake there.

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8f8fed/33425/summary.html
COMMIT: 0bb08f3
CMSSW: CMSSW_13_2_X_2023-06-27-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/42115/33425/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 2 lines to the logs
• Reco comparison results: 12 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3200270
• DQMHistoTests: Total failures: 7
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3200241
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 207 log files, 159 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[aloeliger]

+l1

[gpetruc]

Added the check following @tswilliams's suggestion

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-42115/36111

• This PR adds an extra 36KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File L1Trigger/DemonstratorTools/plugins/GTTFileWriter.cc modified in PR(s): Phase 2 forwardport FastHisto PR1130 #42129
  • File L1Trigger/DemonstratorTools/python/GTTFileWriter_cff.py modified in PR(s): Phase 2 forwardport FastHisto PR1130 #42129

[cmsbuild]

Pull request #42115 was updated. @epalencia, @AdrianoDee, @srimanob, @cmsbuild, @aloeliger can you please check and sign again.

[aloeliger]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8f8fed/33445/summary.html
COMMIT: b58fcf1
CMSSW: CMSSW_13_2_X_2023-06-29-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/42115/33445/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 5 lines to the logs
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3193832
• DQMHistoTests: Total failures: 4
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3193806
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 207 log files, 159 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[aloeliger]

+l1

[srimanob]

+Upgrade

From upgrade side, the relevant code is on L1Trigger/Phase2L1ParticleFlow, where the configuration is updated with extra parameters.

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

[perrotta]

+1