(DAQ) File based protocol update: "initemp" markers + "discardLS" feature [12_6_X]

[missirol]

backport of #40099

PR description:

From the description of #40099 by @smorovic:

Two changes are implemented for the file-based output protocol:

Early "initemp" file marker:

• A file marker is created in constructor (.initemp of .ini depending on the stream type and content availability at construction). From 12_6_X, as noticed in tests, beginRun (or globalBeginRun) in GlobalEvFOutputModule can be called after event processing in the source starts, so creating such markers at beginRun is no longer sufficient. Marker needs to be created early for hltd daemon to know which streams are in the run, and therefore wait for output completion until lumisection can be closed in the merging system. Standard INI file is still created with information from beginRun, except in case of DQMHistograms where it is empty, so creation was moved to constructor.

• Changes are implemented in the GlobalEvFOutputModule (data streams), DQMFileSaverPB (DQMHistograms stream) and L1/HLTriggerJsonMonitoring (L1/HLTRates streams). Note: in combination with correspoding changes in hltd, this patch is required for CMSSW_12_6_X being used in the HLT environment, as collecting output doesn't work with current version of the sw.

Discard LS:

• appearance of a file marker in hltd run directory will trigger discard of specific LS data locally in CMSSW. Motivation for this is freeing space in temporary ramdisks in HLT to allow data processing to be unblocked. With concurrent lumisections (N = 2 by default in HLT), LS potentially doesn't get closed until N new lumisections are queued, so a range of several lumisections is checked by the input source. In the output module, a new file marker check (this is a fast file operation done in ramdisk) is performed before each event is written. Discarding is implemented only for the data streams, since special streams are negligible in size.

PR validation:

None (relies on the validation done for #40099).

If this PR is a backport, please specify the original PR and why you need to backport that PR. If this PR will be backported, please specify to which release cycle the backport is meant for:

#40099

Fix (+new feature) for HLT online workflows.

[cmsbuild]

A new Pull Request was created by @missirol (Marino Missiroli) for CMSSW_12_6_X.

It involves the following packages:

• DQMServices/FileIO (dqm)
• EventFilter/Utilities (daq)
• HLTrigger/JSONMonitoring (hlt)

@Martin-Grunewald, @emanueleusai, @emeschi, @ahmad3213, @cmsbuild, @missirol, @jfernan2, @syuvivida, @pmandrik, @micsucmed, @smorovic, @rvenditti can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @silviodonato, @barvic, @fwyzard this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

• Backported from (DAQ) File based protocol update #40099

[missirol]

please test

[missirol]

type bugfix

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f231cc/29254/summary.html
COMMIT: ee1f105
CMSSW: CMSSW_12_6_X_2022-11-25-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/40156/29254/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 2 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3417239
• DQMHistoTests: Total failures: 3
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3417214
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 206 log files, 48 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[missirol]

+hlt

[smorovic]

+daq

[emanueleusai]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_6_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_13_0_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

[perrotta]

+1