Adding Proxy Support to hltGetConfiguration and fixing converter/db specification bugs with hltIntegrationTests: 12_0_X

[Sam-Harper]

PR description:

The PR updates hltGetConfiguration and friends to be able to use a socks proxy thus simplifying hlt work outside the cern network.

It also fixes a bug in hltIntegrationTests reported by Andres Leonardo Cabrera Mora which has to do with how the database/converter is selected for the "setup menu" download.

--dbproxy will now activate this and --dbproxyhost/--dbproxyport will allow hosts/ports other than localhost:8080

Note this PR is a work in progress, the main v3 converter is not updated (only v3-test) and Andrea just made a good suggestion about alternatives to sha512 hashes which I would like explore. However I thought it would be nice to have the rest of the PR available to also receive comments rather than waiting to explore that suggestion.

PR validation:

 hltIntegrationTests v3-test/run3:/dev/CMSSW_12_0_0/Fake -s /dev/CMSSW_12_0_0/GRun -i /store/data/Run2018D/EphemeralHLTPhysics1/RAW/v1/000/323/790/00000/15FD6507-9B9A-DE49-973B-B63C4DEDC226.root -x  "--globaltag auto:run3_hlt" --dbproxy 
hltGetConfiguration v3-test/run3:/dev/CMSSW_12_0_0/Fake --dbproxy
hltConfigFromDB --configName /dev/CMSSW_12_0_0/GRun/V1  --dbproxy --v3-test --run3

commands all work as expected at RAL (ie very much off the cern network)

if this PR is a backport please specify the original PR and why you need to backport that PR:

backport of #35987

users will be still be using this release for a while and will need HLT services such as this

[cmsbuild]

A new Pull Request was created by @Sam-Harper (Sam Harper) for CMSSW_12_0_X.

It involves the following packages:

• HLTrigger/Configuration (hlt)

@cmsbuild, @missirol, @Martin-Grunewald can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @missirol, @silviodonato this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

• Backported from Adding Proxy Support to hltGetConfiguration and fixing converter/db specification bugs with hltIntegrationTests #35987

[missirol]

backport of #35987

[missirol]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0988ab/20366/summary.html
COMMIT: d5325dd
CMSSW: CMSSW_12_0_X_2021-11-08-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36052/20366/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 39
• DQMHistoTests: Total histograms compared: 2998564
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2998542
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
• Checked 165 log files, 37 edm output root files, 39 DQM output files
• TriggerResults: no differences found

[cmsbuild]

Pull request #36052 was updated. @cmsbuild, @missirol, @Martin-Grunewald can you please check and sign again.

[missirol]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0988ab/20394/summary.html
COMMIT: 07e0db7
CMSSW: CMSSW_12_0_X_2021-11-09-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36052/20394/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 8 differences found in the comparisons
• DQMHistoTests: Total files compared: 39
• DQMHistoTests: Total histograms compared: 2998564
• DQMHistoTests: Total failures: 12
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2998530
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
• Checked 165 log files, 37 edm output root files, 39 DQM output files
• TriggerResults: no differences found

[missirol]

-hlt

It was noticed in #35987 that another fix is needed on top of these updates.
A possible fix for master is in #36069, and the easiest is to include it in this PR.

[missirol]

Suggesting here the backport of the fix implemented in #36069 for master.

[cmsbuild]

Pull request #36052 was updated. @cmsbuild, @missirol, @Martin-Grunewald can you please check and sign again.

[missirol]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0988ab/20497/summary.html
COMMIT: 7e74329
CMSSW: CMSSW_12_0_X_2021-11-12-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36052/20497/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 39
• DQMHistoTests: Total histograms compared: 2998564
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2998536
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
• Checked 165 log files, 37 edm output root files, 39 DQM output files
• TriggerResults: no differences found

[missirol]

+hlt

• backport of #35987 + #36059 + #36069
• these updates only affect some of the scripts used for HLT development and testing;
  the backport is done to ease the work of HLT developers still using 12_0_X

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_0_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_2_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[qliphy]

+1