APE tool: Trend plots, some minor fixes

[mteroerd]

PR description:

This PR is the summary of some changes that were made to the APE estimation tool during the UL2017 and UL2018 campaigns.

It includes:

• a tool for trend plots (hence, the data/ folder and additions in test/plottingTools)
• a way to change marker styles and colors for result plots
• a way to add the number of hits in a sector to result plots
• fix to a small typo in test/batch/startSkim.py
• fix to the wrong ideal alignment being loaded in test/cfgTemplate/apeEstimator_cfg.py
• additional functionality for test/autoSubmitter:
  • Basic support for submitting to condor "CAF queue".
  • Protection against cases where the condor scheduler is unavailable when submitting jobs which used to cause crashes. Now, in this case the submission is retried after some waiting time until it works.
  • Support for running multiple threads to handle measurements in case of large campaigns. Before, it happened that condor jobs (that do the track refits) of measurements finished while another measurement was being handled, so that there was some delay before these measurements were handled, which piled up to longer delay when doing many measurements.
  • Some minor changes like putting some functions into helper.py and introducing new shortcuts for convenience.

PR Validation:

The validation happened while using the tool. Since no changes were made that could affect the APE results themselves, everything was transparent.

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-27656/11208

• This PR adds an extra 36KB to repository

[cmsbuild]

A new Pull Request was created by @mteroerd (Marius Teroerde) for master.

It involves the following packages:

Alignment/APEEstimation

@christopheralanwest, @tocheng, @cmsbuild, @franzoni, @tlampen, @pohsun can you please review it and eventually sign? Thanks.
@mschrode, @mmusich, @tocheng, @adewit this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

[pohsun]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/1757/console Started: 2019/08/01 05:02

[cmsbuild]

+1
Tested at: c11c763
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9e24e7/1757/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9e24e7/1757/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /build/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-9e24e7/22034.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D46_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D46+RecoFullGlobal_2026D46+HARVESTFullGlobal_2026D46

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 32
• DQMHistoTests: Total histograms compared: 2484130
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2483812
• DQMHistoTests: Total skipped: 317
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 31 files compared)
• Checked 132 log files, 14 edm output root files, 32 DQM output files

[mteroerd]

I forgot to mention that this will probably also need a backport to the release on which UL2016ReReco will be done.

[pohsun]

+1
@mteroerd Please submit a PR for backport and relate to this one.

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-27656/11341

• This PR adds an extra 36KB to repository

[cmsbuild]

Pull request #27656 was updated. @christopheralanwest, @tocheng, @cmsbuild, @franzoni, @tlampen, @pohsun can you please check and sign again.

[fabiocos]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/1890/console Started: 2019/08/08 09:35
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/1897/console Started: 2019/08/08 10:43

[cmsbuild]

+1
Tested at: 71e1973
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9e24e7/1897/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9e24e7/1897/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 33
• DQMHistoTests: Total histograms compared: 2715989
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2715659
• DQMHistoTests: Total skipped: 329
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 32 files compared)
• Checked 137 log files, 14 edm output root files, 33 DQM output files

[pohsun]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[fabiocos]

+1