Use python generator when reading StallMonitor output #22601
Conversation
Long running jobs can create very large StallMonitor output files. Reading such a file and putting its information into a python list can take a lot of memory. Instead, switch to using a generator which allows us to only keep results from one line in memory at a time.
|
The code-checks are being triggered in jenkins. |
|
please test |
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-22601/3938 |
|
The tests are being triggered in jenkins. |
|
A new Pull Request was created by @Dr15Jones (Chris Jones) for master. It involves the following packages: FWCore/Concurrency @cmsbuild, @smuzaffar, @Dr15Jones can you please review it and eventually sign? Thanks. cms-bot commands are listed here |
|
Comparison job queued. |
|
+1 |
cmsbuild
added
core-approved
fully-signed
and removed
core-pending
pending-signatures
labels
|
This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2) |
|
Comparison is ready Comparison Summary:
|
|
+1 |
Long running jobs can create very large StallMonitor output files. Reading such a file and putting its information into a python list can take a lot of memory. Instead, switch to using a generator which allows us to only keep results from one line in memory at a time.