Create infrastructure to allow cmsRun to control an external process #28792

Dr15Jones · 2020-01-25T19:36:25Z

PR description:

This adds the FWCore/SharedMemory package which can be used to allow bi-directional communication between a cmsRun module and an external process running on the same node. This includes an example implementation in FWCore/Integration.

PR validation:

The code compiles and all new and existing framework unit tests pass.

This allows testing of memory systems.

The buffer also properly handles growing by alternating between two different shared memory spaces. Additional improvements to external process failures were also added.

The WorkerChannel and ControllerChannel encapsulate the inter-process communication needed.

-Have the ability to also control EDFilters with keepEvents info -Have external module configuration come from the python config.

The recommended way to have a signal handler hand off dealing with the signal to a different thread is to use a pipe between the signal handler and the thread.

cmsbuild · 2020-01-25T19:36:52Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-01-25T19:37:11Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-01-25T19:44:00Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28792/13475

This PR adds an extra 76KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-checks:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28792/13475/code-checks.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28792/13475/code-checks.patch | patch -p1
You can also run scram build code-checks to apply code checks directly

cmsbuild · 2020-01-25T19:51:08Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-02-06T16:22:32Z

Pull request #28792 was updated. @makortel, @smuzaffar, @cmsbuild, @Dr15Jones can you please check and sign again.

cmsbuild · 2020-02-06T17:35:41Z

+1
Tested at: 252c9d9
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b77b1e/4533/summary.html
CMSSW: CMSSW_11_1_X_2020-02-06-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-02-06T17:35:44Z

Comparison job queued.

cmsbuild · 2020-02-06T19:03:36Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b77b1e/4533/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2698043
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2697696
DQMHistoTests: Total skipped: 346
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
Checked 147 log files, 16 edm output root files, 34 DQM output files

makortel · 2020-02-06T22:48:19Z

+1

cmsbuild · 2020-02-06T22:48:44Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @silviodonato, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

silviodonato · 2020-02-07T09:01:34Z

+1

silviodonato · 2020-02-07T16:19:31Z

@Dr15Jones a unit test of the newly created FWCore/SharedMemory is crashing
https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/raw/slc7_amd64_gcc820/CMSSW_11_1_X_2020-02-07-1100/unitTestLogs/FWCore/SharedMemory
Could you have a look?

Looking at the PR tests https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b77b1e/4533/unitTests.log, I see that testFWCoreSharedMemoryMonitorThreadSignals was aborted for some reason.

===== Test "testFWCoreSharedMemoryMonitorThreadSignals" ====
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_11_1_X_2020-02-06-1100/src/FWCore/SharedMemory/test/test_monitor_thread_signals.sh: line 10: 13491 Aborted                 ${CMSSW_BASE}/test/${SCRAM_ARCH}/testFWCoreSharedMemoryMonitorThread 6 &>testFWCoreSharedMemoryMonitorThread.log
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_11_1_X_2020-02-06-1100/src/FWCore/SharedMemory/test/test_monitor_thread_signals.sh: line 22: 13551 Segmentation fault      ${CMSSW_BASE}/test/${SCRAM_ARCH}/testFWCoreSharedMemoryMonitorThread 11 &>testFWCoreSharedMemoryMonitorThread.log

---> test testFWCoreSharedMemoryMonitorThreadSignals succeeded
 
^^^^ End Test testFWCoreSharedMemoryMonitorThreadSignals ^^^^

Dr15Jones · 2020-02-07T17:19:59Z

Looking at the PR tests https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b77b1e/4533/unitTests.log, I see that testFWCoreSharedMemoryMonitorThreadSignals was aborted for some reason.

That is exactly what the test is supposed to do. It internally sends an abort (and in another test a SIGSEGV) to be sure the program properly shutsdown. That is why the pull request test also had

---> test testFWCoreSharedMemoryMonitorThreadSignals succeeded

Dr15Jones · 2020-02-07T17:24:37Z

The IB unit test failure has a return code of 127 which means the shell was unable to find the program. Presumably that means it could not find testFWCoreSharedMemoryMonitorThread because it was not built yet.

@smuzaffar what was the proper command to add to the BuildFile.xml directive to let scram know there is a dependency between two test executables?

smuzaffar · 2020-02-07T17:36:44Z

It is PRE_TEST ( see examples here https://github.com/cms-sw/cmssw/search?q=PRE_TEST&unscoped_q=PRE_TEST )

Dr15Jones · 2020-02-07T22:03:07Z

Turns out the test already does specify PRE_TEST.

Dr15Jones · 2020-02-08T21:22:17Z

@smuzaffar I think the problem is the way I'm trying to find a built test

https://github.com/cms-sw/cmssw/blob/master/FWCore/SharedMemory/test/test_monitor_thread_signals.sh#L10

Do you have a better way to find it?

smuzaffar · 2020-02-10T08:11:05Z

Note that test/arch is part of PATH during scram build runtests so just calling testFWCoreSharedMemoryMonitorThread should work but if you want test_monitor_thread_signals.sh to work standalone then you can do some thing like

testpath="testFWCoreSharedMemoryMonitorThread"
for dir in $CMSSW_BASE $CMSSW_RELEASE_BASE $CMSSW_FULL_RELEASE_BASE ; do
  if [ -e $dir/test/${SCRAM_ARCH}/testFWCoreSharedMemoryMonitorThread ] ; then
    testpath=$dir/test/${SCRAM_ARCH}/testFWCoreSharedMemoryMonitorThread
    break
  fi
done
$testpath 6 >& testFWCoreSharedMemoryMonitorThread.log
``

We can provide a helper script (which will always in path) to find a cmssw file e.g `cmssw-find test/arch/name` could return full path to `test/arch/name` by looking in to CMSSW_BASE, CMSSW_RELEASE_BASE and CMSSW_FULL_RELEASE_BASE.

…(106X) Backport from cms-sw#28792. Additional changes: - FWCore/Integration/bin/interprocess.cc: L108: disable macro CMS_SA_ALLOW. - FWCore/Integration/test/TestInterProcessProd.cc: L3: add header "FWCore/Framework/interface/Run.h". - FWCore/Integration/test/test_TestInterProcessProd_cfg.py: L35,46: default way to specify process.maxEvents, process.options.

Dr15Jones added 13 commits January 24, 2020 15:22

Working test of inter process communication

aa2183f

Full serialize/deserialize using shared memory

ed6fd00

Handle data products created on all transitions

db21a16

Handle external process getting a unix signal

7400631

Added grow parameter to ThingAlgorithm

7f850bf

This allows testing of memory systems.

Share the same memory buffer for serialization

8988a9a

The buffer also properly handles growing by alternating between two different shared memory spaces. Additional improvements to external process failures were also added.

Created 'channel' classes for communication

5c90215

The WorkerChannel and ControllerChannel encapsulate the inter-process communication needed.

Allow buffers to and from Worker

df04325

Additional changes to generalize behavior

a9b93a4

-Have the ability to also control EDFilters with keepEvents info -Have external module configuration come from the python config.

Added FWCore/SharedMemory package

cc1e79d

Use self-pipe trick with signal handler

cd0eecb

The recommended way to have a signal handler hand off dealing with the signal to a different thread is to use a pipe between the signal handler and the thread.

Better handling of the WorkerMonitorThread signals test

f50fba0

Add test for TestInterProcessProd

aed7465

cmsbuild added this to the CMSSW_11_1_X milestone Jan 25, 2020

cmsbuild added code-checks-pending comparison-pending core-pending new-package-pending orp-pending pending-signatures tests-pending labels Jan 25, 2020

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jan 25, 2020

Apply code format

e3706a2

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jan 25, 2020

cmsbuild added tests-started and removed tests-pending labels Feb 6, 2020

cmsbuild added tests-approved and removed tests-started labels Feb 6, 2020

cmsbuild added comparison-available and removed comparison-pending labels Feb 6, 2020

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels Feb 6, 2020

cmsbuild added orp-approved and removed orp-pending labels Feb 7, 2020

cmsbuild merged commit 7b25496 into cms-sw:master Feb 7, 2020

Dr15Jones deleted the testInterProcess branch February 10, 2020 16:09

colizz mentioned this pull request Jul 12, 2020

Backport ExternalGeneratorFilter and core functions it depends on to 10_6_X #30649

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create infrastructure to allow cmsRun to control an external process #28792

Create infrastructure to allow cmsRun to control an external process #28792

Dr15Jones commented Jan 25, 2020

cmsbuild commented Jan 25, 2020

cmsbuild commented Jan 25, 2020

cmsbuild commented Jan 25, 2020

cmsbuild commented Jan 25, 2020

cmsbuild commented Feb 6, 2020

cmsbuild commented Feb 6, 2020

cmsbuild commented Feb 6, 2020

cmsbuild commented Feb 6, 2020

makortel commented Feb 6, 2020

cmsbuild commented Feb 6, 2020

silviodonato commented Feb 7, 2020

silviodonato commented Feb 7, 2020

Dr15Jones commented Feb 7, 2020

Dr15Jones commented Feb 7, 2020

smuzaffar commented Feb 7, 2020

Dr15Jones commented Feb 7, 2020

Dr15Jones commented Feb 8, 2020

smuzaffar commented Feb 10, 2020

Create infrastructure to allow cmsRun to control an external process #28792

Create infrastructure to allow cmsRun to control an external process #28792

Conversation

Dr15Jones commented Jan 25, 2020

PR description:

PR validation:

cmsbuild commented Jan 25, 2020

cmsbuild commented Jan 25, 2020

cmsbuild commented Jan 25, 2020

cmsbuild commented Jan 25, 2020

cmsbuild commented Feb 6, 2020

cmsbuild commented Feb 6, 2020

cmsbuild commented Feb 6, 2020

cmsbuild commented Feb 6, 2020

makortel commented Feb 6, 2020

cmsbuild commented Feb 6, 2020

silviodonato commented Feb 7, 2020

silviodonato commented Feb 7, 2020

Dr15Jones commented Feb 7, 2020

Dr15Jones commented Feb 7, 2020

smuzaffar commented Feb 7, 2020

Dr15Jones commented Feb 7, 2020

Dr15Jones commented Feb 8, 2020

smuzaffar commented Feb 10, 2020