Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DAQ] fix unsafe vector handling in FedRawData input source (13_0_X) #41670

Merged
merged 1 commit into from
May 15, 2023

Conversation

smorovic
Copy link
Contributor

@smorovic smorovic commented May 14, 2023

PR description:

Rare, spurious crashes have been obsever in input source initialization in HLT. Analysis points to a possibility of a vector being reallocated to a different heap location while threads wait on a condition variable.
(Not an urgent issue, since it causes minimal impact online and no data loss).

Solution is to initialize vectors before spawning threads using them.

Also, an opportunity was taken to wrap CV variables within a unique pointer instead of having to delete them in destructor and to remove a memory synchronization point, since creation of a thread already is one.

PR validation:

Tested with 13_0_6 in a DAQ test cluster for production FedRawDataInputSource . Also covered by unit tests for both sources.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Backport of #41668
(only applicable code)
Applies to FedRawDataInputSource and the patch is identical to master branch for this class.
Another class, DAQSource, is only found in CMSSW_13_1_X and master and will not be backported to 13_0_X.

condition-variable thread that is part of a vector being expanded.
- remove synchronization point since creation of a thread already implies a memory barrier.
@cmsbuild
Copy link
Contributor

cmsbuild commented May 14, 2023

A new Pull Request was created by @smorovic (Srecko Morovic) for CMSSW_13_0_X.

It involves the following packages:

  • EventFilter/Utilities (daq)

@cmsbuild, @emeschi, @smorovic can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @missirol this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

@smorovic smorovic changed the title [DAQ] fix data race in input source (13_0_X) [DAQ] fix unsafe vector handling in input sources (13_0_X) May 14, 2023
@smorovic
Copy link
Contributor Author

please test

@smorovic smorovic changed the title [DAQ] fix unsafe vector handling in input sources (13_0_X) [DAQ] fix unsafe vector handling in FedRawData input source (13_0_X) May 14, 2023
@perrotta
Copy link
Contributor

backport of #41668

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-497fc7/32618/summary.html
COMMIT: 73231b0
CMSSW: CMSSW_13_0_X_2023-05-14-0000/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41670/32618/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially removed 2 lines from the logs
  • Reco comparison results: 16 differences found in the comparisons
  • DQMHistoTests: Total files compared: 49
  • DQMHistoTests: Total histograms compared: 3555563
  • DQMHistoTests: Total failures: 10
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3555531
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
  • Checked 213 log files, 164 edm output root files, 49 DQM output files
  • TriggerResults: no differences found

@smorovic
Copy link
Contributor Author

+1

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next CMSSW_13_0_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_13_2_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

@perrotta
Copy link
Contributor

(only applicable code)

@smorovic could you please add in the PR description which are the differences wrt the PR merged in master, and why?

@smorovic
Copy link
Contributor Author

Sure, I added explanation to the PR text.
In short, there is no DAQSource in 13_0_X, only FedRawDataInputSource class.

@perrotta
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit 1f37da9 into cms-sw:CMSSW_13_0_X May 15, 2023
@smorovic smorovic deleted the 13_0_X-daqsource-race branch August 24, 2023 08:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants