Merge pull request #46044 from wddgit/addSplit0TestFiles

Add tests with split level 0 input files to raw data unit tests

DataFormats/Common/test/TestTriggerResultsFormat.sh

@@ -10,7 +10,25 @@ file=testTriggerResults.root
 
 cmsRun ${LOCAL_TEST_DIR}/test_readTriggerResults_cfg.py "$file" || die "Failure using test_readTriggerResults_cfg.py $file" $?
 
-oldFiles="testTriggerResults_CMSSW_13_0_0.root  testTriggerResults_CMSSW_13_1_0_pre3.root"
+# The old files read below were generated as follows.
+#
+#     Check out the release indicated in the filename. Then cherry pick
+#     the commit that adds the original version of the file
+#     DataFormats/Common/test/TestWriteTriggerResults.cc.
+#     Except for BuildFile.xml, this only adds new test files.
+#     There may be minor conflicts or issues in test/BuildFile.xml
+#     that need to be resolved.
+#
+# Run cmsRun with DataFormats/Common/test/create_triggerresults_test_file_cfg.py
+# as the configuration and rename the file that creates.
+
+# Note that TriggerResults is declared in the classes_def.xml file with
+# a requirement that the product is always written with split level 0.
+# Most other raw data products are written with the default split level
+# for the output file. That is why all the test input files in this
+# shell script were written with split level 0.
+
+oldFiles="testTriggerResults_CMSSW_13_0_0_split_0.root  testTriggerResults_CMSSW_13_1_0_pre3_split_0.root"
 for file in $oldFiles; do
   inputfile=$(edmFileInPath DataFormats/Common/data/$file) || die "Failure edmFileInPath DataFormats/Common/data/$file" $?
   cmsRun ${LOCAL_TEST_DIR}/test_readTriggerResults_cfg.py "$inputfile" || die "Failed to read old file $file" $?

DataFormats/DetId/README.md

@@ -7,4 +7,4 @@ The type `std::vector<DetId>` is part of the RAW data, and any changes must be b
 * a file written by the same release
 * files written by (some) earlier releases
 
-If the persistent format of class `std::vector<DetId>` gets changed in the future, please adjust the `TestReadVectorDetId` and `TestWriteVectorDetId` modules accordingly. It is important that every member container has some content in this test. Please also add a new file to the [https://github.com/cms-data/DataFormats-DetId/](https://github.com/cms-data/DataFormats-DetId/) repository, and update the `TestVectorDetId` unit test to read the newly created file. The file name should contain the release or pre-release with which it was written.
+If the persistent format of class `std::vector<DetId>` gets changed in the future, please adjust the `TestReadVectorDetId` and `TestWriteVectorDetId` modules accordingly. It is important that every member container has some content in this test. Please also add new files to the [https://github.com/cms-data/DataFormats-DetId/](https://github.com/cms-data/DataFormats-DetId/) repository, and update the `TestVectorDetId` unit test to read the newly created files. There should be one file written with split level 0 and one with split level 99. The file name should contain the release or pre-release with which it was written and the split level.

DataFormats/DetId/test/TestVectorDetId.sh

@@ -10,7 +10,25 @@ file=testVectorDetId.root
 
 cmsRun ${LOCAL_TEST_DIR}/test_readVectorDetId_cfg.py "$file" || die "Failure using test_readVectorDetId_cfg.py $file" $?
 
-oldFiles="testVectorDetId_CMSSW_13_2_4.root"
+# The old files read below were generated as follows.
+#
+#     Check out the 13_2_4 release and cherry pick the commit that
+#     adds the original version of the file
+#     DataFormats/DetId/test/TestWriteVectorDetId.cc.
+#     Except for BuildFile.xml this only adds new test files.
+#     There may be minor conflicts or issues in test/BuildFile.xml
+#     that need to be resolved.
+#
+# Run cmsRun with DataFormats/DetId/test/create_VectorDetId_test_file_cfg.py
+# as the configuration and rename the file that creates.
+#
+# By default, split level 99 is used (maximum possible splitting).
+# If the suffix "_split_0" is near the end of the filename, the
+# following was added to the configuration of the output module:
+#     "splitLevel = cms.untracked.int32(0)"
+#
+
+oldFiles="testVectorDetId_CMSSW_13_2_4_split_99.root testVectorDetId_CMSSW_13_2_4_split_0.root"
 for file in $oldFiles; do
   inputfile=$(edmFileInPath DataFormats/DetId/data/$file) || die "Failure edmFileInPath DataFormats/DetId/data/$file" $?
   cmsRun ${LOCAL_TEST_DIR}/test_readVectorDetId_cfg.py "$inputfile" || die "Failed to read old file $file" $?

DataFormats/FEDRawData/test/TestFEDRawDataCollectionFormat.sh

@@ -10,7 +10,25 @@ file=testFEDRawDataCollection.root
 
 cmsRun ${LOCAL_TEST_DIR}/test_readFEDRawDataCollection_cfg.py "$file" || die "Failure using test_readFEDRawDataCollection_cfg.py $file" $?
 
-oldFiles="testFEDRawDataCollection_CMSSW_13_0_0.root testFEDRawDataCollection_CMSSW_13_1_0_pre3.root"
+# The old files read below were generated as follows.
+#
+#     Check out the release indicated in the filename. Then cherry pick
+#     the commit that adds the original version of the file
+#     DataFormats/FEDRawData/test/TestWriteFEDRawDataCollection.cc.
+#     Except for BuildFile.xml, this only adds new test files.
+#     There may be minor conflicts or issues in test/BuildFile.xml
+#     that need to be resolved.
+#
+# Run cmsRun with DataFormats/FEDRawData/test/create_FEDRawDataCollection_test_file_cfg.py
+# as the configuration and rename the file that creates.
+
+# Note that FEDRawDataCollection is declared in the classes_def.xml file with
+# a requirement that the product is always written with split level 0.
+# Most other raw data products are written with the default split level
+# for the output file. That is why all the test input files in this
+# shell script were written with split level 0.
+
+oldFiles="testFEDRawDataCollection_CMSSW_13_0_0_split_0.root testFEDRawDataCollection_CMSSW_13_1_0_pre3_split_0.root"
 for file in $oldFiles; do
   inputfile=$(edmFileInPath DataFormats/FEDRawData/data/$file) || die "Failure edmFileInPath DataFormats/FEDRawData/data/$file" $?
   cmsRun ${LOCAL_TEST_DIR}/test_readFEDRawDataCollection_cfg.py "$inputfile" || die "Failed to read old file $file" $?

DataFormats/HLTReco/test/TestTriggerEventFormat.sh

@@ -10,7 +10,25 @@ file=testTriggerEvent.root
 
 cmsRun ${LOCAL_TEST_DIR}/test_readTriggerEvent_cfg.py "$file" || die "Failure using test_readTriggerEvent_cfg.py $file" $?
 
-oldFiles="testTriggerEvent_CMSSW_13_0_0.root testTriggerEvent_CMSSW_13_1_0_pre3.root"
+# The old files read below were generated as follows.
+#
+#     Check out the release indicated in the filename. Then cherry pick
+#     the commit that adds the original version of the file
+#     DataFormats/HLTReco/test/TestWriteTriggerEvent.cc.
+#     Except for BuildFile.xml, this only adds new test files.
+#     There may be minor conflicts or issues in test/BuildFile.xml
+#     that need to be resolved.
+#
+# Run cmsRun with DataFormats/HLTReco/test/create_TriggerEvent_test_file_cfg.py
+# as the configuration and rename the file that creates.
+
+# Note that TriggerEvent is declared in the classes_def.xml file with
+# a requirement that the product is always written with split level 0.
+# Most other raw data products are written with the default split level
+# for the output file. That is why all the test input files in this
+# shell script were written with split level 0.
+
+oldFiles="testTriggerEvent_CMSSW_13_0_0_split_0.root testTriggerEvent_CMSSW_13_1_0_pre3_split_0.root"
 for file in $oldFiles; do
   inputfile=$(edmFileInPath DataFormats/HLTReco/data/$file) || die "Failure edmFileInPath DataFormats/HLTReco/data/$file" $?
   cmsRun ${LOCAL_TEST_DIR}/test_readTriggerEvent_cfg.py "$inputfile" || die "Failed to read old file $file" $?

DataFormats/L1Scouting/README.md

@@ -6,3 +6,6 @@ Any changes to the L1 scouting data formats must be backwards compatible.
 In order to ensure the L1 Scouting formats can be read by future CMSSW releases,
 there is a `TestWriteL1ScoutingDataFormats` unit test, which makes use of the `TestReadL1Scouting` analyzer and the `TestWriteL1Scouting` producer.
 The unit test checks that objects can be written and read properly. 
+If any of the data formats is changed, then 2 new input files
+for the test should be created, one for split level 0 and one for
+split level 99.

DataFormats/L1Scouting/test/TestL1ScoutingFormat.sh

@@ -10,14 +10,37 @@ file=testL1Scouting.root
 
 cmsRun ${LOCAL_TEST_DIR}/read_L1Scouting_cfg.py --inputFile "$file" || die "Failure using read_L1Scouting_cfg.py $file" $?
 
+# The old files read below were generated as follows.
+#
+# Check out the release in the filename and use it without modification to make
+# files with split level 99 (maximum possible splitting for each product)
+#
+# Then execute:
+#
+#   cmsRun DataFormats/L1Scouting/test/create_L1Scouting_test_file_cfg.py
+#
+# Rename the output file.
+#
+# The versions of the classes are encoded in the filenames in
+# alphabetical order. This order is also the order the classes
+# appear in classes_def.xml.
+#
+# For the split level 0 files, do the exact same thing except
+# add the following to the output module configuration.
+#     "splitLevel = cms.untracked.int32(0)"
+
 # test file for muon, jet, e/gamma and energy sums data formats
-oldFile="testL1Scouting_v3_v3_v3_v3_v3_13_3_0_pre5.root"
-inputfile=$(edmFileInPath DataFormats/L1Scouting/data/$oldFile) || die "Failure edmFileInPath DataFormats/L1Scouting/data/$oldFile" $?
-cmsRun ${LOCAL_TEST_DIR}/read_L1Scouting_cfg.py --inputFile "$inputfile" --bmtfStubVersion 0 || die "Failed to read old file $oldFile" $?
+oldFiles="testL1Scouting_v3_v3_v3_v3_v3_14_0_0_split_99.root testL1Scouting_v3_v3_v3_v3_v3_14_0_0_split_0.root"
+for file in $oldFiles; do
+  inputfile=$(edmFileInPath DataFormats/L1Scouting/data/$file) || die "Failure edmFileInPath DataFormats/L1Scouting/data/$file" $?
+  cmsRun ${LOCAL_TEST_DIR}/read_L1Scouting_cfg.py --inputFile "$inputfile" --bmtfStubVersion 0 || die "Failed to read old file $file" $?
+done
 
 # added BMTF input stubs data format
-oldFile="testL1Scouting_v3_v3_v3_v3_v3_v3_14_1_0_pre4.root"
-inputfile=$(edmFileInPath DataFormats/L1Scouting/data/$oldFile) || die "Failure edmFileInPath DataFormats/L1Scouting/data/$oldFile" $?
-cmsRun ${LOCAL_TEST_DIR}/read_L1Scouting_cfg.py --inputFile "$inputfile" --bmtfStubVersion 3 || die "Failed to read old file $oldFile" $?
+oldFiles="testL1Scouting_v3_v3_v3_v3_v3_v3_14_1_0_pre5_split_99.root testL1Scouting_v3_v3_v3_v3_v3_v3_14_1_0_pre5_split_0.root"
+for file in $oldFiles; do
+  inputfile=$(edmFileInPath DataFormats/L1Scouting/data/$file) || die "Failure edmFileInPath DataFormats/L1Scouting/data/$file" $?
+  cmsRun ${LOCAL_TEST_DIR}/read_L1Scouting_cfg.py --inputFile "$inputfile" --bmtfStubVersion 3 || die "Failed to read old file $file" $?
+done
 
 exit 0

DataFormats/L1ScoutingRawData/test/TestSDSRawDataCollectionFormat.sh

@@ -10,7 +10,22 @@ file=testSDSRawDataCollection.root
 
 cmsRun ${LOCAL_TEST_DIR}/read_SDSRawDataCollection_cfg.py "$file" || die "Failure using read_SDSRawDataCollection_cfg.py $file" $?
 
-oldFile="testSDSRawDataCollection_v3_CMSSW_13_3_0_pre5.root"
+# The old files read below were generated as follows.
+#
+# Check out the release in the filename and use it without modification.
+# Then execute:
+#
+#   cmsRun DataFormats/L1ScoutingRawData/test/create_SDSRawDataCollection_test_file_cfg.py
+#
+# Rename the output file. The versions of the class is encoded in the filename.
+#
+# Note that SDSRawDataCollection is declared in the classes_def.xml file with
+# a requirement that the product is always written with split level 0.
+# Most other raw data products are written with the default split level
+# for the output file. That is why all the test input files in this
+# shell script were written with split level 0.
+
+oldFile="testSDSRawDataCollection_v3_CMSSW_14_0_0_split_0.root"
 inputfile=$(edmFileInPath DataFormats/L1ScoutingRawData/data/$oldFile) || die "Failure edmFileInPath DataFormats/L1ScoutingRawData/data/$oldFile" $?
 cmsRun ${LOCAL_TEST_DIR}/read_SDSRawDataCollection_cfg.py "$inputfile" || die "Failed to read old file $oldFile" $?
 

DataFormats/L1TGlobal/README.md

@@ -7,4 +7,4 @@ The class `GlobalObjectMapRecord` is part of the RAW data, and any changes must
 * a file written by the same release
 * files written by (some) earlier releases
 
-If the persistent format of class `GlobalObjectMapRecord` gets changed in the future, please adjust the `TestReadGlobalObjectMapRecord` and `TestWriteGlobalObjectMapRecord` modules accordingly. It is important that every member container has some content in this test. Please also add a new file to the [https://github.com/cms-data/DataFormats-L1TGlobal/](https://github.com/cms-data/DataFormats-L1TGlobal/) repository, and update the `TestGlobalObjectMapRecord` unit test to read the newly created file. The file name should contain the release or pre-release with which it was written.
+If the persistent format of class `GlobalObjectMapRecord` gets changed in the future, please adjust the `TestReadGlobalObjectMapRecord` and `TestWriteGlobalObjectMapRecord` modules accordingly. It is important that every member container has some content in this test. Please also add new files to the [https://github.com/cms-data/DataFormats-L1TGlobal/](https://github.com/cms-data/DataFormats-L1TGlobal/) repository, and update the `TestGlobalObjectMapRecord` unit test to read the newly created files. There should be one file written with split level 0 and one with split level 99. The file name should contain the release or pre-release with which it was written and the split level.

DataFormats/L1TGlobal/test/TestGlobalObjectMapRecordFormat.sh

@@ -10,7 +10,22 @@ file=testGlobalObjectMapRecord.root
 
 cmsRun ${LOCAL_TEST_DIR}/test_readGlobalObjectMapRecord_cfg.py "$file" || die "Failure using test_readGlobalObjectMapRecord_cfg.py $file" $?
 
-oldFiles="testGlobalObjectMapRecord_CMSSW_13_0_0.root testGlobalObjectMapRecord_CMSSW_13_1_0_pre3.root"
+# The old files read below were generated as follows.
+#
+#     Check out the release in the filename and cherry pick the commit that
+#     adds the original version of the file:
+#     DataFormats/L1TGlobal/test/TestWriteGlobalObjectMapRecord.cc
+#
+# Run cmsRun with DataFormats/L1TGlobal/test/create_GlobalObjectMapRecord_test_file_cfg.py
+# as the configuration and rename the file that creates.
+#
+# By default, split level 99 is used (maximum possible splitting).
+# If the suffix "_split_0" is near the end of the filename, the
+# following was added to the configuration of the output module:
+#     "splitLevel = cms.untracked.int32(0)"
+
+oldFiles="testGlobalObjectMapRecord_CMSSW_13_0_0_split_99.root testGlobalObjectMapRecord_CMSSW_13_0_0_split_0.root"
+oldFiles+=" testGlobalObjectMapRecord_CMSSW_13_1_0_pre3_split_99.root testGlobalObjectMapRecord_CMSSW_13_1_0_pre3_split_0.root"
 for file in $oldFiles; do
   inputfile=$(edmFileInPath DataFormats/L1TGlobal/data/$file) || die "Failure edmFileInPath DataFormats/L1TGlobal/data/$file" $?
   cmsRun ${LOCAL_TEST_DIR}/test_readGlobalObjectMapRecord_cfg.py "$inputfile" || die "Failed to read old file $file" $?

DataFormats/Scouting/README.md

@@ -7,6 +7,6 @@ Any changes to the Scouting data formats must be backwards compatible. In order
 * a file written by the same release
 * files written by (some) earlier releases
 
-If the persistent format of any Scouting data format gets changed in the future, please adjust the `TestReadRun3Scouting` and `TestWriteRun3Scouting` modules accordingly. It is important that every member container has some content in this test. Please also add a new file to the [https://github.com/cms-data/DataFormats-Scouting/](https://github.com/cms-data/DataFormats-Scouting/) repository, and update the `TestRun3ScoutingDataFormats` unit test to read the newly created file. The file name should contain the version numbers of the data format classes (from classes_def.xml) in alphabetical order (they are in this alphabetical order already in classes_def.xml) and the release or pre-release with which it was written. If the latest file of Run 3 scouting before the update has not been used in data taking, the file can be deleted.
+If the persistent format of any Scouting data format gets changed in the future, please adjust the `TestReadRun3Scouting` and `TestWriteRun3Scouting` modules accordingly. It is important that every member container has some content in this test. Please also add new files to the [https://github.com/cms-data/DataFormats-Scouting/](https://github.com/cms-data/DataFormats-Scouting/) repository, and update the `TestRun3ScoutingDataFormats` unit test to read the newly created files. There should be one file written with split level 0 and one with split level 99. The file name should contain the version numbers of the data format classes (from classes_def.xml) in alphabetical order (they are in this alphabetical order already in classes_def.xml), the release or pre-release with which it was written and the split level. If the latest file of Run 3 scouting before the update has not been used in data taking, the file can be deleted.
 
 There are analogous tests for Run 2. It is unlikely those formats will change anymore. There will probably be analogous tests added in the future for runs after Run 3 which will need similar maintenance.