Juranic/geomxngs dirschema update (#1361)

* Create geomx-ngs-v2.2.yaml Update pattern and description for lab_processed/primary_analysis/*.xlsx * Update CHANGELOG.md * Documentation: Update geomx-ngs --------- Co-authored-by: Juan Puerto <=>
hubmapconsortium · Sep 12, 2024 · 0b76e77 · 0b76e77
1 parent 2a0fdfb
commit 0b76e77
Show file tree

Hide file tree

Showing 3 changed files with 148 additions and 1 deletion.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,8 @@
 # Changelog
 
+## v0.0.26 (in progress)
+- Update GeoMx NGS directory schema
+
 ## v0.0.25
 - Update GeoMx NGS directory schema
 - Added EPIC dataset field derived_dataset_type to UNIQUE_FIELDS_MAP

diff --git a/docs/geomx-ngs/current/index.md b/docs/geomx-ngs/current/index.md
@@ -28,7 +28,38 @@ Related files:
 <br>
 
 ## Directory schemas
-<summary><b>Version 2.1 (use this one)</b></summary>
+<summary><b>Version 2.2 (use this one)</b></summary>
+
+| pattern | required? | description |
+| --- | --- | --- |
+| <code>extras\/.*</code> | ✓ | Folder for general lab-specific files related to the dataset. |
+| <code>extras\/microscope_hardware\.json</code> | ✓ | **[QA/QC]** A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
+| <code>extras\/microscope_settings\.json</code> |  | **[QA/QC]** A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
+| <code>raw\/.*</code> | ✓ | All raw data files for the experiment. |
+| <code>raw\/[^\/]+_LabWorksheet.txt</code> | ✓ | An Excel spreadsheet to refer to in setting up the library. This file documents all of the samples from a single collection plate. Generated by DSP run, prior to sequencing. |
+| <code>raw\/[^\/]+_config\.ini</code> | ✓ | Needed to generate the DCC file from the fastq file. Contains pipeline processing parameters.  Generated by DSP run, prior to sequencing. |
+| <code>raw\/[^\/]+_SeqCodeIndices\.csv</code> | ✓ | A file with sample information needed by the Illumina software. Use the contents of the SeqCodeIndices.csv file to create a SampleSheet.csv for input to the Illumina sequencer. (NextSeq 1000/2000 users download a SampleSheet.csv and whitelist.txt instead of SeqCodeIndices.csv.)  Generated by DSP run. |
+| <code>raw\/markers\.csv</code> |  | A csv file describing any morphology markers used to guide ROI and/or AOI selection [this should be similar in structure to the antibodies file] |
+| <code>raw\/[^\/]*targets\.pkc</code> | ✓ | The file listing probe barcode sequence and corresponding gene symbol or proteins targeted by that probe. This should be consistent for the same probe panel. |
+| <code>raw\/additional_panels_used\.csv</code> |  | If multiple commercial probe panels were used, then the primary probe panel should be selected in the "oligo_probe_panel" metadata field. The additional panels must be included in this file. Each panel record should include:manufacturer, model/name, product code. |
+| <code>raw\/custom_probe_set\.csv</code> | ✓ | This file should contain any custom probes used and must be included if the metadata field "is_custom_probes_used" is "Yes". The file should minimally include:target gene id, probe seq, probe id. The contents of this file are modeled after the 10x Genomics probe set file (see <https://support.10xgenomics.com/spatial-gene-expression-ffpe/probe-sets/probe-set-file-descriptions/probe-set-file-descriptions#probe_set_csv_file>). |
+| <code>raw\/fastq\/.*</code> | ✓ | Raw sequencing files for the experiment |
+| <code>raw\/fastq\/oligo\/.*</code> | ✓ | Directory containing fastq files pertaining to oligo sequencing. |
+| <code>raw\/fastq\/oligo\/[^\/]+\.fastq\.gz</code> | ✓ | This is a gzip version of the fastq file. This file contains the cell barcode and unique molecular identifier (technical). |
+| <code>raw\/images\/.*</code> |  | Directory containing raw image files. This directory should include at least one raw file. |
+| <code>raw\/images\/overlay\.(?:jpeg&#124;tiff)</code> |  | State whether an overlay image was used to guide ROI selection. If an overlay is used, then the overlay details will be provided in the protocols.io protocol. If used, this needs to be uploaded. It is not included in the OME TIFF. This can be a JPEG or TIFF file |
+| <code>lab_processed\/.*</code> | ✓ | Experiment files that were processed by the lab generating the data. |
+| <code>lab_processed\/Initial\s{1}Dataset\.xlsx</code> | ✓ | **[QA/QC]** An excel spreadsheet that is downloaded from the GeoMx DSP Data Analysis Suite containing QA/QC metrics based on raw, unprocessed target counts. This file contains one row per AOI/segment and no analyses span AOI. The AOIs included in this file can come from different GeoMx runs and hence span Globus uploads. So care must be taken to make sure the appropriate AOIs are included in the file. |
+| <code>lab_processed\/annotations\.xlsx</code> |  | AOI specific annotations. This might include cell type and anatomical information. |
+| <code>lab_processed\/dcc\/.*</code> | ✓ | DCC files generated from fastq by the Nanostring GeoMx NGS Pipeline. |
+| <code>lab_processed\/dcc\/[^\/]+\.dcc</code> | ✓ | DCC files containing target probe counts, generated from fastq by the Nanostring GeoMx NGS Pipeline. |
+| <code>lab_processed\/images\/.*</code> | ✓ | Processed image files |
+| <code>lab_processed\/images\/[^\/]+\.ome\.tiff</code> (example: <code>lab_processed/images/HBM892.MDXS.293.ome.tiff</code>) | ✓ | OME-TIFF files (multichannel, multi-layered) produced by the microscopy experiment. If compressed, must use loss-less compression algorithm. For Visium this stitched file should only include the single capture area relevant to the current dataset. For GeoMx there will be one OME TIFF file per slide, with each slide including multiple AOIs. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0> |
+| <code>lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv</code> | ✓ | This file provides essential documentation pertaining to each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. The required fields are detailed <https://docs.google.com/spreadsheets/d/1xEJSb0xn5C5fB3k62pj1CyHNybpt4-YtvUs5SUMS44o/edit#gid=0> |
+| <code>lab_processed\/primary_analysis\/.*</code> | ✓ | Primary analysis results |
+| <code>lab_processed\/primary_analysis\/[^\/]+\.xlsx</code> | ✓ | **[QA/QC]** File containing results from initial processing by GeoMx DSP Data Analysis Suite including optional removal of segments and targets based on QC flags and LOQ and normalization using Q3 normalization. |
+
+<summary><b>Version 2.1</b></summary>
 
 | pattern | required? | description |
 | --- | --- | --- |

diff --git a/src/ingest_validation_tools/directory-schemas/geomx-ngs-v2.2.yaml b/src/ingest_validation_tools/directory-schemas/geomx-ngs-v2.2.yaml
@@ -0,0 +1,113 @@
+files:
+  -
+    pattern: extras\/.*
+    required: True
+    description: Folder for general lab-specific files related to the dataset.
+  -
+    pattern: extras\/microscope_hardware\.json
+    required: True
+    description: A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document.
+    is_qa_qc: True
+  -
+    pattern: extras\/microscope_settings\.json
+    required: False
+    description: A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document.
+    is_qa_qc: True
+  -
+    pattern: raw\/.*
+    required: True
+    description: All raw data files for the experiment.
+  -
+    pattern: raw\/[^\/]+_LabWorksheet.txt
+    required: True
+    description: An Excel spreadsheet to refer to in setting up the library. This file documents all of the samples from a single collection plate. Generated by DSP run, prior to sequencing.
+  -
+    pattern: raw\/[^\/]+_config\.ini
+    required: True
+    description: Needed to generate the DCC file from the fastq file. Contains pipeline processing parameters.  Generated by DSP run, prior to sequencing.
+  -
+    pattern: raw\/[^\/]+_SeqCodeIndices\.csv
+    required: True
+    description: A file with sample information needed by the Illumina software. Use the contents of the SeqCodeIndices.csv file to create a SampleSheet.csv for input to the Illumina sequencer. (NextSeq 1000/2000 users download a SampleSheet.csv and whitelist.txt instead of SeqCodeIndices.csv.)  Generated by DSP run.
+  -
+    pattern: raw\/markers\.csv
+    required: False
+    description: A csv file describing any morphology markers used to guide ROI and/or AOI selection [this should be similar in structure to the antibodies file]
+  -
+    pattern: raw\/[^\/]*targets\.pkc
+    required: True
+    description: The file listing probe barcode sequence and corresponding gene symbol or proteins targeted by that probe. This should be consistent for the same probe panel. 
+  -
+    pattern: raw\/additional_panels_used\.csv
+    required: False
+    description: If multiple commercial probe panels were used, then the primary probe panel should be selected in the "oligo_probe_panel" metadata field. The additional panels must be included in this file. Each panel record should include:manufacturer, model/name, product code.
+  -
+    pattern: raw\/custom_probe_set\.csv
+    required: True
+    description: This file should contain any custom probes used and must be included if the metadata field "is_custom_probes_used" is "Yes". The file should minimally include:target gene id, probe seq, probe id. The contents of this file are modeled after the 10x Genomics probe set file (see <https://support.10xgenomics.com/spatial-gene-expression-ffpe/probe-sets/probe-set-file-descriptions/probe-set-file-descriptions#probe_set_csv_file>).
+  -
+    pattern: raw\/fastq\/.*
+    required: True
+    description: Raw sequencing files for the experiment
+  -
+    pattern: raw\/fastq\/oligo\/.*
+    required: True
+    description: Directory containing fastq files pertaining to oligo sequencing.
+  -
+    pattern: raw\/fastq\/oligo\/[^\/]+\.fastq\.gz
+    required: True
+    description: This is a gzip version of the fastq file. This file contains the cell barcode and unique molecular identifier (technical).
+    is_qa_qc: False
+  -
+    pattern: raw\/images\/.*
+    required: False
+    description: Directory containing raw image files. This directory should include at least one raw file.
+  -
+    pattern: raw\/images\/overlay\.(?:jpeg|tiff)
+    required: False
+    description: State whether an overlay image was used to guide ROI selection. If an overlay is used, then the overlay details will be provided in the protocols.io protocol. If used, this needs to be uploaded. It is not included in the OME TIFF. This can be a JPEG or TIFF file
+  -
+    pattern: lab_processed\/.*
+    required: True
+    description: Experiment files that were processed by the lab generating the data.
+  -
+    pattern: lab_processed\/Initial\s{1}Dataset\.xlsx
+    required: True
+    description: An excel spreadsheet that is downloaded from the GeoMx DSP Data Analysis Suite containing QA/QC metrics based on raw, unprocessed target counts. This file contains one row per AOI/segment and no analyses span AOI. The AOIs included in this file can come from different GeoMx runs and hence span Globus uploads. So care must be taken to make sure the appropriate AOIs are included in the file.
+    is_qa_qc: True
+  -
+    pattern: lab_processed\/annotations\.xlsx
+    required: False
+    description: AOI specific annotations. This might include cell type and anatomical information.
+  -
+    pattern: lab_processed\/dcc\/.*
+    required: True
+    description: DCC files generated from fastq by the Nanostring GeoMx NGS Pipeline.
+  -
+    pattern: lab_processed\/dcc\/[^\/]+\.dcc
+    required: True
+    description: DCC files containing target probe counts, generated from fastq by the Nanostring GeoMx NGS Pipeline.
+  -
+    pattern: lab_processed\/images\/.*
+    required: True
+    description: Processed image files
+  -
+    pattern: lab_processed\/images\/[^\/]+\.ome\.tiff
+    required: True
+    description: OME-TIFF files (multichannel, multi-layered) produced by the microscopy experiment. If compressed, must use loss-less compression algorithm. For Visium this stitched file should only include the single capture area relevant to the current dataset. For GeoMx there will be one OME TIFF file per slide, with each slide including multiple AOIs. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0>
+    is_qa_qc: False
+    example: lab_processed/images/HBM892.MDXS.293.ome.tiff
+  -
+    pattern: lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv
+    required: True
+    description: This file provides essential documentation pertaining to each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. The required fields are detailed <https://docs.google.com/spreadsheets/d/1xEJSb0xn5C5fB3k62pj1CyHNybpt4-YtvUs5SUMS44o/edit#gid=0>
+    is_qa_qc: False
+  -
+    pattern: lab_processed\/primary_analysis\/.*
+    required: True
+    description: Primary analysis results
+  -
+    pattern: lab_processed\/primary_analysis\/[^\/]+\.xlsx
+    required: True
+    description: File containing results from initial processing by GeoMx DSP Data Analysis Suite including optional removal of segments and targets based on QC flags and LOQ and normalization using Q3 normalization.
+    is_qa_qc: True