Juranic/mibi visium dirschema update (#1352)

* Create mibi-v2.1.yaml Add extras\/microscope_hardware\.json and extras\/microscope_settings\.json; change raw\/images\/[^\/]+\.ome\.tiff from required to optional * Create visium-no-probes-v2.2.yaml Remove segmentation masks (lab_processed\/annotations\/.*, that has segmentation-masks-v2 dependency) * Update CHANGELOG.md * General: Update mibi & visium docs. --------- Co-authored-by: jpuerto-psc <[email protected]> Co-authored-by: Juan Puerto <=>
hubmapconsortium · Aug 27, 2024 · 38863a3 · 38863a3
1 parent 093446f
commit 38863a3
Show file tree

Hide file tree

Showing 5 changed files with 168 additions and 2 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -9,6 +9,8 @@
 - Update Phenocycler docs
 - Update MERFISH directory schema
 - Add next-gen Cell DIVE directory schema
+- Update MIBI directory schema
+- Update Visium no probes directory schema
 
 ## v0.0.23
 - Add token to validation_utils.get_assaytype_data, replace URL string concatenation with urllib

diff --git a/docs/mibi/current/index.md b/docs/mibi/current/index.md
@@ -28,7 +28,23 @@ Related files:
 <br>
 
 ## Directory schemas
-<summary><b>Version 2.0 (use this one)</b></summary>
+<summary><b>Version 2.1 (use this one)</b></summary>
+
+| pattern | required? | description |
+| --- | --- | --- |
+| <code>extras\/.*</code> | ✓ | Folder for general lab-specific files related to the dataset. [Exists in all assays] |
+| <code>extras\/microscope_hardware\.json</code> | ✓ | **[QA/QC]** A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
+| <code>extras\/microscope_settings\.json</code> |  | **[QA/QC]** A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
+| <code>raw\/.*</code> | ✓ | This is a directory containing raw data. |
+| <code>raw\/images\/.*</code> | ✓ | Raw image files. Using this subdirectory allows for harmonization with other more complex assays, like Visium that includes both raw imaging and sequencing data. |
+| <code>raw\/images\/[^\/]+\.ome\.tiff</code> |  | Raw image file. |
+| <code>raw\/images\/tiles\.csv</code> |  | This file contains the approximate coordinates for each of the tiled raw images. |
+| <code>lab_processed\/.*</code> | ✓ | Experiment files that were processed by the lab generating the data. |
+| <code>lab_processed\/images\/.*</code> | ✓ | This is a directory containing processed image files |
+| <code>lab_processed\/images\/[^\/]+\.ome\.tiff</code> | ✓ | OME-TIFF file (multichannel, multi-layered) produced by the experiment. If compressed, must use loss-less compression algorithm. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0> |
+| <code>lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv</code> | ✓ | This file provides essential documentation pertaining to each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. The required fields are detailed <https://docs.google.com/spreadsheets/d/1xEJSb0xn5C5fB3k62pj1CyHNybpt4-YtvUs5SUMS44o/edit#gid=0> |
+
+<summary><b>Version 2.0</b></summary>
 
 | pattern | required? | description |
 | --- | --- | --- |

diff --git a/docs/visium-no-probes/current/index.md b/docs/visium-no-probes/current/index.md
@@ -30,7 +30,29 @@ REQUIRED - For this assay, you must also prepare and submit two additional metad
 <br>
 
 ## Directory schemas
-<summary><b>Version 2.1 (use this one)</b></summary>
+<summary><b>Version 2.2 (use this one)</b></summary>
+
+| pattern | required? | description |
+| --- | --- | --- |
+| <code>extras\/.*</code> | ✓ | Folder for general lab-specific files related to the dataset |
+| <code>extras\/microscope_hardware\.json</code> | ✓ | **[QA/QC]** A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
+| <code>extras\/microscope_settings\.json</code> |  | **[QA/QC]** A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
+| <code>raw\/.*</code> | ✓ | All raw data files for the experiment. |
+| <code>raw\/[^\/]+\.gpr</code> | ✓ | This is a 10X Genomics layout file that's generated by 10X and individualized for each Visium slide. This is a text file and can be generated using this 10X web form <https://support.10xgenomics.com/spatial-gene-expression/software/pipelines/latest/using/slidefile-download> along with the unique 10X Visium slide ID. |
+| <code>raw\/fastq\/.*</code> | ✓ | Raw sequencing files for the experiment |
+| <code>raw\/fastq\/RNA\/.*</code> | ✓ | Directory containing fastq files pertaining to RNAseq sequencing. |
+| <code>raw\/fastq\/RNA\/[^\/]+_R[^\/]+\.fastq\.gz</code> | ✓ | This is a GZip'd version of the forward and reverse fastq files from RNAseq sequencing (R1 and R2). |
+| <code>raw\/images\/.*</code> | ✓ | Directory containing raw image files. This directory should include at least one raw file. |
+| <code>raw\/images\/[^\/]+\.(?:xml&#124;scn&#124;vsi&#124;svs&#124;czi&#124;tiff)</code> | ✓ | Raw microscope file for the experiment |
+| <code>lab_processed\/.*</code> | ✓ | Experiment files that were processed by the lab generating the data. |
+| <code>lab_processed\/images\/.*</code> | ✓ | Processed image files |
+| <code>lab_processed\/images\/[^\/]+\.ome\.tiff</code> (example: <code>lab_processed/images/HBM892.MDXS.293.ome.tiff</code>) | ✓ | OME-TIFF files (multichannel, multi-layered) produced by the microscopy experiment. If compressed, must use loss-less compression algorithm. For Visium this stitched file should only include the single capture area relevant to the current dataset. For GeoMx there will be one OME TIFF file per slide, with each slide including multiple AOIs. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0> |
+| <code>lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv</code> | ✓ | This file provides essential documentation pertaining to each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. The required fields are detailed <https://docs.google.com/spreadsheets/d/1xEJSb0xn5C5fB3k62pj1CyHNybpt4-YtvUs5SUMS44o/edit#gid=0> |
+| <code>lab_processed\/images\/[^\/]+\.json</code> |  | This file is the output from LoupeBrowser, when a data provider manually denotes which spots on the slide contain tissue. This file is optionally used by 10X SpaceRanger. |
+| <code>lab_processed\/transformations\/.*</code> |  | This directory contains transformation matrices that capture how each modality is aligned with the other and can be used to visualize overlays of multimodal data. This is needed to overlay images from the exact same tissue section (e.g., MALDI imaging mass spec, autofluorescence microscopy, MxIF, histological stains). In these cases data type may have different pixel sizes and slightly different orientations (i.e., one may be rotated relative to another). |
+| <code>lab_processed\/transformations\/[^\/]+\.txt</code> |  | Transformation matrices used to overlay images from the exact same tissue section (e.g., MALDI imaging mass spec, autofluorescence microscopy, MxIF, histological stains). |
+
+<summary><b>Version 2.1</b></summary>
 
 | pattern | required? | description | dependent on |
 | --- | --- | --- | --- |

diff --git a/src/ingest_validation_tools/directory-schemas/mibi-v2.1.yaml b/src/ingest_validation_tools/directory-schemas/mibi-v2.1.yaml
@@ -0,0 +1,48 @@
+files:
+  -
+    pattern: extras\/.*
+    required: True
+    description: Folder for general lab-specific files related to the dataset. [Exists in all assays]
+  -
+    pattern: extras\/microscope_hardware\.json
+    required: True
+    description: A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document.
+    is_qa_qc: True
+  -
+    pattern: extras\/microscope_settings\.json
+    required: False
+    description: A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document.
+    is_qa_qc: True
+  -  
+    pattern: raw\/.*
+    required: True
+    description: This is a directory containing raw data. 
+  -  
+    pattern: raw\/images\/.*
+    required: True
+    description: Raw image files. Using this subdirectory allows for harmonization with other more complex assays, like Visium that includes both raw imaging and sequencing data.
+  -
+    pattern: raw\/images\/[^\/]+\.ome\.tiff
+    required: False
+    description: Raw image file.
+  -
+    pattern: raw\/images\/tiles\.csv
+    required: False
+    description: This file contains the approximate coordinates for each of the tiled raw images.
+  -
+    pattern: lab_processed\/.*
+    required: True
+    description: Experiment files that were processed by the lab generating the data.
+  -
+    pattern: lab_processed\/images\/.*
+    required: True
+    description: This is a directory containing processed image files
+  -
+    pattern: lab_processed\/images\/[^\/]+\.ome\.tiff
+    required: True
+    description: OME-TIFF file (multichannel, multi-layered) produced by the experiment. If compressed, must use loss-less compression algorithm. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0>
+    is_qa_qc: False
+  -
+    pattern: lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv
+    required: True
+    description: This file provides essential documentation pertaining to each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. The required fields are detailed <https://docs.google.com/spreadsheets/d/1xEJSb0xn5C5fB3k62pj1CyHNybpt4-YtvUs5SUMS44o/edit#gid=0>
diff --git a/src/ingest_validation_tools/directory-schemas/visium-no-probes-v2.2.yaml b/src/ingest_validation_tools/directory-schemas/visium-no-probes-v2.2.yaml
@@ -0,0 +1,78 @@
+files:
+  -
+    pattern: extras\/.*
+    required: True
+    description: Folder for general lab-specific files related to the dataset
+  -
+    pattern: extras\/microscope_hardware\.json
+    required: True
+    description: A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document.
+    is_qa_qc: True
+  -
+    pattern: extras\/microscope_settings\.json
+    required: False
+    description: A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document.
+    is_qa_qc: True
+  -
+    pattern: raw\/.*
+    required: True
+    description: All raw data files for the experiment.
+  -
+    pattern: raw\/[^\/]+\.gpr
+    required: True
+    description: This is a 10X Genomics layout file that's generated by 10X and individualized for each Visium slide. This is a text file and can be generated using this 10X web form <https://support.10xgenomics.com/spatial-gene-expression/software/pipelines/latest/using/slidefile-download> along with the unique 10X Visium slide ID.
+    is_qa_qc: False
+  -
+    pattern: raw\/fastq\/.*
+    required: True
+    description: Raw sequencing files for the experiment
+  -  
+    pattern: raw\/fastq\/RNA\/.*
+    required: True
+    description: Directory containing fastq files pertaining to RNAseq sequencing.
+  -
+    pattern: raw\/fastq\/RNA\/[^\/]+_R[^\/]+\.fastq\.gz
+    required: True
+    description: This is a GZip'd version of the forward and reverse fastq files from RNAseq sequencing (R1 and R2).
+    is_qa_qc: False
+  -
+    pattern: raw\/images\/.*
+    required: True
+    description: Directory containing raw image files. This directory should include at least one raw file.
+  -
+    pattern: raw\/images\/[^\/]+\.(?:xml|scn|vsi|svs|czi|tiff)
+    required: True
+    description: Raw microscope file for the experiment
+    is_qa_qc: False
+  -
+    pattern: lab_processed\/.*
+    required: True
+    description: Experiment files that were processed by the lab generating the data.
+  -
+    pattern: lab_processed\/images\/.*
+    required: True
+    description: Processed image files
+  -
+    pattern: lab_processed\/images\/[^\/]+\.ome\.tiff
+    required: True
+    description: OME-TIFF files (multichannel, multi-layered) produced by the microscopy experiment. If compressed, must use loss-less compression algorithm. For Visium this stitched file should only include the single capture area relevant to the current dataset. For GeoMx there will be one OME TIFF file per slide, with each slide including multiple AOIs. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0>
+    is_qa_qc: False
+    example: lab_processed/images/HBM892.MDXS.293.ome.tiff
+  -
+    pattern: lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv
+    required: True
+    description: This file provides essential documentation pertaining to each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. The required fields are detailed <https://docs.google.com/spreadsheets/d/1xEJSb0xn5C5fB3k62pj1CyHNybpt4-YtvUs5SUMS44o/edit#gid=0>
+    is_qa_qc: False
+  -
+    pattern: lab_processed\/images\/[^\/]+\.json
+    required: False
+    description: This file is the output from LoupeBrowser, when a data provider manually denotes which spots on the slide contain tissue. This file is optionally used by 10X SpaceRanger.
+  -
+    pattern: lab_processed\/transformations\/.*
+    required: False
+    description: This directory contains transformation matrices that capture how each modality is aligned with the other and can be used to visualize overlays of multimodal data. This is needed to overlay images from the exact same tissue section (e.g., MALDI imaging mass spec, autofluorescence microscopy, MxIF, histological stains). In these cases data type may have different pixel sizes and slightly different orientations (i.e., one may be rotated relative to another).
+  -
+    pattern: lab_processed\/transformations\/[^\/]+\.txt
+    required: False
+    description: Transformation matrices used to overlay images from the exact same tissue section (e.g., MALDI imaging mass spec, autofluorescence microscopy, MxIF, histological stains). 
+    is_qa_qc: False