hubmapconsortium · jpuerto-psc · Nov 12, 2024 · Nov 11, 2024 · Nov 11, 2024 · Nov 12, 2024
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,6 @@
 # Changelog
 ## v0.0.29 (in progress)
+- Add CosMX directory schema
 
 ## v0.0.28
 - Update Xenium directory schema

diff --git a/docs/cosmx/current/index.md b/docs/cosmx/current/index.md
@@ -28,5 +28,30 @@ Related files:
 <br>
 
 ## Directory schemas
-<summary><a href="https://docs.google.com/spreadsheets/d/1LE-iyY2E6eP4E8jhgP6rhsvjESrdHXWYrMwKTvNkI5Y"><b>Version 2.0 (use this one)</b> (draft - submission of data prepared using this schema will be supported by Sept. 30) </a></summary>
+<summary><b>Version 2.0 (use this one)</b></summary>
+
+| pattern | required? | description |
+| --- | --- | --- |
+| <code>extras\/.*</code> | ✓ | Folder for general lab-specific files related to the dataset. |
+| <code>extras\/microscope_hardware\.json</code> | ✓ | **[QA/QC]** A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
+| <code>extras\/microscope_settings\.json</code> |  | **[QA/QC]** A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
+| <code>raw\/.*</code> | ✓ | All raw data files for the experiment. |
+| <code>raw\/data\/[^\/]*_exprMat_file\.csv</code> | ✓ | Cell expression matrix with raw counts of genes for each identified cell. Apart from the gene specific columns, additional columns include: the origin of the cell (fov, unique cell id); negative probes; probes associated with the fiducial frame (SystemControl) |
+| <code>raw\/data\/[^\/]*_fov_positions_file\.csv</code> | ✓ | FOV Positions file that provides an overview of the tissue locations and to help specify separate regions and/or tissues on the slide. This contains information about the Slide - slide number it comes from; FOV - field of view; X_mm/Y_mm - x/y coordinates of FOV positions in mm (previously in px) |
+| <code>raw\/data\/[^\/]*_metadata_file\.csv</code> | ✓ | Cell metadata file containing the following information - the origin of the cell (fov, unique cell id); physical properties of the cell (area, aspect ratio, width, height); location of the cell centroid within each FOV (center X/Y local) and global position (center X/Y global); information about the protein staining (min/max intensity); type of protein, which may be specific to each experiment but generally includes DAPI, Membrane, PanCK, CD45; other (e.g. Seurat information if that pipeline was used within AtoMx, some data quality information) |
+| <code>raw\/data\/[^\/]*_tx_file\.csv</code> | ✓ | Cell transcript file |
+| <code>raw\/data\/[^\/]*_config\.ini</code> | ✓ | Needed to generate the DCC file from the fastq file. Contains pipeline processing parameters.  Generated by DSP run, prior to sequencing. |
+| <code>raw\/markers\.csv</code> | ✓ | A csv file describing any morphology markers used to guide ROI and/or AOI selection. |
+| <code>raw\/additional_panels_used\.csv</code> |  | If multiple commercial probe panels were used, then the primary probe panel should be selected in the "oligo_probe_panel" metadata field. The additional panels must be included in this file. Each panel record should include: manufacturer, model/name, product code. |
+| <code>raw\/gene_panel\.csv</code> | ✓ | The list of target genes. The expected format is: gene_id (ensembl ID), gene_name. |
+| <code>raw\/custom_probe_set\.csv</code> |  | This file should contain any custom probes used and must be included if the metadata field "is_custom_probes_used" is "Yes". The file should minimally include: target gene id, probe seq, probe id. The contents of this file are modeled after the 10x Genomics probe set file (see <https://support.10xgenomics.com/spatial-gene-expression-ffpe/probe-sets/probe-set-file-descriptions/probe-set-file-descriptions#probe_set_csv_file>). |
+| <code>raw\/transcript_locations\.csv</code> | ✓ | The origin of the coordinate is 0,0 at the top left corner of the image. The file should include: gene name, x, y, z (optional), quality score (optional). It is expected that the first row in the file contains the column header. |
+| <code>raw\/custom_gene_list\.csv</code> |  | This describes the target genes profiled by the assay. For advanced design, this can be probes sequences for splicing or other analysis for any target of interest. The format should minimally contain: gene name, ensemble ID |
+| <code>raw\/probes\.csv</code> |  | A CSV file describing the probe panel used. This is typically what's used to specific the probe set when ordering a probe panel for a Xenium run. |
+| <code>raw\/images\/overlay\.(?:jpeg&#124;tiff)</code> |  | State whether an overlay image was used to guide ROI selection. If an overlay is used, then the overlay details will be provided in the protocols.io protocol. If used, this needs to be uploaded. It is not included in the OME TIFF. This can be a JPEG or TIFF file |
+| <code>raw\/images\/preview_scan\.png</code> | ✓ | Assists in selection of regions of FOVs using the grid FOV placement tool. |
+| <code>lab_processed\/.*</code> | ✓ | Experiment files that were processed by the lab generating the data. |
+| <code>lab_processed\/images\/.*</code> | ✓ | Processed image files |
+| <code>lab_processed\/images\/[^\/]+\.ome\.tiff</code> (example: <code>lab_processed/images/HBM892.MDXS.293.ome.tiff</code>) | ✓ | OME-TIFF files (multichannel, multi-layered) produced by the microscopy experiment. If compressed, must use loss-less compression algorithm. For Visium this stitched file should only include the single capture area relevant to the current dataset. For GeoMx there will be one OME TIFF file per slide, with each slide including multiple AOIs. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0> |
+| <code>lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv</code> | ✓ | This file provides essential documentation pertaining to each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. The required fields are detailed <https://docs.google.com/spreadsheets/d/1xEJSb0xn5C5fB3k62pj1CyHNybpt4-YtvUs5SUMS44o/edit#gid=0> |
 
@@ -1,4 +1,94 @@
-draft: true
 files:
   -
-    draft_link: 'https://docs.google.com/spreadsheets/d/1LE-iyY2E6eP4E8jhgP6rhsvjESrdHXWYrMwKTvNkI5Y'
+    pattern: extras\/.*
+    required: True
+    description: Folder for general lab-specific files related to the dataset.
+  -
+    pattern: extras\/microscope_hardware\.json
+    required: True
+    description: A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document.
+    is_qa_qc: True
+  -
+    pattern: extras\/microscope_settings\.json
+    required: False
+    description: A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document.
+    is_qa_qc: True
+  -
+    pattern: raw\/.*
+    required: True
+    description: All raw data files for the experiment.
+  -
+    pattern: raw\/data\/[^\/]*_exprMat_file\.csv
+    required: True
+    description: "Cell expression matrix with raw counts of genes for each identified cell. Apart from the gene specific columns, additional columns include: the origin of the cell (fov, unique cell id); negative probes; probes associated with the fiducial frame (SystemControl)"
+  -
+    pattern: raw\/data\/[^\/]*_fov_positions_file\.csv
+    required: True
+    description: FOV Positions file that provides an overview of the tissue locations and to help specify separate regions and/or tissues on the slide. This contains information about the Slide - slide number it comes from; FOV - field of view; X_mm/Y_mm - x/y coordinates of FOV positions in mm (previously in px)
+  -
+    pattern: raw\/data\/[^\/]*_metadata_file\.csv
+    required: True
+    description: Cell metadata file containing the following information - the origin of the cell (fov, unique cell id); physical properties of the cell (area, aspect ratio, width, height); location of the cell centroid within each FOV (center X/Y local) and global position (center X/Y global); information about the protein staining (min/max intensity); type of protein, which may be specific to each experiment but generally includes DAPI, Membrane, PanCK, CD45; other (e.g. Seurat information if that pipeline was used within AtoMx, some data quality information)
+  -
+    pattern: raw\/data\/[^\/]*_tx_file\.csv
+    required: True
+    description: Cell transcript file
+  -
+    pattern: raw\/data\/[^\/]*_config\.ini
+    required: True
+    description: Needed to generate the DCC file from the fastq file. Contains pipeline processing parameters.  Generated by DSP run, prior to sequencing.
+  -
+    pattern: raw\/markers\.csv
+    required: True
+    description: A csv file describing any morphology markers used to guide ROI and/or AOI selection.
+  -
+    pattern: raw\/additional_panels_used\.csv
+    required: False
+    description: 'If multiple commercial probe panels were used, then the primary probe panel should be selected in the "oligo_probe_panel" metadata field. The additional panels must be included in this file. Each panel record should include: manufacturer, model/name, product code.'
+  -
+    pattern: raw\/gene_panel\.csv
+    required: True
+    description: 'The list of target genes. The expected format is: gene_id (ensembl ID), gene_name.'
+  -
+    pattern: raw\/custom_probe_set\.csv
+    required: False
+    description: 'This file should contain any custom probes used and must be included if the metadata field "is_custom_probes_used" is "Yes". The file should minimally include: target gene id, probe seq, probe id. The contents of this file are modeled after the 10x Genomics probe set file (see <https://support.10xgenomics.com/spatial-gene-expression-ffpe/probe-sets/probe-set-file-descriptions/probe-set-file-descriptions#probe_set_csv_file>).'
+  -
+    pattern: raw\/transcript_locations\.csv
+    required: True
+    description: 'The origin of the coordinate is 0,0 at the top left corner of the image. The file should include: gene name, x, y, z (optional), quality score (optional). It is expected that the first row in the file contains the column header.'
+  -
+    pattern: raw\/custom_gene_list\.csv
+    required: False
+    description: 'This describes the target genes profiled by the assay. For advanced design, this can be probes sequences for splicing or other analysis for any target of interest. The format should minimally contain: gene name, ensemble ID'
+  -
+    pattern: raw\/probes\.csv
+    required: False
+    description: A CSV file describing the probe panel used. This is typically what's used to specific the probe set when ordering a probe panel for a Xenium run.
+  -
+    pattern: raw\/images\/overlay\.(?:jpeg|tiff)
+    required: False
+    description: State whether an overlay image was used to guide ROI selection. If an overlay is used, then the overlay details will be provided in the protocols.io protocol. If used, this needs to be uploaded. It is not included in the OME TIFF. This can be a JPEG or TIFF file
+  -
+    pattern: raw\/images\/preview_scan\.png
+    required: True
+    description: Assists in selection of regions of FOVs using the grid FOV placement tool.
+  -
+    pattern: lab_processed\/.*
+    required: True
+    description: Experiment files that were processed by the lab generating the data.
+  -
+    pattern: lab_processed\/images\/.*
+    required: True
+    description: Processed image files
+  -
+    pattern: lab_processed\/images\/[^\/]+\.ome\.tiff
+    required: True
+    description: OME-TIFF files (multichannel, multi-layered) produced by the microscopy experiment. If compressed, must use loss-less compression algorithm. For Visium this stitched file should only include the single capture area relevant to the current dataset. For GeoMx there will be one OME TIFF file per slide, with each slide including multiple AOIs. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0>
+    is_qa_qc: False
+    example: lab_processed/images/HBM892.MDXS.293.ome.tiff
+  -
+    pattern: lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv
+    required: True
+    description: This file provides essential documentation pertaining to each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. The required fields are detailed <https://docs.google.com/spreadsheets/d/1xEJSb0xn5C5fB3k62pj1CyHNybpt4-YtvUs5SUMS44o/edit#gid=0>
+    is_qa_qc: False