Merge pull request #876 from fractal-analytics-platform/874_updated_t…

…asks_info

Add tasks docs_info

.github/workflows/ci_poetry.yml

@@ -87,8 +87,17 @@ jobs:
       - name: Install dependencies (including fractal-tasks extra)
         run: poetry install --with dev --without docs --no-interaction -E fractal-tasks
 
-      - name: Check manifest task metadata
-        run: poetry run python fractal_tasks_core/dev/check_manifest.py
+      - name: Check if manifest has changed
+        run: |
+          poetry run python fractal_tasks_core/dev/create_manifest.py
+          echo "*.json diff=json" >> .gitattributes && git config diff.json.textconv "jq --sort-keys '.' \$1"
+          git diff ./fractal_tasks_core/__FRACTAL_MANIFEST__.json
+          if [ -n "$(git diff --exit-code ./fractal_tasks_core/__FRACTAL_MANIFEST__.json)" ]; then
+          echo "__FRACTAL_MANIFEST__.json has changed. Please run 'poetry run python fractal_tasks_core/dev/create_manifest.py' and commit the changes."
+            exit 1
+          else
+            echo "__FRACTAL_MANIFEST__.json has not changed."
+          fi
 
       - name: Cache Pooch folder
         id: cache-pooch-folder

CHANGELOG.md

@@ -1,5 +1,11 @@
 **Note**: Numbers like (\#123) point to closed Pull Requests on the fractal-tasks-core repository.
 
+# 1.3.4
+
+* Manifest creation:
+    * Support providing `docs_info=file:task_info/description.md` (\#876).
+    * Deprecate `check_manifest.py` module, in favor of additional GitHub action steps (\#876).
+
 # 1.3.3
 
 * Add new metadata (authors, category, modality, tags) to manifest models and to tasks (\#855).

fractal_tasks_core/dev/create_manifest.py

@@ -22,6 +22,7 @@
     create_schema_for_single_task,
 )
 from fractal_tasks_core.dev.lib_task_docs import create_docs_info
+from fractal_tasks_core.dev.lib_task_docs import read_docs_info_from_file
 
 
 logging.basicConfig(level=logging.INFO)
@@ -126,11 +127,19 @@ def create_manifest(
                     task_dict[f"args_schema_{kind}"] = schema
 
         # Update docs_info, based on task-function description
-        docs_info = create_docs_info(
-            executable_non_parallel=task_obj.executable_non_parallel,
-            executable_parallel=task_obj.executable_parallel,
-            package=package,
-        )
+        docs_info = task_dict.get("docs_info")
+        if docs_info is None:
+            docs_info = create_docs_info(
+                executable_non_parallel=task_obj.executable_non_parallel,
+                executable_parallel=task_obj.executable_parallel,
+                package=package,
+            )
+        elif docs_info.startswith("file:"):
+            docs_info = read_docs_info_from_file(
+                docs_info=docs_info,
+                task_list_path=task_list_module.__file__,
+            )
+
         if docs_info is not None:
             task_dict["docs_info"] = docs_info
         if docs_link is not None:

fractal_tasks_core/dev/lib_task_docs.py

@@ -57,7 +57,7 @@ def create_docs_info(
     executable_non_parallel: Optional[str] = None,
     executable_parallel: Optional[str] = None,
     package: str = "fractal_tasks_core",
-) -> list[str]:
+) -> str:
     """
     Return task description based on function docstring.
     """
@@ -81,3 +81,38 @@ def create_docs_info(
     docs_info = "".join(docs_info)
     logging.info("[create_docs_info] END")
     return docs_info
+
+
+def read_docs_info_from_file(
+    *,
+    docs_info: str,
+    task_list_path: str,
+) -> str:
+    """
+    Return task description based on the content of a file.
+
+    An example of valid argument is
+    ```
+    docs_info = "file:relative/path/info.md"
+    ```
+    where the path is relative to the folder where `task_list.py` is.
+    """
+    logging.info("[read_docs_info_from_file] START")
+
+    # Preliminary checks
+    if not docs_info.startswith("file:"):
+        raise ValueError(f"Invalid docs_info='{docs_info}'.")
+    relative_path = Path(docs_info[5:])
+    if relative_path.is_absolute():
+        raise ValueError(
+            f"Invalid docs_info='{docs_info}' (path must be relative)."
+        )
+
+    base_path = Path(task_list_path).parent
+    docs_path = (base_path / relative_path).as_posix()
+    logging.info(f"[read_docs_info_from_file] Reading docs from {docs_path}")
+    with open(docs_path, "r") as f:
+        docs_info = f.read()
+    logging.info("[read_docs_info_from_file] END")
+
+    return docs_info

fractal_tasks_core/dev/task_info/apply_registration_to_image.md

@@ -0,0 +1,8 @@
+### Purpose
+- **Applies pre-calculated registration** transformations to images in an **HCS** OME-Zarr dataset, aligning all acquisitions to a specified reference acquisition.
+- **Masks regions not included** in the registered ROI table and aligns both intensity and label images.
+- Replaces the non-aligned image with the newly aligned image in the dataset if `overwrite input` is selected.
+- Typically used as the third task in a workflow, following `Calculate Registration (image-based)` and `Find Registration Consensus`.
+
+### Limitations
+- If `overwrite input` is selected, the non-aligned image is permanently deleted, which may impact workflows requiring access to the original images.

fractal_tasks_core/dev/task_info/calculate_registration_image_based.md

@@ -0,0 +1,9 @@
+### Purpose
+- **Computes image-based registration** transformations for acquisitions in **HCS** OME-Zarr datasets.
+- Processes images grouped by well, under the assumption that each well contains one image per acquisition.
+- Calculates transformations for **specified regions of interest (ROIs)** and stores the results in the corresponding ROI table.
+- Typically used as the first task in a workflow, followed by `Find Registration Consensus` and optionally `Apply Registration to Image`.
+
+### Limitations
+- Supports only HCS OME-Zarr datasets, leveraging their acquisition metadata and well-based image grouping.
+- Assumes each well contains a single image per acquisition.

fractal_tasks_core/dev/task_info/cellpose_segmentation.md

@@ -0,0 +1,10 @@
+### Purpose
+- **Segments images using Cellpose models**.
+- Supports both **built-in Cellpose models** (shipped with Cellpose) and **user-trained models**.
+- Accepts dual image input for segmentation.
+- Can process **arbitrary regions of interest (ROIs)**, including whole images, fields of view (FOVs), or masked outputs from prior segmentations, based on corresponding ROI tables.
+- Provides access to all advanced Cellpose parameters.
+- Allows custom rescaling options per channel, particularly useful for sparse images.
+
+### Limitations
+- Compatible only with Cellpose 2.x models; does not yet support 3.x models.

fractal_tasks_core/dev/task_info/convert_cellvoyager_multiplex.md

@@ -0,0 +1,8 @@
+### Purpose
+- Converts **multiplexed 2D and 3D images from CellVoyager CV7000/8000** systems into OME-Zarr format, storing each acquisition as a separate OME-Zarr image in the same OME-Zarr plate.
+- Creates **OME-Zarr HCS plates**, combining all fields of view for each acquisition in a well into a single image.
+- Saves Fractal **region-of-interest (ROI) tables** for both individual fields of view and the entire well.
+- Handles overlapping fields of view by adjusting their positions to be non-overlapping, while preserving the original position data as additional columns in the ROI tables.
+
+### Limitations
+- This task currently does not support time-resolved data and ignores the time fields in CellVoyager metadata.

fractal_tasks_core/dev/task_info/convert_cellvoyager_to_ome_zarr.md

@@ -0,0 +1,8 @@
+### Purpose
+- Converts **2D and 3D images from CellVoyager CV7000/8000** systems into OME-Zarr format, creating OME-Zarr HCS plates and combining all fields of view in a well into a single image.
+- Saves Fractal **region-of-interest (ROI) tables** for both individual fields of view and the entire well.
+- Handles overlapping fields of view by adjusting their positions to be non-overlapping while retaining the original position data as additional columns in the ROI tables.
+- Supports processing multiple plates in a single task.
+
+### Limitations
+- Currently, this task does not support time-resolved data and ignores the time fields in CellVoyager metadata.

fractal_tasks_core/dev/task_info/find_registration_consensus.md

@@ -0,0 +1,7 @@
+### Purpose
+- Determines the **consensus alignment** region across all selected acquisitions **within each well of an HCS OME-Zarr dataset**.
+- Generates a new ROI table for each image, defining consensus regions that are aligned across all acquisitions.
+- Typically used as the second task in a workflow, following `Calculate Registration (image-based)` and optionally preceding `Apply Registration to Image`.
+
+### Limitations
+- Supports only HCS OME-Zarr datasets, leveraging their acquisition metadata and well-based image grouping.

fractal_tasks_core/dev/task_info/illumination_correction.md

@@ -0,0 +1,7 @@
+### Purpose
+- **Corrects illumination** in OME-Zarr images using **pre-calculated flatfield profiles**.
+- Optionally performs **background subtraction**.
+
+### Limitations
+- Requires pre-calculated flatfield profiles in TIFF format.
+- Supports only fixed-value background subtraction; background subtraction profiles are not supported.

fractal_tasks_core/dev/task_info/import_ome_zarr.md

@@ -0,0 +1,10 @@
+### Purpose
+- Imports a **single OME-Zarr dataset** into the Fractal framework for further processing.
+- Supports importing either a **full OME-Zarr HCS plate** or an **individual OME-Zarr image**.
+- Ensures the OME-Zarr dataset is located in the `zarr_dir` specified by the dataset.
+- Generates the necessary **image list metadata** required for processing the OME-Zarr with Fractal.
+- Optionally **adds new ROI tables** to the existing OME-Zarr, enabling compatibility with many other tasks.
+
+### Limitations
+- Supports only OME-Zarr datasets already present in the `zarr_dir` of the corresponding dataset.
+- Assumes the input OME-Zarr is correctly structured and formatted for compatibility with the Fractal framework.

fractal_tasks_core/dev/task_info/napari_workflows_wrapper.md

@@ -0,0 +1,26 @@
+### Purpose
+- Executes a **napari workflow** on the regions of interest (ROIs) within a single OME-NGFF image.
+- Processes specified images and labels as inputs to the workflow, producing outputs such as new labels and data tables.
+- Offers **flexibility in defining input and output** specifications to customize the workflow for specific datasets and analysis needs.
+
+### Limitations
+- Currently supports only Napari workflows that utilize functions from the `napari-segment-blobs-and-things-with-membranes` module. Other Napari-compatible modules are not supported.
+
+### Input Specifications
+Napari workflows require explicit definitions of input and output data.
+Example of valid `input_specs`:
+```json
+{
+    "in_1": {"type": "image", "channel": {"wavelength_id": "A01_C02"}},
+    "in_2": {"type": "image", "channel": {"label": "DAPI"}},
+    "in_3": {"type": "label", "label_name": "label_DAPI"}
+}
+```
+
+Example of valid `output_specs`:
+```json
+{
+    "out_1": {"type": "label", "label_name": "label_DAPI_new"},
+    "out_2": {"type": "dataframe", "table_name": "measurements"},
+}
+```

fractal_tasks_core/dev/task_info/projection.md

@@ -0,0 +1,7 @@
+### Purpose
+- Performs **Z-axis projection of intensity images** using a specified projection method.
+- **Generates a new OME-Zarr HCS plate** to store the projected data.
+
+### Limitations
+- Supports projections only for OME-Zarr HCS plates; other collections of OME-Zarrs are not yet supported.
+- Currently limited to data in the CZYX format.