Draft a pipeline recipe specification

[ambrosejcarr]

The starfish specification indicates that starfish will accept a JSON pipeline recipe, which it will translate into a series of CLI calls.

From the spaceTx Specification:

[The recipe] represents the chosen sequence of pipeline modules [...] and the parameters for each module. The sequence of pipeline modules can be expressed as a JSON file (ideally as calls to a command line tool, see Starfish for a preliminary example) and the parameters can also be stored as a JSON file (see Starfish for example input parameters).

This recipe will need to solve both multi-fov processing and the processing of different stack types (nuclei, hybridization)

• draft the format of this JSON recipe

Depends on #292 , #311

[ambrosejcarr]

@berl this issue should address the vagueness in the spaceTx specification of the pipeline recipe. Anything I missed in the description?

[berl]

looks good. I think the 0th version of the multi-FOV processing can treat every FOV equivalently and process all of the FOVs in the manifest. In the future Simone and I both would like to see incremental processing whereby (e.g.) rounds 1-3 can be uploaded and processed, then round 4 appended and processed without re-running rounds 1-3. I've added this as an issue- likely for the next project!

[ttung]

Proposal

A significantly simplified snippet of python code will represent the pipeline recipe. There is one method, compute that will represent each pipeline stage. First argument is the pipeline component type, second argument is the algorithm. Argument 3..N are arguments passed to the algorithm's run method. Keyword arguments are for tuning parameters.

There are two magic variables: file_input and file_output. file_input is an array consisting of all the input files passed into starfish along with the recipe. At the end of the run, anything written to the file_output array will be written out to disk.

Example invocation: starfish run my_recipe_file -i input_0 -i input_1 -o output_0 -o output_1 -o output_2

Example recipe file:

filtered = compute("filter", "WhiteTophat", file_input[0], masking_radius=15, is_volume=False)
file_output[0] = compute("detect_spots", "BlobDetector", filtered, min_sigma=1, max_sigma=2, num_sigma=1, threshold=1, overlap=1)

[ttung]

I've also thought about making the input and output files be able to start from an experiment.json file. Something like:

starfish filter WhiteTophat -i @experiment.json[fov_001][primary]

Thoughts?

[ambrosejcarr]

@ttung two questions:

How does starfish know what input is of what type, and what does that imply about how it serializes or deserializes objects?

What object types are valid?

[ttung]

Can I say magic? :)

Ok actually it’s inspecting the method signature to figure it out.

It supports ImageStack, IntensityTable, ExpressionMatrix, and Codebook

[ttung]

Implementation in #311