Inplace conversion to Starfish format seems slow

[njmei]

A 2 round, 5 channel, 29 fov, 44 zplane mini-experiment takes about 30 minutes to convert to starfish format. This is likely to be problematic for our pipeline especially if we want to be converting + analyzing data on a per round basis as soon as it comes off the microscope.

It seems like the majority of time is still being taken up by write calls by slicedimage even for inplace conversion.

zipped cProfile output file:
starfish_conversion.prof.tar.gz

Visualization with snakeviz:

CC: @berl

[ttung]

What does format_survey_data.py do?

[njmei]

Basically it reads in our metadata format and organizes it so that starfish can query it to fetch tiles. Then it calls write_irregular_experiment_json. The source for it is:

def format_survey_data(input_manifest_path: Path, output_dir: Path,
                       aux_probes: set = {'DAPI'}):
    survey_manifest = SurveyManifest(manifest_path=input_manifest_path,
                                     aux_probes=aux_probes)

    aux_tfs = {aux_type: SurveyAuxTileFetcher(survey_manifest, aux_type)
               for aux_type in aux_probes}
    tile_fetchers = {'primary': SurveyPrimaryTileFetcher(survey_manifest),
                     **aux_tfs}

    inplace.enable_inplace_mode()

    write_irregular_experiment_json(
        path=str(output_dir.resolve()),
        tile_format=ImageFormat.TIFF,
        image_tile_coordinates=survey_manifest.expt_dims,
        tile_fetchers=tile_fetchers,
        postprocess_func=add_codebook,
        fov_path_generator=fov_path_generator,
        tile_opener=inplace.inplace_tile_opener
    )

[njmei]

It's totally possible that conversion is as fast as it can be. I'm just curious whether it could be faster. So I should probably change the subject name.

[ttung]

I verified that your tile fetchers don't need to return any sane data. That should shave 900+ seconds from your profile. As long as it returns correctly shaped data, the actual data is discarded.

I'm codifying this expectation into a test.

[njmei]

Oh, so instead of the tile_data FetchedTile method returning skimage.io.imread(path_of_tiff) (like it currently does) just have it return a dummy of some sort (e.g. np.empty())?

[ttung]

Yes, any np array of the correct shape and dtype should work.

[njmei]

Hmm... I wonder if that will give significant time savings. Even if you pass bogus data, it will still get sha256 hashed num_rounds * num_fovs * num_chs times.

[ttung]

I am somewhat skeptical that the extra sha computation will cost that much time. It's far more likely that the extra encoding into tiff costs way more time.

[ttung]

In any case, can you just give it a whirl? Should be an easy change. :)

[njmei]

Bizarrely, things got worse (swear I wasn't running or doing anything else!)?

Here's what I did:

    @functools.lru_cache(maxsize=1)
    def tile_data(self) -> np.ndarray:
        return np.zeros((self.tile.shape.y, self.tile.shape.x))

zipped cProfile output file:
starfish_conversion_2.prof.tar.gz

snakeviz:

format_survey_data.py:62 is just:

    @property
    def sha256(self) -> str:
        hasher = hashlib.sha256()
        with open(self.tile.full_filepath, 'rb') as fh:
            hasher.update(fh.read())
        return hasher.hexdigest()

[njmei]

@ttung Hey Tony, I had tried your suggestion, but it did not appear to make things any faster...

see: #1388 (comment)

[ttung]

Is this the same data set? I'm a bit suspicious since the imsave portion should be the same, but the new profile is 670s while the old profile is only 339s.

[njmei]

Yeah exact same dataset. Very odd, I know...