You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It should be possible to fuse any connected subgraph of blockwise operations into a single blockwise operation.
For the single output case, it is possible to trace each block of the output to its (transitive) input blocks. The blockwise functions would be composed appropriately.
Handling memory requirements would need some care. It might be useful to break up the required memory for an operation into compressed_in, in, out, compressed_out (where the compressed parts are the buffers used by Zarr). Then it would be possible to drop the compressed parts when combining operations (since they no longer write to Zarr).
But subgraphs with large numbers of inputs might still hit memory limits, so it would not always be possible to combine the whole subgraph into one. This suggests that when checking for if fusion can take place, memory limits should be taken into account.
This could be generalised for multiple outputs (#69), but the two issues are somewhat independent.
The text was updated successfully, but these errors were encountered:
It should be possible to fuse any connected subgraph of blockwise operations into a single blockwise operation.
For the single output case, it is possible to trace each block of the output to its (transitive) input blocks. The blockwise functions would be composed appropriately.
Handling memory requirements would need some care. It might be useful to break up the required memory for an operation into compressed_in, in, out, compressed_out (where the compressed parts are the buffers used by Zarr). Then it would be possible to drop the compressed parts when combining operations (since they no longer write to Zarr).
But subgraphs with large numbers of inputs might still hit memory limits, so it would not always be possible to combine the whole subgraph into one. This suggests that when checking for if fusion can take place, memory limits should be taken into account.
This could be generalised for multiple outputs (#69), but the two issues are somewhat independent.
The text was updated successfully, but these errors were encountered: