New MM::StorageDevice and addition to the MMCore API #535
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…he system level to a different interface number
MMCore solution filter updated
# Conflicts: # DeviceAdapters/ArduinoCounter/ArduinoCounter.vcxproj # micromanager.sln
# Conflicts: # micromanager.sln
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
Introducing the new MM::StorageDevice. There are two main changes to the micro-manger back end: a new device type (Storage) and the extension of the MMCoreAPI. Both changes are lateral because they extend capabilities without affecting existing functionality. There are no new dependencies for the code base. Any existing client code, including the Micro-manager front end, can safely ignore this extension.
The new feature enables MMCore to save datasets on the back end using a storage device adapter. Developers can add support for various formats by creating device adapters like they would add support for a new camera. There are two significant advantages in utilizing this architecture:
The most important one is that the interface between the microscopy application and the file storage is defined by the MMCore API and allows for designing generic, storage/format-independent acquisition engines and acquisition scripts. To change the file format, we only change the device configuration, but the acquisition code remains the same. Or mainly the same, except for some specific properties - pretty much as what happens when we change the camera. Adding support for new file formats or using better implementations does not involve changing the existing code in any significant way.
The second advantage is that by embedding the storage implementation in the MMCore, we can achieve a higher efficiency (data rate), as images do not have to cross the MMCore API boundary.
Status
We plan on adding more documentation and some examples over the coming days and weeks. This PR is work in progress and only a draft to get some early feedback. Implementation is currently functional, but incomplete. Also, the micro-manager build for the new version may not be completely compatible with the main branch. This will be corrected in the next update.
To go over the new API, you can see MMCore.h and MMDevice.h. There are currently two device adapters: BigTiff (with chunking) and Zarr (based on https://github.com/acquire-project/acquire-driver-zarr), both located in the "go2scope" device library. The bigTiff adapter is fully functional, while the Zarr currently supports only writing, not reading.
We are still working on integrating the now MMCore back-end storage into the Micro-manager application (Java). It works but is still not ready for the PR.
Credits
Nathan Clack and the Chan Zuckerberg Initiative team provided material, advisory, and moral support for this effort. Nico Stuurman and Talley Lambert provided early feedback and help. Milos Jovanovic built the fully functional and highly efficient BigTiff adapter based on the early specification of the interface.