Conversion software

[Lestropie]

This is something that's been in my head from the outset, but given discussion around candidate proposals I think I need to list it here explicitly.

There is a definite risk of naivety in this project. While I've tried to contemplate the task of diffusion model representation in the most general case possible rather than simply enumerating a list of models in the form in which they are provided in respective softwares, there are almost certainly confounds that I have failed to identify. I have also failed to keep myself up to date with the evolution of BIDS & derivatives of such, so there may be emerging guidelines or conventions that I'm not aware of. I believe that if this BEP were to be presented to the community in its current state or close to it, it would be seen as purely academic, not sufficiently authoritative, and there would not be a strong motivation to adhere to it.

What I personally see as a prerequisite for this BEP to attract any wider interest is a software package for importing and exporting BIDS Diffusion model Derivatives. This would need to:

• Take diffusion model data as they are natively produced in various software packages, and convert them into BIDS Derivatives;
• Take BIDS Derivatives and convert them into the format expected by the native software package such that that software is agnostic to the fact that the data were not self-generated;
• Where possible, take data that were produced by one software, and convert it through the intermediate representation of BIDS Derivatives into the format expected by another software package, such that the second software reads data from the first package as though it was self-generated;
• Enable conversions within the BIDS Derivatives representation; e.g. conversion between SH conventions, changing of reference axes between scanner and image;
• Validate a BIDS Derivatives dataset and confirm that the requisite sidecar data to faciliate robust interpretation is present;
• Consider reasonably exhaustively the set of softwares / diffusion models in the community;
• (optional) Interface possible conversions between representations; e.g. define how one might get from a tensor representation to a 1-volume 3-vector representation based on the principal eigenvector.

Going through this process is likely to highlight issues with the current design. As such, if the extension were to be pushed first, and relevant software conversion tools were then developed after the fact, it would likely just undermine the extension as there could be a lot that would need to change in the specification in order to become functional. I would personally rather try to get the fundamental specification design correct and iron out the bugs and erroneous assumptions by actually attempting to use it before attempting to claim authority via the specification and demand that others use it.

This could be an interesting project for an e.g. computer science student to lead (I've not had success recruiting someone myself). It should also be public so that community members can propose contributions to support the softwares and models they themselves work with. I'd be happy to broadcast on the MRtrix3 forum for contributions once the basic framework is in place. It would also be useful if anyone with experience with a broader range of software packages has informed opinions on what kind of structure such a package would take, or if there's an existing software package that would operate as a suitable exemplar.

[francopestilli]

hi @Lestropie thanks for the thoughtful comments. It sounds like you think BIDS should be more abstract, more top-level, and abstract than any specific software and that there should be mechanisms to translate a BIDS dataset to what is expected by each software. Does that sound alright?

RE this is a good exercise for a CS student, indeed! As soon as we have a candidate we should try this as it will help identify weak points and issues.

[arokem]

+1 Software that implements conversions to/from the spec would be valuable, and a good thing to do sooner rather than later. I think that for many of the models, conversions to/from the proposed BIDS format are quite straightforward and should constitute (at most) reordering and/or concatenation of volumes of data + file renaming. For conversions between different SH representations, much of the work is already completed in the DIPY software (see https://github.com/dipy/dipy/blob/master/dipy/reconst/shm.py), and maybe also elsewhere, so there's a good basis to build on.

[Lestropie]

It sounds like you think BIDS should be more abstract, more top-level, and abstract than any specific software

This is probably an observation of my overall proposed approach to diffusion derivatives moreso than the content of this particular thread. :-P

there should be mechanisms to translate a BIDS dataset to what is expected by each software.

Preferably both directions, but the opposite is probably the higher priority of the two. If we simply present a specification document, but for the reader there is not even a tool available to produce data that conform to that specification, let alone validate whether or not data they have manually manipulated is or is not compatible with the specification, then there is little reason for the reader to invest the effort in engaging with the specification. At the very least we should be able to demonstrate to the community that we have gone to the effort of converting from existing softwares outputs to our proposed specification and can demonstrate that such a conversion is well-posed and sensible. In the absence of such, the community will think that we are being lazy by deferring the construction of such to someone else.

[Lestropie]

Reply to comment in #34 but that is more appropriate here:

It will not be possible / adequate in all circumstances to have purely "conversion" from the output of a tool into BEP016.

• Quite often, BEP016 will specify an OPTIONAL metadata field for which the relevant information is not available if the only data you have at hand is the output generated by that software; ie. some degree of provenance has been lost.
• In cases where there are multiple files generated by a tool, but the names of those files are under user control rather than specified a priori by the software, the interface to a conversion tool may become more complex.

Therefore we need to distinguish between tools that:

1. Take an output of an existing tool and convert it into BEP016;
2. As a BIDS App, take a DWI input, internally apply some tool, keeping track of / manipulating any relevant settings, and write the output as BEP016.

The former is more appealing, and I think it makes sense to commence there, but there is some chance that along the way the limitations described above may be encountered, and so I wanted this prospect to be documented.

[Lestropie]

Whoops; I'd already opined about this in PeerHerholz/bids_bep16_conv#7.