Proper data type for upload of In Vitro Data

[Icardij]

Hello,

I am currently working on a optical physiology data set for in vivo data, and was recently asked to add some of our preliminary in vitro data to the set as well. All our in vitro data is simple data, and we store it in a excel file, without movies, images, ROIs, or most any other data that is asked for in the default optical phys NWB file format. This brings me to my question, is there a way to upload a excel or matfile along with a description of the data contained within? if not is there another file type that can be used to contain, and upload this data? I am hoping to house all this data together in a single file, since by the end of our project it will describe a small amount of data on a few hundred viral constructs, and would be easier to read together, but I am unsure if that is possible.

Thanks

[bendichter]

@Icardij there is not currently a way to upload excel or .mat files. Can you describe a little more about your in vitro data? What are you recording? What does the data represent?

We have tutorials demonstrating how to add patch clamp data in Python and MATLAB. Would this be helpful?

[Icardij]

We are recording fluorescence of purified protein. It is supplementary information for the "Developing CaMPARI3" Dandiset. Because of the way the fluorescence was recorded we don't have images, or ROIs to segment. I have been using the optical physiology module in MATLAB for our in vivo data, but due to the plane segmentation core being a prerequisite for the fluorescence, I can't use it for this data. I'm not well versed in programing, coding, or ether python or matlab language, so there may be simple solution to this, but I couldn't find anything after a few hours of searching. I think all I need is a way to upload fluorescence without a image or ROIs, but I'm not positive if that exists, or if there's a better way of uploading the data then that.

[CodyCBakerPhD]

Was this fluorescence recorded using fibre photometry?

Was the in vitro sample taken from a slice from the in vivo subject (if so do you have all the subject information, such as species, age, and sex)?

As a fallback, you can always add this as a custom generic TimeSeries if it doesn't follow any of the more specialized representations; but to upload to DANDI you will need to attach the 'effective' subject metadata that relates to the companion in vivo data

[Icardij]

I am unsure of how it was recorded, since it was recorded in a collaborator's lab. I am currently waiting on a answer from them.

No, this is not samples taken from in vivo subjects. We are developing a calcium sensor, and all the in vitro data is acquired before testing in vivo.

As for the fallback, we do have unique identifiers for each protein, so we could create subject metadata, though the species, age, and sex would be irrelevant, since it's just purified protein.

[Icardij]

Just got a answer on the recording, the fluorescence is recorded just by a photodiode.

Purified protein is placed in a solution, a excitatory laser is shone into the solution, and the output is read by a photodiode, which is the number we get.

[CodyCBakerPhD]

@Icardij OK, sorry for the delay here. This took some internal discussion how to handle this case as it is the first time we've seen data from a non-animal participant. Previous in vitro contributions have been tissue slices from a previously living animal subject so the same rules applied there as for in vivo

For the NWB file, I'd do the following

• Create it with the following name pattern (DANDI organize helper will not help us here): sub-<name of purified protein>_ses-flourescence-recording_ophys.nwb
• Give a session_id of flourescence-recording to the NWB file metadata
• Add the flourescence data from the photodiode as a basic TimeSeries in acquisition with a name and description appropriately set to describe the data
• Add a Subject with subject_id=<name of purified protein>; ignore the age, sex, and species fields; I recommend adding a description that talks about how it's in vitro and any other details about the protein you think are important
• Once you create your dandiset, and you have downloaded your local copy when preparing for upload, create the following folder in that dandiset named: sub-<name of purified protein> and copy+paste or move your NWB file into this folder

At that point, ping me again on this discussion and I'll provide instructions on how to upload - I'll try to push some efforts forward on our side to create an established workflow for this, but if you beat me to it then I can provide a workaround

[bendichter]

https://www.dandiarchive.org/handbook/13_upload/#data-uploadmanagement-workflow

[Icardij]

I am likely making a simple mistake, however I can not get the validation to ignore more than one check. It seems to be taking only the last check I ask it to ignore, and I can't figure out how to create a list of all checks to ignore. As mentioned before this is purified protein, so has to age, sex, or species all of which are critical validation errors. Is there a way to set up the validation command that it ignores these three things while still validating the file for upload?

[bendichter]

Can you please provide examples of the commands you are using?

[CodyCBakerPhD]

@Icardij we are working on a PR to the validator that will admit in vitro data for your purposes: NeurodataWithoutBorders/nwbinspector#431

The standard DANDI workflow Ben linked will not work out the box for your data

[CodyCBakerPhD]

@Icardij OK, sorry that took so long to get through

Here's what you need to do to upload now

pip install -U nwbinspector

(should show up as 0.4.33)

Then modify all your current files so that your subject_id starts with the keyword protein. Here is a simple script I put together, you'd just need to specify the dandi base folder on your side

import pathlib

import h5py

dandiset_folder = pathlib.Path("<change this to match your system>/000XYZ")
dandiset_files = dandiset_folder.rglob("*.nwb")

for dandiset_file in dandiset_files:
    with h5py.File(name=dandiset_file, mode="r+") as file:
        current_subject_id = file["general"]["subject"]["subject_id"][()].decode("utf-8")
        if not current_subject_id.startswith("protein"):
            file["general"]["subject"]["subject_id"][()] = f"protein{current_subject_id}"

for provenance I'd also adjust your original conversion script just in case you ever need to refer back it in the future

After that you should be able to pass validation and upload everything; let me know if anything doesn't work this time