Is there a way to automatically search through DANDI files?

[mudd54]

Hello,
I have derived data that I’m looking to link to the data of these 3 dandisets : https://dandiarchive.org/dandiset/000713?search=neuropixels&pos=2,https://dandiarchive.org/dandiset/000021, https://dandiarchive.org/dandiset/000409?search=IBL&pos=2 .

To do so I need to match my local files with the corresponding nwb hosted on DANDI. We are planning to use LINDI to do this @magland

It doesn’t seem feasible to search manually for each matching file. Is there a way to search automatically or programmatically through folders and file names on DANDI? These files have session ids from the original data but they do not seem to be available on DANDI that is obvious to me.
Thank you.

[magland]

@mudd54 I'm glad you're going to make use of LINDI for this, and I'm happy to assist.

You can use the dandiarchive python API for this
https://dandi.readthedocs.io/en/latest/modref/dandiarchive.html

Let me know if you want me to provide a template script to help you get started.

[magland]

@mudd54 There is an indexer that runs every morning that prepares lindi.json files for new public files on DANDI. It only runs partially for a fixed amount of time, so some may not be prepared yet. But I thought that 000713 was fully covered. Are you sure that asset ID corresponds to an asset that's actually in the current version of the 000713 Dandiset?

[mudd54]

@magland Below is the code I use to get the asset ID. I use dandiarchive to search through assets until an asset path matches my regex, and then I swap the corresponding asset id into the lindi urls.

I believe the dandiset url that gets parsed is current. Additionally, I've been manually double checking some of the asset IDs by accessing the metadata through dandi on my browser, and the asset IDs in the metadata and what my code returns match up.

If it is an issue with the indexer, I can wait until the dandiset has been completed.

def get_url_from_abi_visbehave_session_id(session_id_string, ecephys_sessions_df):
    
    '''
    Get the lindi url from the unique session identifier (allen visbehave)

    session_id_string: (str) String of the session id 
    ecephys_sessions_df: Csv of the ecephys metadata for this dandiset

    returns: lindi url corresponding to the dandi file 
    '''

    # dandi file folders are named after mice ids
    session_id_mask = ecephys_sessions_df['ecephys_session_id'] == int(session_id_string)

    dandi_folder = str(ecephys_sessions_df['mouse_id'][session_id_mask].item())
    dandi_doa = ecephys_sessions_df['date_of_acquisition'][session_id_mask].item()
    dandi_file_name_regex = datetime_to_regex(dandi_doa, dandi_folder)

    # get the url of the dandiset folder 
    dandi_folder_url = f'https://dandiarchive.org/dandiset/000713/0.240702.1725/files?location=sub-{dandi_folder}'

    # using the dandiarchive module, navigate through the dandi folder until a file name matches the regex.
    # This is unfortunately necessary because the session start times in the metadata csvs do not 
    # exactly match the session start times that are used to generate file names -_- 
    # looks like a timezone issue because the dates, minutes and seconds align, while the hours do not.
    # alternately, we could open files and check the session id ('identifier') inside, however it will be slow to load 
    parsed_url = dandiarchive.parse_dandi_url(dandi_folder_url)
    dandi_asset_id = str
    dandi_asset_path = str

    with parsed_url.navigate() as (client, dandiset, assets):

        asset_ids = parsed_url.get_asset_ids(client)

        for asset, asset_id in zip(assets,asset_ids):
            if re.match(dandi_file_name_regex, asset.path):
                print('Match found!')
                print(f'Asset path: {asset.path}, Asset ID: {asset_id} for session {session_id_string}')
                dandi_asset_id = asset_id
                dandi_asset_path = asset.path
                break
            
    
    if dandi_asset_id == '':
        print('No match found for the dandi file name')
        return None, None

    # URL of the remote .nwb.lindi.json file
    url = f'https://lindi.neurosift.org/dandi/dandisets/000713/assets/{dandi_asset_id}/nwb.lindi.json'
    dandi_asset_name = dandi_asset_path.split('/')[-1]

    return url, dandi_asset_name

[magland]

Makes sense. To figure this out, it would be helpful if you could provide the path of the asset with the ID that is not indexed.

[mudd54]

@magland Here's a csv file of the ones I'm interested in that have dead links (paths are in the second column).

I've been using the https://lindi.neurosift.org/dandi/dandisets/000713/assets/[asset_id]/nwb.lindi.json url format BTW. Thank you for your assistance with this.

000713_dead_links.csv

[mudd54]

@magland We are now at the stage where we would like to try uploading our nwbs with additional data as described here https://github.com/magland/lindi-dandi .

What would be the best way to go about uploading staged data to DANDI? It seems that currently lindi does not have the on_upload_blob and on_upload_main functions needed to run LindiH5pyFile.upload() . Can we adapt code from this https://github.com/magland/lindi-dandi/blob/main/devel/lindi_test_2.py file?

Additionally, is there special authentication needed? Would it be possible to obtain an api key for doing this?
Thank you for all your help.