Uniqueness of object_ids in generated analysis nwb files

[samuelbray32]

Is your feature request related to a problem? Please describe.

• In the current implementation of AnalysisNwbfile.create() the new nwb inherits the object id of both the nwb file and objects within it that aren't removed.
• This causes errors when attempting to make a dandiset since uniqueness is violated. Relevant to (3) in Features to allow sharing of full pipeline and results for paper publication #861
• The UUID of the object id is not a hash of the data, so in theory I think it would be ok to alter them to be unique.

Describe the solution you'd like
A retroactive solution needs to:

• alter the object_ids of existing files
• correct the stored datajoint checksum/hash
• correct any spyglass utilities that use this to find data

A forward solution needs to:

• ensure unique object ids are created when creating a new analysis file

Describe alternatives you've considered
Would need dandi to change restriction policy on overlapping uuid. Since the same uuid do redundantly refer to duplicated data in different nwb files this might not be unreasonable.

[samuelbray32]

@rly do you have any thoughts on altering uuids in existing nwb files?

[samuelbray32]

Good news: Dandi naming tools only require that the top level filename be unique. This can be achieved by editing the h5 file

[rly]

@samuelbray32 re: forward solution. Are new analysis NWB files created via the export function? You can generate new object IDs for each object in the nwbfile by calling nwbfile.generate_new_id() before export.

Ref: https://hdmf.readthedocs.io/en/stable/export.html#what-happens-to-object-ids-when-i-export

For older files, yes, you would need to edit the existing NWB file via h5py and do the following steps as you mentioned. For editing, you could do something like:

import h5py
from uuid import uuid4

path = "..."
def replace_object_id(name, obj):
    if isinstance(obj, (h5py.Group, h5py.Dataset)):
         if "object_id" in obj.attrs.keys():
             obj.attrs["object_id"] = str(uuid4())

with h5py.File(path, "r+") as f:
    f.attrs["object_id"] = str(uuid4())
    f.visititems(replace_object_id)