No metadata record found in Dandiset

[ZhuJiaBCM]

Hi, I'm trying to upload nwb fies, they passed validation, but I got this,

(base) C:\Windows\System32<dandi ID> dandi upload
2023-08-07 13:42:30,441 [ INFO] Note: NumExpr detected 40 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-08-07 13:42:30,442 [ INFO] NumExpr defaulting to 8 threads.
2023-08-07 13:42:33,197 [ INFO] Logs saved in C:\Users\NuoLiLab\AppData\Local\dandi\dandi-cli\Logs\20230807184228Z-20316.log
Error: No metadata record found in Dandiset

What does this metadata record mean? I have descriptions for the dandiset under "metadata" oprion

[bendichter]

@ZhuJiaBCM I think that means you don't have a .yaml file in your current working directory.

[ZhuJiaBCM]

Hi Ben, Thank you for replying. Then may I ask how to generate a .yaml file? I'm learning to use ANDI from scratch. I went through user guide, but it's not mentioned there. Is it generated when dandiset was built? I just got added to my dandiset by my boss. He created this dandiset. If so, can I regenerate a yaml file? Best, Jia Postdoctoral Associate Department of Neuroscience Baylor College of Medicine T632, 1 Baylor Plaza, Houston, TX, U.S.A. 77030

…

________________________________ From: Ben Dichter ***@***.***> Sent: Monday, August 7, 2023 17:24 To: dandi/helpdesk ***@***.***> Cc: Zhu, Jia ***@***.***>; Mention ***@***.***> Subject: Re: [dandi/helpdesk] No metadata record found in Dandiset (Discussion #108) ***CAUTION:*** This email is not from a BCM Source. Only click links or open attachments you know are safe.

________________________________ @ZhuJiaBCM<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ZhuJiaBCM&d=DwMCaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=P8svWHR0Yg1-eQthtvupNw&m=an9A5lSLuNy6LfzyGUBZFYyeAzhOUCAmQUoYZS4sP25Doo-Gr9IGK3KvPANHkShr&s=pd1wT9cXydcIcM86DNSrH613BC5MM35v1I4gjQxedp0&e=> I think that means you don't have a .yaml file in your current working directory. — Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dandi_helpdesk_discussions_108-23discussioncomment-2D6663190&d=DwMCaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=P8svWHR0Yg1-eQthtvupNw&m=an9A5lSLuNy6LfzyGUBZFYyeAzhOUCAmQUoYZS4sP25Doo-Gr9IGK3KvPANHkShr&s=yq-XlvobigdIbDPbLROSGNxETdZp0IUADq019eghLoc&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_BBVK2ZKGJ3CF4ZFOFF4MKDDXUFTJNANCNFSM6AAAAAA3HMYXU4&d=DwMCaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=P8svWHR0Yg1-eQthtvupNw&m=an9A5lSLuNy6LfzyGUBZFYyeAzhOUCAmQUoYZS4sP25Doo-Gr9IGK3KvPANHkShr&s=JZMOMO1U5iZKgRNyP_I1Zw1dc7XrXKxYPcO_-MwnAqs&e=>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[bendichter]

@ZhuJiaBCM Are you sure you are following these instructions? In particular, see 2e where you first need to download the dandiset and then cd into it. Are you sure you did that? The yaml file should be in that folder. What dandiset is this?

[ZhuJiaBCM]

Hi Ben, My dandiset is 000572. I started over again from downloading dandiset, this time it says I already have a .yaml file. And I tested with uploading 1/3 of my files, this is what I got. (base) C:\Users\NuoLiLab>dandi download https://dandiarchive.org/dandiset/000572/draft 2023-08-08 13:23:53,991 [ INFO] Note: NumExpr detected 40 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2023-08-08 13:23:53,992 [ INFO] NumExpr defaulting to 8 threads. PATH SIZE DONE DONE% CHECKSUM STATUS MESSAGE 000572\dandiset.yaml updating PATH SIZE DONE DONE% CHECKSUM STATUS MESSAGE 000572\dandiset.yaml error already exists Summary: 0 Bytes 1 error 1 already exists <0.00% 2023-08-08 13:23:57,514 [ INFO] Logs saved in C:\Users\NuoLiLab\AppData\Local\dandi\dandi-cli\Logs\20230808182352Z-11880.log (base) C:\Users\NuoLiLab>cd 000572 (base) C:\Users\NuoLiLab\000572>dandi organize D:\Data_share\Upload\ 2023-08-08 13:26:44,786 [ INFO] Note: NumExpr detected 40 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2023-08-08 13:26:44,787 [ INFO] NumExpr defaulting to 8 threads. 2023-08-08 13:26:46,970 [ INFO] Loading metadata from 62 files [Parallel(n_jobs=-1)]: Using backend LokyBackend with 40 concurrent workers. [Parallel(n_jobs=-1)]: Done 4 out of 62 | elapsed: 5.4s remaining: 1.3min [Parallel(n_jobs=-1)]: Done 11 out of 62 | elapsed: 5.4s remaining: 25.5s [Parallel(n_jobs=-1)]: Done 18 out of 62 | elapsed: 5.4s remaining: 13.5s [Parallel(n_jobs=-1)]: Done 25 out of 62 | elapsed: 5.5s remaining: 8.1s [Parallel(n_jobs=-1)]: Done 32 out of 62 | elapsed: 5.5s remaining: 5.2s [Parallel(n_jobs=-1)]: Done 39 out of 62 | elapsed: 5.6s remaining: 3.2s [Parallel(n_jobs=-1)]: Done 46 out of 62 | elapsed: 5.6s remaining: 1.9s [Parallel(n_jobs=-1)]: Done 53 out of 62 | elapsed: 5.6s remaining: 0.9s [Parallel(n_jobs=-1)]: Done 60 out of 62 | elapsed: 5.7s remaining: 0.1s [Parallel(n_jobs=-1)]: Done 62 out of 62 | elapsed: 5.7s finished 2023-08-08 13:26:52,767 [ INFO] Symlink and hardlink tests both failed; setting files_mode='copy' 2023-08-08 13:26:52,801 [ INFO] Logs saved in C:\Users\NuoLiLab\AppData\Local\dandi\dandi-cli\Logs\20230808182642Z-17484.log Error: 62 paths already exist: C:\Users\NuoLiLab\000572\sub-BAYLORNL23\sub-BAYLORNL23_ses-20190225_ecephys+ogen.nwb, C:\Users\NuoLiLab\000572\sub-BAYLORNL23\sub-BAYLORNL23_ses-20190226_ecephys+ogen.nwb, C:\Users\NuoLiLab\000572\sub-BAYLORNL23\sub-BAYLORNL23_ses-20190228_ecephys+ogen.nwb, C:\Users\NuoLiLab\000572\sub-BAYLORNL23\sub-BAYLORNL23_ses-20190301_ecephys+ogen.nwb, C:\Users\NuoLiLab\000572\sub-BAYLORNL23\sub-BAYLORNL23_ses-20190318_ecephys+ogen.nwb and more. Remove them first. So now my question becomes how to remove the paths? Best, Jia Postdoctoral Associate Department of Neuroscience Baylor College of Medicine T632, 1 Baylor Plaza, Houston, TX, U.S.A. 77030

…

________________________________ From: Ben Dichter ***@***.***> Sent: Tuesday, August 8, 2023 12:53 To: dandi/helpdesk ***@***.***> Cc: Zhu, Jia ***@***.***>; Mention ***@***.***> Subject: Re: [dandi/helpdesk] No metadata record found in Dandiset (Discussion #108) ***CAUTION:*** This email is not from a BCM Source. Only click links or open attachments you know are safe.

________________________________ @ZhuJiaBCM<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ZhuJiaBCM&d=DwMCaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=P8svWHR0Yg1-eQthtvupNw&m=m3fCEGKJqQ6HZgkBf_m4-_nt4IgvUvjY_DYr6YVg4hWxTL17Zkc0oMEafna0xqaa&s=dirpPuGPXvNfSRx1VtxpU4QBmlJyHv-N-R29yziboo4&e=> Are you sure you are following these instructions<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.dandiarchive.org_handbook_13-5Fupload_&d=DwMCaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=P8svWHR0Yg1-eQthtvupNw&m=m3fCEGKJqQ6HZgkBf_m4-_nt4IgvUvjY_DYr6YVg4hWxTL17Zkc0oMEafna0xqaa&s=FKgF3keDUU_LXKRc7ZP7I0_f5JjJKvufaePUxHxwZuQ&e=>? In particular, see 2e where you first need to download the dandiset and then cd into it. Are you sure you did that? The yaml file should be in that folder. What dandiset is this? — Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dandi_helpdesk_discussions_108-23discussioncomment-2D6672155&d=DwMCaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=P8svWHR0Yg1-eQthtvupNw&m=m3fCEGKJqQ6HZgkBf_m4-_nt4IgvUvjY_DYr6YVg4hWxTL17Zkc0oMEafna0xqaa&s=sSfEwpmMLtiqmQjeQH4VoTNRLPSOKogKEWnfqBTFCYc&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_BBVK2ZJIKVSHOWKUFZ5MSLLXUJ4JLANCNFSM6AAAAAA3HMYXU4&d=DwMCaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=P8svWHR0Yg1-eQthtvupNw&m=m3fCEGKJqQ6HZgkBf_m4-_nt4IgvUvjY_DYr6YVg4hWxTL17Zkc0oMEafna0xqaa&s=u5lQJEOYJixCgSb3nLb2xWmavpnKIAiakt8Hxt0mDZg&e=>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[CodyCBakerPhD]

This may be due to some confusion on dandi organize as to which files are being replaced in the current DANDI set.

I'm unable to view your DANDI set to confirm (guessing it is embargoed)

What I would suggest is try that first step again with the flag

dandi download https://dandiarchive.org/dandiset/000572/draft --download dandiset.yaml

and that will make a more empty directory (only the .yaml file should be in the folder) from which to proceed with your dandi organize and dandi upload (which will then replace any files on the DANDI set with the new ones, if their file names are the same but contents are different)

[ZhuJiaBCM]

Hi Ben, This seems not working for me. (base) C:\Users\NuoLiLab>dandi download https://dandiarchive.org/dandiset/000572/draft --download dandiset.yaml 2023-08-08 22:45:35,837 [ INFO] Note: NumExpr detected 40 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2023-08-08 22:45:35,837 [ INFO] NumExpr defaulting to 8 threads. PATH SIZE DONE DONE% CHECKSUM STATUS MESSAGE 000572\dandiset.yaml updating PATH SIZE DONE DONE% CHECKSUM STATUS MESSAGE 000572\dandiset.yaml error already exists Summary: 0 Bytes 1 error 1 already exists <0.00% 2023-08-08 22:45:39,294 [ INFO] Logs saved in C:\Users\NuoLiLab\AppData\Local\dandi\dandi-cli\Logs\20230809034534Z-3896.log (base) C:\Users\NuoLiLab>cd 000572 (base) C:\Users\NuoLiLab\000572>dandi organize D:\Data_share\Upload\ 2023-08-08 22:47:08,661 [ INFO] Note: NumExpr detected 40 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2023-08-08 22:47:08,661 [ INFO] NumExpr defaulting to 8 threads. 2023-08-08 22:47:10,909 [ INFO] Loading metadata from 62 files [Parallel(n_jobs=-1)]: Using backend LokyBackend with 40 concurrent workers. [Parallel(n_jobs=-1)]: Done 4 out of 62 | elapsed: 5.5s remaining: 1.4min [Parallel(n_jobs=-1)]: Done 11 out of 62 | elapsed: 5.6s remaining: 26.1s [Parallel(n_jobs=-1)]: Done 18 out of 62 | elapsed: 5.6s remaining: 13.8s [Parallel(n_jobs=-1)]: Done 25 out of 62 | elapsed: 5.6s remaining: 8.4s [Parallel(n_jobs=-1)]: Done 32 out of 62 | elapsed: 5.6s remaining: 5.3s [Parallel(n_jobs=-1)]: Done 39 out of 62 | elapsed: 5.7s remaining: 3.3s [Parallel(n_jobs=-1)]: Done 46 out of 62 | elapsed: 5.7s remaining: 1.9s [Parallel(n_jobs=-1)]: Done 53 out of 62 | elapsed: 5.8s remaining: 0.9s [Parallel(n_jobs=-1)]: Done 60 out of 62 | elapsed: 5.8s remaining: 0.1s [Parallel(n_jobs=-1)]: Done 62 out of 62 | elapsed: 5.8s finished 2023-08-08 22:47:16,829 [ INFO] Symlink and hardlink tests both failed; setting files_mode='copy' 2023-08-08 22:47:16,870 [ INFO] Logs saved in C:\Users\NuoLiLab\AppData\Local\dandi\dandi-cli\Logs\20230809034707Z-9036.log Error: 62 paths already exist: C:\Users\NuoLiLab\000572\sub-BAYLORNL23\sub-BAYLORNL23_ses-20190225_ecephys+ogen.nwb, C:\Users\NuoLiLab\000572\sub-BAYLORNL23\sub-BAYLORNL23_ses-20190226_ecephys+ogen.nwb, C:\Users\NuoLiLab\000572\sub-BAYLORNL23\sub-BAYLORNL23_ses-20190228_ecephys+ogen.nwb, C:\Users\NuoLiLab\000572\sub-BAYLORNL23\sub-BAYLORNL23_ses-20190301_ecephys+ogen.nwb, C:\Users\NuoLiLab\000572\sub-BAYLORNL23\sub-BAYLORNL23_ses-20190318_ecephys+ogen.nwb and more. Remove them first. Best, Jia Postdoctoral Associate Department of Neuroscience Baylor College of Medicine T632, 1 Baylor Plaza, Houston, TX, U.S.A. 77030

…

________________________________ From: Cody Baker ***@***.***> Sent: Tuesday, August 8, 2023 14:26 To: dandi/helpdesk ***@***.***> Cc: Zhu, Jia ***@***.***>; Mention ***@***.***> Subject: Re: [dandi/helpdesk] No metadata record found in Dandiset (Discussion #108) ***CAUTION:*** This email is not from a BCM Source. Only click links or open attachments you know are safe.

________________________________ This may be due to some confusion on dandi organize as to which files are being replaced in the current DANDI set. I'm unable to view your DANDI set to confirm (guessing it is embargoed) What I would suggest is try that first step again with the flag dandi download https://dandiarchive.org/dandiset/000572/draft --download dandiset.yaml and that will make a more empty directory from which to proceed with your dandi organize and dandi upload (which will then replace any files on the DANDI set with the new ones, if their file names are the same but contents are different) — Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dandi_helpdesk_discussions_108-23discussioncomment-2D6672844&d=DwMCaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=P8svWHR0Yg1-eQthtvupNw&m=A8zEDvZbzY_wpVK7lZEm3c8LMFiF0_H2VIfdz81_ZZUkL3hpFplYoMLnM7qlhzB4&s=ErZnyEM-TiVWaV-I6LCreY4vRk9Z5ABmjkStM8CbLZg&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_BBVK2ZIFOJVGJVTQWYSIRADXUKHGPANCNFSM6AAAAAA3HMYXU4&d=DwMCaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=P8svWHR0Yg1-eQthtvupNw&m=A8zEDvZbzY_wpVK7lZEm3c8LMFiF0_H2VIfdz81_ZZUkL3hpFplYoMLnM7qlhzB4&s=aKMyiWP_zHB5_eW4GwQeV4kcbQvpQwCa-_v33vJAW2g&e=>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[bendichter]

What are you trying to do exactly? Are you trying to replace all the files in the dandiset?

[ZhuJiaBCM]

The aim is simple. I just want to upload my files onto this dandiset. However, after several rounds of trial and error. Now the "paths" are on the dandiset, the data had not even been uploaded yet. Then I have to deal with these "exiting paths", which I totally couldn't see from the dandiset. [cid:881e4ffe-7455-4280-a24b-1a7a59107731] There is literally no file on there. Jia Postdoctoral Associate Department of Neuroscience Baylor College of Medicine T632, 1 Baylor Plaza, Houston, TX, U.S.A. 77030

…

________________________________ From: Ben Dichter ***@***.***> Sent: Tuesday, August 8, 2023 22:53 To: dandi/helpdesk ***@***.***> Cc: Zhu, Jia ***@***.***>; Mention ***@***.***> Subject: Re: [dandi/helpdesk] No metadata record found in Dandiset (Discussion #108) ***CAUTION:*** This email is not from a BCM Source. Only click links or open attachments you know are safe.

________________________________ What are you trying to do exactly? Are you trying to replace all the files in the dandiset? — Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dandi_helpdesk_discussions_108-23discussioncomment-2D6675772&d=DwMCaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=P8svWHR0Yg1-eQthtvupNw&m=4KvgaaAfvY0NAvFd0cPlSIOdIEfETChPyVpOgUidJ8_ksICYL1liXWqy45Em66rA&s=KAL02a0q3-bgb7g0sKFVPIPe0S_wldtCrZgScqudMww&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_BBVK2ZPAQGGAPCGS4LX3RJDXUMCTNANCNFSM6AAAAAA3HMYXU4&d=DwMCaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=P8svWHR0Yg1-eQthtvupNw&m=4KvgaaAfvY0NAvFd0cPlSIOdIEfETChPyVpOgUidJ8_ksICYL1liXWqy45Em66rA&s=ECC8mzGqOiRWKesdY5c6IsvlNdNCA0fycnA6MN55zec&e=>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[CodyCBakerPhD]

Despite the errors reported from running dandi organize, does the local folder created via the --download dandiset.yaml have the desired files inside it? If so the last thing to try is simply dandi upload.

Does that proceed to upload the files to the archive?

If no, please grab a time on my calendly so we can meet one on one to get to the bottom of this

[ZhuJiaBCM]

I go to the local folder, deleted everything, redo it again. As following, (base) C:\Users\NuoLiLab>dandi download https://dandiarchive.org/dandiset/000572/draft --download dandiset.yaml 2023-08-08 23:37:07,707 [ INFO] Note: NumExpr detected 40 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2023-08-08 23:37:07,708 [ INFO] NumExpr defaulting to 8 threads. PATH SIZE DONE DONE% CHECKSUM STATUS MESSAGE 000572\dandiset.yaml updating Summary: 0 Bytes 1 updating <0.00% 2023-08-08 23:37:10,817 [ INFO] Logs saved in C:\Users\NuoLiLab\AppData\Local\dandi\dandi-cli\Logs\20230809043706Z-20068.log Error: 'charmap' codec can't encode character '\u0107' in position 944: character maps to <undefined> (base) C:\Users\NuoLiLab>cd 000572 (base) C:\Users\NuoLiLab\000572>dandi organize C:\Users\NuoLiLab\000572\ 2023-08-08 23:46:27,583 [ INFO] Note: NumExpr detected 40 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2023-08-08 23:46:27,584 [ INFO] NumExpr defaulting to 8 threads. 2023-08-08 23:46:29,661 [ INFO] Loading metadata from 62 files [Parallel(n_jobs=-1)]: Using backend LokyBackend with 40 concurrent workers. C:\ProgramData\Anaconda3\lib\site-packages\hdmf\spec\namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.5.0 because version 1.7.0 is already loaded. warn("Ignoring cached namespace '%s' version %s because version %s is already loaded." C:\ProgramData\Anaconda3\lib\site-packages\hdmf\spec\namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.1.0 because version 0.4.0 is already loaded. warn("Ignoring cached namespace '%s' version %s because version %s is already loaded." C:\ProgramData\Anaconda3\lib\site-packages\hdmf\spec\namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.5.0 because version 1.7.0 is already loaded. ... ... 62 files went through [Parallel(n_jobs=-1)]: Done 4 out of 62 | elapsed: 6.7s remaining: 1.6min [Parallel(n_jobs=-1)]: Done 11 out of 62 | elapsed: 6.8s remaining: 32.1s [Parallel(n_jobs=-1)]: Done 18 out of 62 | elapsed: 6.9s remaining: 17.0s [Parallel(n_jobs=-1)]: Done 25 out of 62 | elapsed: 6.9s remaining: 10.3s [Parallel(n_jobs=-1)]: Done 32 out of 62 | elapsed: 7.0s remaining: 6.5s [Parallel(n_jobs=-1)]: Done 39 out of 62 | elapsed: 7.1s remaining: 4.1s [Parallel(n_jobs=-1)]: Done 46 out of 62 | elapsed: 7.2s remaining: 2.4s [Parallel(n_jobs=-1)]: Done 53 out of 62 | elapsed: 7.3s remaining: 1.2s [Parallel(n_jobs=-1)]: Done 60 out of 62 | elapsed: 7.3s remaining: 0.1s [Parallel(n_jobs=-1)]: Done 62 out of 62 | elapsed: 7.4s finished 2023-08-08 23:46:37,197 [ INFO] Hard link support autodetected; setting files_mode='hardlink' 2023-08-08 23:46:37,254 [ INFO] Organized 62 paths. Visit C:\Users\NuoLiLab\000572/ 2023-08-08 23:46:37,255 [ INFO] Logs saved in C:\Users\NuoLiLab\AppData\Local\dandi\dandi-cli\Logs\20230809044626Z-14164.log (base) C:\Users\NuoLiLab\000572>dandi upload 2023-08-08 23:47:25,798 [ INFO] Note: NumExpr detected 40 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2023-08-08 23:47:25,798 [ INFO] NumExpr defaulting to 8 threads. 2023-08-08 23:47:28,891 [ INFO] Logs saved in C:\Users\NuoLiLab\AppData\Local\dandi\dandi-cli\Logs\20230809044724Z-1564.log Error: No metadata record found in Dandiset In my local folder, after running dandi organize, it looks like this, nwb files are here, organized sub folders are here, and the downloaded yaml file is also here. However, the error is "no metadata record found in Dandiset" again. Everything goes back to beginning, my original question...... But I noiced the yaml file is 0kb. That might be the problem? [cid:e1ce5cc8-5f50-49be-a971-ceb491f81329] [cid:32841ab4-7779-4f8f-9346-caa5b22a8444] I set up a meeting Fri. 8/11 morning, 9am EST. Best, Jia Postdoctoral Associate Department of Neuroscience Baylor College of Medicine T632, 1 Baylor Plaza, Houston, TX, U.S.A. 77030

…

________________________________ From: Cody Baker ***@***.***> Sent: Tuesday, August 8, 2023 23:28 To: dandi/helpdesk ***@***.***> Cc: Zhu, Jia ***@***.***>; Mention ***@***.***> Subject: Re: [dandi/helpdesk] No metadata record found in Dandiset (Discussion #108) ***CAUTION:*** This email is not from a BCM Source. Only click links or open attachments you know are safe.

________________________________ Despite the errors reported from running dandi organize, does the local folder created via the --download dandiset.yaml have the files inside it? If so the last thing to try is simply dandi upload. Does that proceed to upload the files to the archive? If no, please grab a time on my calendly<https://urldefense.proofpoint.com/v2/url?u=https-3A__calendly.com_cody-2Dc-2Dbaker_60min-3Fmonth-3D2023-2D08&d=DwMCaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=P8svWHR0Yg1-eQthtvupNw&m=1_XhorJx0aBO6675jWWpYcH8x69zbCC2fuoTgSTwwmfTyBAWBSlTlKkSPEtMj7E6&s=MSYAUmgb4eQDZFls8i8rVwOHPRzTYqYUYMhfmaQljxA&e=> so we can meet one on one to get to the bottom of this — Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dandi_helpdesk_discussions_108-23discussioncomment-2D6675936&d=DwMCaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=P8svWHR0Yg1-eQthtvupNw&m=1_XhorJx0aBO6675jWWpYcH8x69zbCC2fuoTgSTwwmfTyBAWBSlTlKkSPEtMj7E6&s=DSZzGx64tIuMA--Ppj8csWgsSTsE07bybTF25Bc-m9I&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_BBVK2ZIGZRECTFFF3D5VZ3LXUMGVVANCNFSM6AAAAAA3HMYXU4&d=DwMCaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=P8svWHR0Yg1-eQthtvupNw&m=1_XhorJx0aBO6675jWWpYcH8x69zbCC2fuoTgSTwwmfTyBAWBSlTlKkSPEtMj7E6&s=b_SQCEnoGhGIO1aUs2gJuFPve7VOfV2FjhZIrdyALAI&e=>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[CodyCBakerPhD]

Great, I'll see you on Friday to resolve these issues

But I noiced the yaml file is 0kb. That might be the problem?

That would be a big problem; sorry for not catching that earlier. The dandiset.yaml file should be small but non-zero and that is a necessary step for determining destination of the upload to the server

Now that I see the exact error output from that step, my hypothesis is that there is some entry in the DANDI set metadata (that you can view and later from the webpage) that contains a special character, likely the character 'ć', and this causes an encoding problem when it tries to write the metadata to the local file.

Can you look through the DANDI set metadata on the web page to see if there are any entries that look like that? And if so, try setting it to a classic 'c' and then try to dandi download ... --download dandiset.yaml again and see what the file size is?

[ZhuJiaBCM]

Hi Cody, I have two things to tell you, First, I do have a letter ć, in one of the authors' name! LOL She is from the Nehterlands. So I will try to replace it with a normal "c" and see if it works. Second, can I reschedule our meeting? I'm currently in Bates College in Maine, for a research meeting. Friday morning used to be free for me, but now it's not. I want to take the shuttle bus the meeting offers to the airport so I don't need to wait for a Uber. But it will departure at our meeting time!!! Shall I cancel Friday's meeting and use the same link you sent previously to reschedule another appointment, how about Saturday? I will try uploading again tonight, if it works, the following meeting might not be needed then. But Friday's meeting has to be canceled, sorry for the inconvenience. Best, Jia Postdoctoral Associate Department of Neuroscience Baylor College of Medicine T632, 1 Baylor Plaza, Houston, TX, U.S.A. 77030

…

________________________________ From: Cody Baker ***@***.***> Sent: Wednesday, August 9, 2023 1:05 To: dandi/helpdesk ***@***.***> Cc: Zhu, Jia ***@***.***>; Mention ***@***.***> Subject: Re: [dandi/helpdesk] No metadata record found in Dandiset (Discussion #108) ***CAUTION:*** This email is not from a BCM Source. Only click links or open attachments you know are safe.

________________________________ Great, I'll see you on Friday to resolve these issues But I noiced the yaml file is 0kb. That might be the problem? That would be a big problem; sorry for not catching that earlier. The dandiset.yaml file should be small but non-zero and that is a necessary step for determining destination of the upload to the server Now that I see the exact error output from that step, my hypothesis is that there is some entry in the DANDI set metadata (that you can view and later from the webpage) that contains a special character, likely the character 'ć', and this causes an encoding problem when it tries to write the metadata to the local file. Can you look through the DANDI set metadata on the web page to see if there are any entries that look like that? And if so, try setting it to a classic 'c' and then try to dandi download ... --download dandiset.yaml again and see what the file size is? — Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dandi_helpdesk_discussions_108-23discussioncomment-2D6676448&d=DwMFaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=P8svWHR0Yg1-eQthtvupNw&m=jM-g1r7Im26LKFzpOXobSGYDRaVGNXmUV1c0nAEAVH0tVRx3bIrwWp6M5CeGrmrb&s=KGL6oYiurKglPykZiE4WLmrQt-33JsRyDMjSRCOfgrA&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_BBVK2ZOMG6JDI3WWESVYZO3XUMSCRANCNFSM6AAAAAA3HMYXU4&d=DwMFaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=P8svWHR0Yg1-eQthtvupNw&m=jM-g1r7Im26LKFzpOXobSGYDRaVGNXmUV1c0nAEAVH0tVRx3bIrwWp6M5CeGrmrb&s=uISaIbIRKh0tJKltfZ_BFW2duAiGVIIAhJc6L4dwxXg&e=>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[ZhuJiaBCM]

Hi Cody, I changed the letter c and regenerated a 3kb yaml file. It seems to be working now. Now I have another question, after running " dandi organize", the data in my local folder were organized into separate subfolders according to animal ID, while the raw nwb files are still listed under the main folder, then I have two copies of the nwb files. My question is will the two copies all been uploaded? Shall I move the raw files elsewhere and upload the organized subfolders. Best, Jia Postdoctoral Associate Department of Neuroscience Baylor College of Medicine T632, 1 Baylor Plaza, Houston, TX, U.S.A. 77030

…

________________________________ From: Zhu, Jia ***@***.***> Sent: Wednesday, August 9, 2023 13:35 To: dandi/helpdesk ***@***.***> Subject: Re: [dandi/helpdesk] No metadata record found in Dandiset (Discussion #108) Hi Cody, I have two things to tell you, First, I do have a letter ć, in one of the authors' name! LOL She is from the Nehterlands. So I will try to replace it with a normal "c" and see if it works. Second, can I reschedule our meeting? I'm currently in Bates College in Maine, for a research meeting. Friday morning used to be free for me, but now it's not. I want to take the shuttle bus the meeting offers to the airport so I don't need to wait for a Uber. But it will departure at our meeting time!!! Shall I cancel Friday's meeting and use the same link you sent previously to reschedule another appointment, how about Saturday? I will try uploading again tonight, if it works, the following meeting might not be needed then. But Friday's meeting has to be canceled, sorry for the inconvenience. Best, Jia Postdoctoral Associate Department of Neuroscience Baylor College of Medicine T632, 1 Baylor Plaza, Houston, TX, U.S.A. 77030

________________________________ From: Cody Baker ***@***.***> Sent: Wednesday, August 9, 2023 1:05 To: dandi/helpdesk ***@***.***> Cc: Zhu, Jia ***@***.***>; Mention ***@***.***> Subject: Re: [dandi/helpdesk] No metadata record found in Dandiset (Discussion #108) ***CAUTION:*** This email is not from a BCM Source. Only click links or open attachments you know are safe.

________________________________ Great, I'll see you on Friday to resolve these issues But I noiced the yaml file is 0kb. That might be the problem? That would be a big problem; sorry for not catching that earlier. The dandiset.yaml file should be small but non-zero and that is a necessary step for determining destination of the upload to the server Now that I see the exact error output from that step, my hypothesis is that there is some entry in the DANDI set metadata (that you can view and later from the webpage) that contains a special character, likely the character 'ć', and this causes an encoding problem when it tries to write the metadata to the local file. Can you look through the DANDI set metadata on the web page to see if there are any entries that look like that? And if so, try setting it to a classic 'c' and then try to dandi download ... --download dandiset.yaml again and see what the file size is? — Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dandi_helpdesk_discussions_108-23discussioncomment-2D6676448&d=DwMFaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=P8svWHR0Yg1-eQthtvupNw&m=jM-g1r7Im26LKFzpOXobSGYDRaVGNXmUV1c0nAEAVH0tVRx3bIrwWp6M5CeGrmrb&s=KGL6oYiurKglPykZiE4WLmrQt-33JsRyDMjSRCOfgrA&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_BBVK2ZOMG6JDI3WWESVYZO3XUMSCRANCNFSM6AAAAAA3HMYXU4&d=DwMFaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=P8svWHR0Yg1-eQthtvupNw&m=jM-g1r7Im26LKFzpOXobSGYDRaVGNXmUV1c0nAEAVH0tVRx3bIrwWp6M5CeGrmrb&s=uISaIbIRKh0tJKltfZ_BFW2duAiGVIIAhJc6L4dwxXg&e=>. You are receiving this because you were mentioned.Message ID: ***@***.***>