As a data custodian, I want the Deep Archive to work around invalid URLs in the Registry

[nutjob4life]

Checked for duplicates

No - I haven't checked

🧑‍🔬 User Persona(s)

Users like the ones in NASA-PDS/operations#476

💪 Motivation

The Registry API seems to be loaded with some bad data, namely file paths like

'ops:Data_File_Info.ops:file_ref': ['https://pds-rings.seti.org/pds4/bundles/cassini_uvis_solarocc_beckerjarmak2023//data/collection_data.csv']

with a // between cassini_uvis_solarocc_beckerjarmak2023 and data. This causes the Deep Archive to output Submission Information Packages with double-slashes in them too, causing validation errors.

📖 Additional Details

See NASA-PDS/operations#476 for a specific example.

Acceptance Criteria

Given a document in OpenSearch containing double-slashes in the URL path
When I perform pds-deep-registry-archive on the bundle containing that document
Then I expect the file paths and URLs output to be "cleaned" up to single slashes

⚙️ Engineering Details

No response

[jordanpadams]

@nutjob4life I just realized I never triaged this and we probably need this cleaned up ASAP to unblock that operations ticket.

[nutjob4life]

@jordanpadams on it!

[jordanpadams]

Thanks @nutjob4life 🎉

[gxtchen]

@nutjob4life what url should I use to run the deep-registry-archive? I got "ValueError: 🤷‍♀️ The bundle urn:nasa:pds:cassini_uvis_solarocc_beckerjarmak2023::1.1 cannot be found in the registry at https://pds.nasa.gov/api/search/1.0/"

[nutjob4life]

Hi @gxtchen, I don't know the answer to this.

I believe the URL is correct but perhaps the registry is missing some data? @tloubrieu-jpl @jordanpadams could you take a peek? When I run it, I get the same thing:

mirasol 209 % .v/bin/pds-deep-registry-archive --site PDS_RNG urn:nasa:pds:cassini_uvis_solarocc_beckerjarmak2023::1.1
INFO 👟 PDS Deep Registry-based Archive, version 1.3.0
ERROR 💥 We got an unexpected error; sorry it didn't work out
Traceback (most recent call last):
  File "/Users/kelly/Documents/Clients/JPL/PDS/Development/nasa-pds/deep-archive/src/pds2/aipgen/registry.py", line 375, in main
    generatedeeparchive(args.url, args.bundle, args.site, not args.include_latest_collection_only)
  File "/Users/kelly/Documents/Clients/JPL/PDS/Development/nasa-pds/deep-archive/src/pds2/aipgen/registry.py", line 350, in generatedeeparchive
    prefixlen, bac, title = _comprehendregistry(url, bundlelidvid, allcollections)
  File "/Users/kelly/Documents/Clients/JPL/PDS/Development/nasa-pds/deep-archive/src/pds2/aipgen/registry.py", line 224, in _comprehendregistry
    raise ValueError(f"🤷‍♀️ The bundle {bundlelidvid} cannot be found in the registry at {url}")
ValueError: 🤷‍♀️ The bundle urn:nasa:pds:cassini_uvis_solarocc_beckerjarmak2023::1.1 cannot be found in the registry at https://pds.nasa.gov/api/search/1.0/
INFO 👋 Thanks for using this program! Bye!

[jordanpadams]

@gxtchen you cannot test this this with the public registry until the multi-tenancy migration has completed: NASA-PDS/registry#185

[jordanpadams]

you can try downloading and loading that data into a local registry and test with that

[tloubrieu-jpl]

@gxtchen can wait to test that until the API is up again.