Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Waiting on shelf file updates: Check pickle file ordering #6

Open
rfrenchseti opened this issue Dec 4, 2023 · 3 comments
Open

Waiting on shelf file updates: Check pickle file ordering #6

rfrenchseti opened this issue Dec 4, 2023 · 3 comments

Comments

@rfrenchseti
Copy link
Collaborator

From rms-webtools created by rfrenchseti: SETI/rms-webtools#30

_get_shelf in pdsfile.py is sorting the pickle files as they are read because they are coming in out of order. But Python 3 stores dictionaries in insertion order, so we need to investigate why the pickle files are out of order. It could just be that some of the files are old and were written with Python 2, in which case we can update the pickle files and remove the sort.

@rfrenchseti
Copy link
Collaborator Author

This is probably no longer an issue since all files have been regenerated since this issue was filed. If this is true then the sort can be removed.

@rfrenchseti
Copy link
Collaborator Author

All of the shelf directories appear to have been updated since early 2021. We could start by choosing some of the oldest pickle files, reading them in, and checking whether they are already sorted.

@rfrenchseti
Copy link
Collaborator Author

From pull #52:

Some old pickle files like ones created in 2020 doesn't have sorted keys, for example: 'holdings/_infoshelf-archives-volumes/COISS_3xxx_info.pickle'
For ones created in 2021, they will have sorted keys, for example: 'holdings/_infoshelf-archives-volumes/COISS_0xxx_info.pickle' and 'holdings/_infoshelf-archives-volumes/HSTJx_xxxx_info.pickle', we can remove the sorting at line 5085 of _get_shelf function in pdsfile.py if all the shelf files are generated after 2021.

@rfrenchseti rfrenchseti changed the title Check pickle file ordering Waiting on shelf file updates: Check pickle file ordering Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants