Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Granules with very large feature counts cannot be added to hydrocron due to lambda timeout #258

Closed
torimcd opened this issue Oct 25, 2024 · 0 comments · Fixed by #259
Closed
Assignees
Labels
bug Something isn't working

Comments

@torimcd
Copy link
Collaborator

torimcd commented Oct 25, 2024

Some granules contain upwards of 140,000 feature records (eg SWOT_L2_HR_LakeSP_Prior_020_123_AR_20240825T010843_20240825T011234_PIC0_01.zip). The lambda that submits the records to be inserted into dynamodb times out after 10 minutes during unpacking the features from the shapefile and adding units and other info from the filename to each record.
My guess is this is mostly due to the for loop here:

def assemble_attributes(geodf, attributes):

Because this happens during the granule unpacking none of the features get loaded before the timeout.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: ✅ Done
Development

Successfully merging a pull request may close this issue.

1 participant