Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Metadata updates assume that keys are the same between old and new version #2325

Closed
MatMoore opened this issue Nov 13, 2023 · 0 comments · Fixed by #2462
Closed

🐛 Metadata updates assume that keys are the same between old and new version #2325

MatMoore opened this issue Nov 13, 2023 · 0 comments · Fixed by #2462
Assignees
Labels
bug Something isn't working

Comments

@MatMoore
Copy link
Contributor

MatMoore commented Nov 13, 2023

Describe the bug.

When updating example_prison_data_product_metadata to add a retentionPeriod, I got this stacktrace:

[ERROR] KeyError: 'retentionPeriod'Traceback (most recent call last):
  File "/var/task/update_metadata.py", line 42, in handler
    new_version = version_creator.update_metadata(new_metadata)  File "/var/task/versioning.py", line 98, in update_metadata
    state = metadata_update_type(metadata)  File "/var/task/versioning.py", line 135, in metadata_update_type
    changed_fields = data_product_metadata.changed_fields()  File "/var/task/data_product_metadata.py", line 245, in changed_fields
    if new_version[field] == latest_version[field]:
 | [ERROR] KeyError: 'retentionPeriod' Traceback (most recent call last):
   File "/var/task/update_metadata.py", line 42, in handler
     new_version = version_creator.update_metadata(new_metadata)   File "/var/task/versioning.py", line 98, in update_metadata
     state = metadata_update_type(metadata)   File "/var/task/versioning.py", line 135, in metadata_update_type
     changed_fields = data_product_metadata.changed_fields()   File "/var/task/data_product_metadata.py", line 245, in changed_fields
     if new_version[field] == latest_version[field]:
-- | --

This logic assumes the set of keys is always the same when updating the metadata, but this is not the case, since

  1. the metadata schema can evolve
  2. the user may add or remove fields that are optional in the schema

To Reproduce

  1. Create a data product with minimal required fields
  2. Update the data product to add a new field

Expected Behaviour

The metadata should be updated successfully and a new minor version created

Additional context

No response

@MatMoore MatMoore added bug Something isn't working data-platform-labs labels Nov 13, 2023
@tom-webber tom-webber self-assigned this Nov 21, 2023
@tom-webber tom-webber moved this from 🧐 To Do to 💨 In Progress in Analytical Platform Nov 21, 2023
@tom-webber tom-webber moved this from 💨 In Progress to 👀 In Review in Analytical Platform Nov 22, 2023
@github-project-automation github-project-automation bot moved this from 👀 In Review to ✨ Done ✨ in Analytical Platform Nov 23, 2023
jhpyke pushed a commit that referenced this issue Jan 8, 2024
* fix KeyError if new field added to metadata as part of update
fixes #2325
Update tests to allow testing multiple metadata objects
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants