fix(ingestor-api)!: store STAC item as string in DynamoDB #26
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sharkinsspatial
force-pushed
the
fix/item_serialization
branch
from
a3246d4 to
afc3655
Compare
sharkinsspatial
force-pushed
the
fix/item_serialization
branch
from
afc3655 to
caee41a
Compare
sharkinsspatial
changed the title
alukach
force-pushed
the
fix/item_serialization
branch
from
4ba3bdd to
bb864e0
Compare
alukach
force-pushed
the
fix/item_serialization
branch
from
bb864e0 to
ec31203
Compare
alukach
approved these changes
Mar 27, 2023
There was a problem hiding this comment.
Looks great @sharkinsspatial! Thanks for adding those tests. I added a few more (once some are written, it makes it so much easier to expand).
I rewrote a bit of the code to make use of fastapi.encoders.jsonable_encoder() which I think makes for slightly cleaner code.
Feeling really good about this change!
Comment on lines
76
to
88
| class Config: | ||
| json_encoders = { | ||
| # Custom JSON serializer to ensure that item encodes as string. | ||
| # NOTE: when serializing, must call as ingestion.json(models_as_dict=False) | ||
| Item: lambda item: item.json(by_alias=True), | ||
| } | ||
|
|
||
| def json(self, *args, **kwargs): | ||
| # Update default to not represent models (e.g. `items` property) as a dict to | ||
| # allow our `json_encoders` override to properly serialize `items` property | ||
| kwargs.setdefault("models_as_dict", False) | ||
| return super().json(*args, **kwargs) | ||
|
|
There was a problem hiding this comment.
@sharkinsspatial My rationale here is that if we need to put all this cruft in to make special considerations for serializing our Ingestion object for DynamoDB, it would probably make for cleaner code if we just put that cruft into the dynamodb_dict() method.
alukach
changed the title
alukach
changed the title
alukach
changed the title
alukach
changed the title
alukach
changed the title
alukach
added a commit
that referenced
this pull request
Mar 28, 2023
* fix: stac item serialization in ingestion model * fix: use pydantic parsing and remove float conversion * test: add initial test for missing ingestor coverage * fix: fix updated psql function name changed in db bootstrapper * style: fix black errors * test: add utils.load_items test for new item serialization * style: black and isort style changes * Make use of jsonable_encoder, buildout tests for submitting STAC items --------- Co-authored-by: Anthony Lukach <[email protected]>
alukach
added a commit
that referenced
this pull request
Mar 28, 2023
* fix: stac item serialization in ingestion model * fix: use pydantic parsing and remove float conversion * test: add initial test for missing ingestor coverage * fix: fix updated psql function name changed in db bootstrapper * style: fix black errors * test: add utils.load_items test for new item serialization * style: black and isort style changes * Make use of jsonable_encoder, buildout tests for submitting STAC items --------- Co-authored-by: Anthony Lukach <[email protected]>
github-actions bot
pushed a commit
that referenced
this pull request
Mar 28, 2023
## [2.6.3](v2.6.2...v2.6.3) (2023-03-28) ### Bug Fixes * **ingestor-api:** store STAC item as string in DynamoDB ([#26](#26)) ([bd7a1fa](bd7a1fa))
6 tasks
anayeaye
added a commit
to NASA-IMPACT/veda-backend
that referenced
this pull request
Feb 26, 2024
# What Fixes the float epsg code issue caused by dynamodb serialization Picked up the fix from developmentseed/eoapi-cdk#26 # How test Tested by running ingest from here: https://mv5tl66nca.execute-api.us-west-2.amazonaws.com/ Test item: https://vr3ofct5u8.execute-api.us-west-2.amazonaws.com/api/stac/collections/geoglam/items/CropMonitor_202205_test_2 ```python import pystac item = pystac.Item.from_file("https://vr3ofct5u8.execute-api.us-west-2.amazonaws.com/api/stac/collections/geoglam/items/CropMonitor_202205_test_2") print(item.properties["proj:epsg"]) # should give 4326 instead of 4326.0 4326 ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses the issue raised in NASA-IMPACT/veda-data#57. The PR updates the ingestor code to serialize the STAC
itemproperty of theingestionas a string prior to storing it in DynamoDB. Additionally, it updates the code used to deserializeingestionsqueried from the DynamoDB table and load them intopgstacand adds unit test coverage for some of theingestorcode paths (which previously had no coverage).