Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pickle dictionary when it isn't JSON serializable #2390

Merged
merged 16 commits into from
May 9, 2024
Merged

Conversation

samhita-alla
Copy link
Contributor

@samhita-alla samhita-alla commented May 6, 2024

Tracking issue

Why are the changes needed?

This is useful for serializing dictionaries that aren't JSON serializable. To enable pickling the data, the dict type needs to be annotated with allow_pickle set to True:

@task
def t1() -> Annotated[dict, kwtypes(allow_pickle=True)]
    ...

Pickling only happens if the dict isn't JSON serializable.

What changes were proposed in this pull request?

Modify DictTransformer to add support for pickling dictionaries and include relevant test cases. I've also updated the BotoTask output type to allow pickling when the dict isn't JSON serializable.

How was this patch tested?

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Docs link

Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
@samhita-alla samhita-alla marked this pull request as ready for review May 8, 2024 05:06
Signed-off-by: Samhita Alla <[email protected]>
Copy link

codecov bot commented May 8, 2024

Codecov Report

Attention: Patch coverage is 28.00000% with 36 lines in your changes are missing coverage. Please review.

Project coverage is 50.25%. Comparing base (afa1fc7) to head (a1bde7e).
Report is 15 commits behind head on master.

❗ Current head a1bde7e differs from pull request most recent head 405827d. Consider uploading reports for the commit 405827d to get more accurate results

Files Patch % Lines
flytekit/core/type_engine.py 29.16% 30 Missing and 4 partials ⚠️
flytekit/core/promise.py 0.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           master    #2390       +/-   ##
===========================================
- Coverage   75.79%   50.25%   -25.54%     
===========================================
  Files         181      181               
  Lines       18275    18393      +118     
  Branches     2567     3601     +1034     
===========================================
- Hits        13852     9244     -4608     
- Misses       3823     8714     +4891     
+ Partials      600      435      -165     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Samhita Alla <[email protected]>
flytekit/core/type_engine.py Outdated Show resolved Hide resolved
flytekit/core/type_engine.py Outdated Show resolved Hide resolved
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
@samhita-alla samhita-alla merged commit e6e08f9 into master May 9, 2024
47 checks passed
austin362667 pushed a commit to austin362667/flytekit that referenced this pull request May 10, 2024
* pickle dict when it isn't JSON serializable

Signed-off-by: Samhita Alla <[email protected]>

* lint

Signed-off-by: Samhita Alla <[email protected]>

* fix circular import and add dick pickling to boto agent

Signed-off-by: Samhita Alla <[email protected]>

* make dict optional

Signed-off-by: Samhita Alla <[email protected]>

* import pickle

Signed-off-by: Samhita Alla <[email protected]>

* incorporate suggestion by @pingsutw to fix metadata passage to to_python_value method

Signed-off-by: Samhita Alla <[email protected]>

* lint

Signed-off-by: Samhita Alla <[email protected]>

* lint

Signed-off-by: Samhita Alla <[email protected]>

* replace literal with literalmap

Signed-off-by: Samhita Alla <[email protected]>

* fix test

Signed-off-by: Samhita Alla <[email protected]>

* fix boto test

Signed-off-by: Samhita Alla <[email protected]>

* incorporate @pingsutw's suggestions

Signed-off-by: Samhita Alla <[email protected]>

* revert outputs and fix lint

Signed-off-by: Samhita Alla <[email protected]>

* lint

Signed-off-by: Samhita Alla <[email protected]>

* lint

Signed-off-by: Samhita Alla <[email protected]>

* update boto agent test

Signed-off-by: Samhita Alla <[email protected]>

---------

Signed-off-by: Samhita Alla <[email protected]>
fiedlerNr9 pushed a commit that referenced this pull request Jul 25, 2024
* pickle dict when it isn't JSON serializable

Signed-off-by: Samhita Alla <[email protected]>

* lint

Signed-off-by: Samhita Alla <[email protected]>

* fix circular import and add dick pickling to boto agent

Signed-off-by: Samhita Alla <[email protected]>

* make dict optional

Signed-off-by: Samhita Alla <[email protected]>

* import pickle

Signed-off-by: Samhita Alla <[email protected]>

* incorporate suggestion by @pingsutw to fix metadata passage to to_python_value method

Signed-off-by: Samhita Alla <[email protected]>

* lint

Signed-off-by: Samhita Alla <[email protected]>

* lint

Signed-off-by: Samhita Alla <[email protected]>

* replace literal with literalmap

Signed-off-by: Samhita Alla <[email protected]>

* fix test

Signed-off-by: Samhita Alla <[email protected]>

* fix boto test

Signed-off-by: Samhita Alla <[email protected]>

* incorporate @pingsutw's suggestions

Signed-off-by: Samhita Alla <[email protected]>

* revert outputs and fix lint

Signed-off-by: Samhita Alla <[email protected]>

* lint

Signed-off-by: Samhita Alla <[email protected]>

* lint

Signed-off-by: Samhita Alla <[email protected]>

* update boto agent test

Signed-off-by: Samhita Alla <[email protected]>

---------

Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Jan Fiedler <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants