-
Notifications
You must be signed in to change notification settings - Fork 301
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pickle dictionary when it isn't JSON serializable #2390
Conversation
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
…hon_value method Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #2390 +/- ##
===========================================
- Coverage 75.79% 50.25% -25.54%
===========================================
Files 181 181
Lines 18275 18393 +118
Branches 2567 3601 +1034
===========================================
- Hits 13852 9244 -4608
- Misses 3823 8714 +4891
+ Partials 600 435 -165 ☔ View full report in Codecov by Sentry. |
Signed-off-by: Samhita Alla <[email protected]>
plugins/flytekit-aws-sagemaker/flytekitplugins/awssagemaker_inference/boto3_agent.py
Show resolved
Hide resolved
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
* pickle dict when it isn't JSON serializable Signed-off-by: Samhita Alla <[email protected]> * lint Signed-off-by: Samhita Alla <[email protected]> * fix circular import and add dick pickling to boto agent Signed-off-by: Samhita Alla <[email protected]> * make dict optional Signed-off-by: Samhita Alla <[email protected]> * import pickle Signed-off-by: Samhita Alla <[email protected]> * incorporate suggestion by @pingsutw to fix metadata passage to to_python_value method Signed-off-by: Samhita Alla <[email protected]> * lint Signed-off-by: Samhita Alla <[email protected]> * lint Signed-off-by: Samhita Alla <[email protected]> * replace literal with literalmap Signed-off-by: Samhita Alla <[email protected]> * fix test Signed-off-by: Samhita Alla <[email protected]> * fix boto test Signed-off-by: Samhita Alla <[email protected]> * incorporate @pingsutw's suggestions Signed-off-by: Samhita Alla <[email protected]> * revert outputs and fix lint Signed-off-by: Samhita Alla <[email protected]> * lint Signed-off-by: Samhita Alla <[email protected]> * lint Signed-off-by: Samhita Alla <[email protected]> * update boto agent test Signed-off-by: Samhita Alla <[email protected]> --------- Signed-off-by: Samhita Alla <[email protected]>
* pickle dict when it isn't JSON serializable Signed-off-by: Samhita Alla <[email protected]> * lint Signed-off-by: Samhita Alla <[email protected]> * fix circular import and add dick pickling to boto agent Signed-off-by: Samhita Alla <[email protected]> * make dict optional Signed-off-by: Samhita Alla <[email protected]> * import pickle Signed-off-by: Samhita Alla <[email protected]> * incorporate suggestion by @pingsutw to fix metadata passage to to_python_value method Signed-off-by: Samhita Alla <[email protected]> * lint Signed-off-by: Samhita Alla <[email protected]> * lint Signed-off-by: Samhita Alla <[email protected]> * replace literal with literalmap Signed-off-by: Samhita Alla <[email protected]> * fix test Signed-off-by: Samhita Alla <[email protected]> * fix boto test Signed-off-by: Samhita Alla <[email protected]> * incorporate @pingsutw's suggestions Signed-off-by: Samhita Alla <[email protected]> * revert outputs and fix lint Signed-off-by: Samhita Alla <[email protected]> * lint Signed-off-by: Samhita Alla <[email protected]> * lint Signed-off-by: Samhita Alla <[email protected]> * update boto agent test Signed-off-by: Samhita Alla <[email protected]> --------- Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Jan Fiedler <[email protected]>
Tracking issue
Why are the changes needed?
This is useful for serializing dictionaries that aren't JSON serializable. To enable pickling the data, the
dict
type needs to be annotated withallow_pickle
set toTrue
:Pickling only happens if the dict isn't JSON serializable.
What changes were proposed in this pull request?
Modify
DictTransformer
to add support for pickling dictionaries and include relevant test cases. I've also updated theBotoTask
output type to allow pickling when the dict isn't JSON serializable.How was this patch tested?
Setup process
Screenshots
Check all the applicable boxes
Related PRs
Docs link