fix: fix PySparkProcessor init params type #3354

andre-marcos-perez · 2022-09-08T08:15:37Z

Issue #, if available:

Description of changes: In the sagemaker.spark.processing module, the PySparkProcessor object constructor params instance_type and instance_count params have wrong typing:

instance_type is a str but is being typed as int;
instance_count is an int but is being typed as str.

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2022-09-23T22:48:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: e1642d9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-09-23T23:04:15Z

Codecov Report

Merging #3354 (37d503b) into master (1fa2377) will decrease coverage by 0.20%.
The diff coverage is 88.30%.

@@            Coverage Diff             @@
##           master    #3354      +/-   ##
==========================================
- Coverage   89.17%   88.96%   -0.21%     
==========================================
  Files         204      212       +8     
  Lines       18979    20435    +1456     
==========================================
+ Hits        16924    18180    +1256     
- Misses       2055     2255     +200

Impacted Files	Coverage Δ
src/sagemaker/feature_store/feature_group.py	`76.52% <ø> (ø)`
src/sagemaker/huggingface/model.py	`89.87% <ø> (ø)`
src/sagemaker/pipeline.py	`85.33% <ø> (ø)`
src/sagemaker/sklearn/model.py	`92.98% <ø> (-1.76%)`	⬇️
...sagemaker/workflow/monitor_batch_transform_step.py	`0.00% <0.00%> (ø)`
src/sagemaker/xgboost/model.py	`96.36% <ø> (-1.82%)`	⬇️
...agemaker/model_monitor/clarify_model_monitoring.py	`89.84% <27.58%> (-5.09%)`	⬇️
src/sagemaker/training_compiler/config.py	`80.43% <60.00%> (-2.90%)`	⬇️
src/sagemaker/pytorch/estimator.py	`94.73% <63.63%> (-5.27%)`	⬇️
src/sagemaker/sklearn/estimator.py	`93.93% <63.63%> (-6.07%)`	⬇️
... and 49 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

sagemaker-bot · 2022-09-23T23:04:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: e1642d9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-09-23T23:21:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: e1642d9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-09-23T23:27:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: e1642d9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-09-24T00:01:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: e1642d9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-09-29T00:07:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 30d0d3b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-09-29T00:29:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 30d0d3b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-09-29T00:44:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 30d0d3b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-09-29T00:54:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 30d0d3b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-09-29T01:16:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 30d0d3b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

vikas0203

/bot run all

sagemaker-bot · 2022-10-19T20:46:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: de02cde
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-10-19T21:11:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: de02cde
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-10-19T21:27:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: de02cde
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-10-19T21:46:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: de02cde
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-10-20T00:25:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: de02cde
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala

/bot run all

sagemaker-bot · 2022-11-24T01:11:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 0cc38db
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-02T02:56:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 37d503b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-02T04:54:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 37d503b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala

/bot run slow-tests

sagemaker-bot · 2022-12-02T20:23:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 37d503b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

This reverts commit de58941.

* fix: type hint of PySparkProcessor __init__ (#3297) * fix: fix PySparkProcessor __init__ params type (#3354) * fix: Revert "fix: type hint of PySparkProcessor __init__" (#3524) * fix: type hint of PySparkProcessor __init__ (#3297) * fix: fix PySparkProcessor __init__ params type (#3354) * fix: Revert "fix: type hint of PySparkProcessor __init__" (#3524) * feature: AutoGluon 0.6.1 image_uris * feature: AutoGluon 0.6.1 image_uris * Added processors for training images for consistency * Formatting * Formatting Co-authored-by: Kevin <[email protected]> Co-authored-by: André Perez <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]>

* fix: type hint of PySparkProcessor __init__ (aws#3297) * fix: fix PySparkProcessor __init__ params type (aws#3354) * fix: Revert "fix: type hint of PySparkProcessor __init__" (aws#3524) * fix: type hint of PySparkProcessor __init__ (aws#3297) * fix: fix PySparkProcessor __init__ params type (aws#3354) * fix: Revert "fix: type hint of PySparkProcessor __init__" (aws#3524) * feature: AutoGluon 0.6.1 image_uris * feature: AutoGluon 0.6.1 image_uris * Added processors for training images for consistency * Formatting * Formatting Co-authored-by: Kevin <[email protected]> Co-authored-by: André Perez <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]>

* fix: type hint of PySparkProcessor __init__ (#3297) * fix: fix PySparkProcessor __init__ params type (#3354) * fix: Revert "fix: type hint of PySparkProcessor __init__" (#3524) * fix: type hint of PySparkProcessor __init__ (#3297) * fix: fix PySparkProcessor __init__ params type (#3354) * fix: Revert "fix: type hint of PySparkProcessor __init__" (#3524) * feature: AutoGluon 0.6.1 image_uris * feature: AutoGluon 0.6.1 image_uris * Added processors for training images for consistency * Formatting * Formatting Co-authored-by: Kevin <[email protected]> Co-authored-by: André Perez <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]>

* feature: Add create inf rec api to session * save * fix error handling in submit. update docstring * add in job_name param * fix: type hint of PySparkProcessor __init__ (aws#3297) * fix: fix PySparkProcessor __init__ params type (aws#3354) * feature: add describe inf rec api to session * feature: add describe inf rec api to session * feature: add describe inf rec api to session * add verbose logging in wait_for_inference_recommendations_job * improve unit tests * switch to cloudwatch logs * add unit tests * fix unit test * make default logging level to Verbose * address comments in PR * address some comments * fix typo missing s * add initial print because it took a while for cw logstream to appear * print and conditional optimizations and update tests * optimize log print * fix polling for logs empty events * cover empty event case * add spacing Co-authored-by: Gary Wang <[email protected]> Co-authored-by: Kevin <[email protected]> Co-authored-by: André Perez <[email protected]> Co-authored-by: Raymond Liu <[email protected]>

* fix: type hint of PySparkProcessor __init__ (aws#3297) * fix: fix PySparkProcessor __init__ params type (aws#3354) * fix: Revert "fix: type hint of PySparkProcessor __init__" (aws#3524) * fix: type hint of PySparkProcessor __init__ (aws#3297) * fix: fix PySparkProcessor __init__ params type (aws#3354) * fix: Revert "fix: type hint of PySparkProcessor __init__" (aws#3524) * feature: AutoGluon 0.6.1 image_uris * feature: AutoGluon 0.6.1 image_uris * Added processors for training images for consistency * Formatting * Formatting Co-authored-by: Kevin <[email protected]> Co-authored-by: André Perez <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]>

andre-marcos-perez changed the title ~~fix: fix PySparkPySparkProcessor __init__ params type~~ fix: fix PySparkProcessor __init__ params type Sep 16, 2022

claytonparnell force-pushed the master branch from 64eee1d to de02cde Compare October 19, 2022 17:13

vikas0203 reviewed Oct 19, 2022

View reviewed changes

vikas0203 approved these changes Oct 25, 2022

View reviewed changes

vikas0203 requested review from a team, mufaddal-rohawala and knikure and removed request for a team November 23, 2022 20:42

vikas0203 assigned mufaddal-rohawala Nov 23, 2022

mufaddal-rohawala approved these changes Nov 24, 2022

View reviewed changes

mufaddal-rohawala reviewed Dec 2, 2022

View reviewed changes

mufaddal-rohawala merged commit de58941 into aws:master Dec 2, 2022

mufaddal-rohawala added a commit that referenced this pull request Dec 9, 2022

Revert "fix: fix PySparkProcessor __init__ params type (#3354)"

19ae8a6

This reverts commit de58941.

mufaddal-rohawala mentioned this pull request Dec 9, 2022

Revert "fix: fix PySparkProcessor __init__ params type" #3525

Closed

mizanfiu pushed a commit to mizanfiu/sagemaker-python-sdk that referenced this pull request Dec 13, 2022

fix: fix PySparkProcessor __init__ params type (aws#3354)

942361b

gradientsky pushed a commit to gradientsky/sagemaker-python-sdk that referenced this pull request Dec 20, 2022

fix: fix PySparkProcessor __init__ params type (aws#3354)

17af232

gradientsky pushed a commit to gradientsky/sagemaker-python-sdk that referenced this pull request Dec 20, 2022

fix: fix PySparkProcessor __init__ params type (aws#3354)

ff5268e

hballuru pushed a commit to hballuru/sagemaker-python-sdk that referenced this pull request Dec 21, 2022

fix: fix PySparkProcessor __init__ params type (aws#3354)

704d1d5

Shegufta pushed a commit to Shegufta/sagemaker-python-sdk that referenced this pull request Dec 21, 2022

fix: fix PySparkProcessor __init__ params type (aws#3354)

a965c4c

jmahlik pushed a commit to jmahlik/sagemaker-python-sdk that referenced this pull request Dec 22, 2022

fix: fix PySparkProcessor __init__ params type (aws#3354)

7249638

wcarpenter1-godaddy pushed a commit to wcarpenter1-godaddy/sagemaker-python-sdk that referenced this pull request Feb 8, 2023

fix: fix PySparkProcessor __init__ params type (aws#3354)

2cc4d47

JoseJuan98 pushed a commit to JoseJuan98/sagemaker-python-sdk that referenced this pull request Mar 4, 2023

fix: fix PySparkProcessor __init__ params type (aws#3354)

b5d0042

tejaschumbalkar pushed a commit to tejaschumbalkar/sagemaker-python-sdk that referenced this pull request Mar 17, 2023

fix: fix PySparkProcessor __init__ params type (aws#3354)

b760181

tejaschumbalkar pushed a commit to tejaschumbalkar/sagemaker-python-sdk that referenced this pull request Mar 20, 2023

fix: fix PySparkProcessor __init__ params type (aws#3354)

1f4f396

nmadan pushed a commit to nmadan/sagemaker-python-sdk that referenced this pull request Apr 18, 2023

fix: fix PySparkProcessor __init__ params type (aws#3354)

36e5fb1

nmadan pushed a commit to nmadan/sagemaker-python-sdk that referenced this pull request Apr 18, 2023

fix: fix PySparkProcessor __init__ params type (aws#3354)

c651b22

nmadan pushed a commit to nmadan/sagemaker-python-sdk that referenced this pull request Apr 18, 2023

fix: fix PySparkProcessor __init__ params type (aws#3354)

1fe7a0c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: fix PySparkProcessor init params type #3354

fix: fix PySparkProcessor init params type #3354

andre-marcos-perez commented Sep 8, 2022 •

edited

Loading

sagemaker-bot commented Sep 23, 2022

codecov-commenter commented Sep 23, 2022 •

edited

Loading

sagemaker-bot commented Sep 23, 2022

sagemaker-bot commented Sep 23, 2022

sagemaker-bot commented Sep 23, 2022

sagemaker-bot commented Sep 24, 2022

sagemaker-bot commented Sep 29, 2022

sagemaker-bot commented Sep 29, 2022

sagemaker-bot commented Sep 29, 2022

sagemaker-bot commented Sep 29, 2022

sagemaker-bot commented Sep 29, 2022

vikas0203 left a comment

sagemaker-bot commented Oct 19, 2022

sagemaker-bot commented Oct 19, 2022

sagemaker-bot commented Oct 19, 2022

sagemaker-bot commented Oct 19, 2022

sagemaker-bot commented Oct 20, 2022

mufaddal-rohawala left a comment

sagemaker-bot commented Nov 24, 2022

sagemaker-bot commented Dec 2, 2022

sagemaker-bot commented Dec 2, 2022

mufaddal-rohawala left a comment

sagemaker-bot commented Dec 2, 2022

fix: fix PySparkProcessor __init__ params type #3354

fix: fix PySparkProcessor __init__ params type #3354

Conversation

andre-marcos-perez commented Sep 8, 2022 • edited Loading

Merge Checklist

General

Tests

sagemaker-bot commented Sep 23, 2022

AWS CodeBuild CI Report

codecov-commenter commented Sep 23, 2022 • edited Loading

Codecov Report

sagemaker-bot commented Sep 23, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Sep 23, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Sep 23, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Sep 24, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Sep 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Sep 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Sep 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Sep 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Sep 29, 2022

AWS CodeBuild CI Report

vikas0203 left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Oct 19, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Oct 19, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Oct 19, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Oct 19, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Oct 20, 2022

AWS CodeBuild CI Report

mufaddal-rohawala left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Nov 24, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 2, 2022

AWS CodeBuild CI Report

mufaddal-rohawala left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Dec 2, 2022

AWS CodeBuild CI Report

fix: fix PySparkProcessor init params type #3354

fix: fix PySparkProcessor init params type #3354

andre-marcos-perez commented Sep 8, 2022 •

edited

Loading

codecov-commenter commented Sep 23, 2022 •

edited

Loading