Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SNOW-1417060 Pylance throws errors when using copy_options with copy_into_location method #1828

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

SNOW-1417060 Pylance throws errors when using copy_options with copy_into_location method #1828

wants to merge 2 commits into from

Conversation

Yoshi-Egawa
Copy link
Contributor

  1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

    Fixes SNOW-1417060

  2. Fill out the following pre-review checklist:

    • I am adding a new automated test(s) to verify correctness of my new code
      • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
    • I am adding new logging messages
    • I am adding a new telemetry message
    • I am adding new credentials
    • I am adding a new dependency
    • If this is a new feature/behavior, I'm adding the Local Testing parity changes.

  3. Please describe how your code solves the related issue.

    • Enhanced Type Safety with TypedDict: We've replaced the generic Dict[str, Any] for the copy_options argument with a TypedDict. This enforces stricter type checking, ensuring that only expected options are passed to the function. This improves code reliability and maintainability.
    • Overload Method for Pylance Recognition: An additional overload method has been added specifically for the block option when its type is bool. This improves code clarity and type recognition within development tools like Pylance, making the code easier to understand and navigate.

@Yoshi-Egawa Yoshi-Egawa requested a review from a team as a code owner June 25, 2024 14:39
Yoshi-Egawa
Yoshi-Egawa commented Jun 25, 2024
View reviewed changes
src/snowflake/snowpark/_internal/type_utils.py
@@ -25,6 +25,7 @@
Optional,
Tuple,
Type,
TypedDict,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TypedDict and total arguments are supported starting with Python 3.8.

https://docs.python.org/3.8/library/typing.html?highlight=typeddict#typing.TypedDict

src/snowflake/snowpark/dataframe_writer.py Outdated
Comment on lines 273 to 332
@overload
def copy_into_location(
self,
location: str,
*,
partition_by: Optional[ColumnOrSqlExpr] = None,
file_format_name: Optional[str] = None,
file_format_type: Optional[str] = None,
format_type_options: Optional[Dict[str, str]] = None,
header: bool = False,
statement_params: Optional[Dict[str, str]] = None,
block: bool = True,
**copy_options: Unpack[CopyOptions],
) -> Union[List[Row], AsyncJob]:
... # pragma: no cover

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The type hint of a function definition without overload did not work with csv method or json method, and a pylancen error occurred, so I also defined it with overload.

@Yoshi-Egawa Yoshi-Egawa changed the title Pylance throws errors when using copy_options with copy_into_location method SNOW-1417060 Pylance throws errors when using copy_options with copy_into_location method Jun 27, 2024
@Yoshi-Egawa Yoshi-Egawa force-pushed the fix/copy_options-type branch from 4431afc to 3c6b852 Compare October 4, 2024 15:32
@Yoshi-Egawa Yoshi-Egawa mentioned this pull request Oct 4, 2024
Closed
@Yoshi-Egawa
Copy link
Contributor Author

@sfc-gh-yuwang, @sfc-gh-jrose, @sfc-gh-aalam
Hello everyone,

I hope you're doing well. I wanted to kindly check in on the status of the pull request. I know you must be busy, but if you have a moment to take a look, I’d really appreciate any feedback or comments you might have.

sfc-gh-aling
sfc-gh-aling reviewed Oct 10, 2024
View reviewed changes
src/snowflake/snowpark/_internal/type_utils.py Outdated
Comment on lines 993 to 1249
overwrite: bool
single: bool
max_file_size: float
include_query_id: bool
detailed_output: bool
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think value in string form is also allowed?
also are the keys case sensitive?

if I have a copy options looking like

option_dict = {"OVERWRITE": "TRUE"}

would pylance complain?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sfc-gh-aling
I missed checking this point.
Keys are not case-sensitive.
I have corrected this on my end. (575cb81)

Current state:
image

Desired state:
image

@Yoshi-Egawa Yoshi-Egawa force-pushed the fix/copy_options-type branch from 3c6b852 to bacefbe Compare January 24, 2025 23:59
@Yoshi-Egawa Yoshi-Egawa requested a review from a team as a code owner January 24, 2025 23:59
@Yoshi-Egawa Yoshi-Egawa force-pushed the fix/copy_options-type branch from bacefbe to 575cb81 Compare January 25, 2025 02:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sfc-gh-aling sfc-gh-aling sfc-gh-aling left review comments

@sfc-gh-yuwang sfc-gh-yuwang Awaiting requested review from sfc-gh-yuwang sfc-gh-yuwang is a code owner automatically assigned from snowflakedb/snowpark-python-api-reviewers

@sfc-gh-aalam sfc-gh-aalam Awaiting requested review from sfc-gh-aalam sfc-gh-aalam is a code owner automatically assigned from snowflakedb/snowpark-python-api-reviewers

@sfc-gh-jrose sfc-gh-jrose Awaiting requested review from sfc-gh-jrose sfc-gh-jrose is a code owner automatically assigned from snowflakedb/snowpark-python-api-reviewers

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Yoshi-Egawa @sfc-gh-aling