Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support s3 path for click Dir param type #2547

Merged
merged 2 commits into from
Jul 2, 2024

Conversation

Future-Outlier
Copy link
Member

@Future-Outlier Future-Outlier commented Jul 2, 2024

Why are the changes needed?

We only support local directory for click Dir param type, now we want to support remote s3 path also.

What changes were proposed in this pull request?

Use FileAccessProvider.is_remote to check if the path is remote or not.

How was this patch tested?

remote execution and unit test.

Setup process

import os

from flytekit import task, workflow
from flytekit.types.directory import FlyteDirectory
from flytekit.types.file import FlyteFile
from click.testing import CliRunner
from flytekit.clis.sdk_in_container import pyflyte


@task
def list_dir(d: FlyteDirectory):
    for fname in os.listdir(d):
        print(f"Found file {fname}")


@workflow
def dir_wf(d: FlyteDirectory):
    list_dir(d=d)

@task
def list_file(f: FlyteFile):
    print(f)
    with open(f, "r") as wf:
        print(wf.read())

@workflow
def file_wf(f: FlyteFile):
    list_file(f=f)


if __name__ == "__main__":
    runner = CliRunner()
    result = runner.invoke(pyflyte.main, ["run", "--remote", 
                                          "PR/s3-path-command-line/test.py", "dir_wf", "--d", "/Users/future-outlier/code/dev/PR/s3-path-command-line"])
    print(result.output)

    result = runner.invoke(pyflyte.main, ["run", "--remote", 
                                          "/Users/future-outlier/code/dev/PR/s3-path-command-line/test.py", "dir_wf", "--d", "s3://my-s3-bucket/a"])
    print(result.output)

Screenshots

Terminal screenshots

(DEV) future@outlier ~ % python PR/s3-path-command-line/test.py
Running Execution on Remote.

[✔] Go to http://localhost:30080/console/projects/flytesnacks/domains/development/executions/f45bc46a9b00c47239e8 to see execution in the console.

Running Execution on Remote.

[✔] Go to http://localhost:30080/console/projects/flytesnacks/domains/development/executions/f223ab0832cac4f058bd to see execution in the console.

use local dir

image #### use remote dir image

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

@Future-Outlier Future-Outlier changed the title Support s3 path for Dir param type Support s3 path for click Dir param type Jul 2, 2024
Copy link

codecov bot commented Jul 2, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.45%. Comparing base (2719b34) to head (d0ae5a6).
Report is 3 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #2547       +/-   ##
===========================================
+ Coverage   77.65%   90.45%   +12.79%     
===========================================
  Files         236       49      -187     
  Lines       20789     1917    -18872     
  Branches     3661        0     -3661     
===========================================
- Hits        16144     1734    -14410     
+ Misses       4017      183     -3834     
+ Partials      628        0      -628     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@wild-endeavor wild-endeavor merged commit 825a50e into master Jul 2, 2024
48 checks passed
bgedik pushed a commit to bgedik/flytekit that referenced this pull request Jul 3, 2024
Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: bugra.gedik <[email protected]>
fiedlerNr9 pushed a commit that referenced this pull request Jul 25, 2024
Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Jan Fiedler <[email protected]>
mao3267 pushed a commit to mao3267/flytekit that referenced this pull request Jul 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants