Only transform the literal that matches the map task index #2484
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Haytham Abuelfutuh <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>
…/flytekit into array-node-input-transformation
Signed-off-by: Haytham Abuelfutuh <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
…-input-transformation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #2484 +/- ##
===========================================
- Coverage 73.97% 50.98% -23.00%
===========================================
Files 250 185 -65
Lines 21592 18677 -2915
Branches 3644 3645 +1
===========================================
- Hits 15973 9522 -6451
- Misses 4973 8674 +3701
+ Partials 646 481 -165 ☔ View full report in Codecov by Sentry. |
EngHabu
requested review from
wild-endeavor,
kumare3,
eapolinario,
pingsutw,
cosmicBboy and
samhita-alla
as code owners
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
pingsutw
approved these changes
Jun 17, 2024
bgedik
pushed a commit
to bgedik/flytekit
that referenced
this pull request
Jul 3, 2024
…2484) * Only transform the literal that matches the map task index Signed-off-by: Haytham Abuelfutuh <[email protected]> * kevin's update Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * make it work locally Signed-off-by: Haytham Abuelfutuh <[email protected]> * lint Signed-off-by: Haytham Abuelfutuh <[email protected]> * add a test Signed-off-by: Kevin Su <[email protected]> * check (flyteorg#2486) Signed-off-by: Kevin Su <[email protected]> * Only fail for unbound batch pickled inputs for map tasks Signed-off-by: Haytham Abuelfutuh <[email protected]> * Unit tests Signed-off-by: Haytham Abuelfutuh <[email protected]> * lint Signed-off-by: Haytham Abuelfutuh <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Haytham Abuelfutuh <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: bugra.gedik <[email protected]>
fiedlerNr9
pushed a commit
that referenced
this pull request
Jul 25, 2024
* Only transform the literal that matches the map task index Signed-off-by: Haytham Abuelfutuh <[email protected]> * kevin's update Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * make it work locally Signed-off-by: Haytham Abuelfutuh <[email protected]> * lint Signed-off-by: Haytham Abuelfutuh <[email protected]> * add a test Signed-off-by: Kevin Su <[email protected]> * check (#2486) Signed-off-by: Kevin Su <[email protected]> * Only fail for unbound batch pickled inputs for map tasks Signed-off-by: Haytham Abuelfutuh <[email protected]> * Unit tests Signed-off-by: Haytham Abuelfutuh <[email protected]> * lint Signed-off-by: Haytham Abuelfutuh <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Haytham Abuelfutuh <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jan Fiedler <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tracking issue
fixes flyteorg/flyte#5483
Why are the changes needed?
Existing code only trims map task inputs before invoking the python subtask. This is incorrect as it leads to flytekit unnecessarily downloading all inputs for the entire maptask (potentially 10s of thousands) for each subtask.
What changes were proposed in this pull request?
For local execution, continue with the current behavior of handling all subtasks inputs when the map task is invoked.
For remote execution, index into the inputs list using the maptask index and only transform this input.
How was this patch tested?
Setup process
Screenshots
Check all the applicable boxes