Dynamic local execution

[wild-endeavor]

TL;DR

Local execution for dynamic workflows never correctly mimicked a production run. Because of the execution context that was set, tasks would return native Python values instead of promises, leading to issues like the >> operator not working.

This limitation was also introducing runtime bugs in user code because users would use the output of tasks in ways that required them to be native Python values, though this issue was partially mitigated in #1121 by forcing a compilation when run locally.

This bug is also holding up implementation of the human-in-the-loop work. And also makes it so that you can't set node level overrides if you want to run locally.

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

Dynamic tasks are supposed to mimic workflows, so we are taking some code from the workflow local_execute pattern.

Tracking Issue

flyteorg/flyte#3093

[codecov]

Codecov Report

Merging #1343 (a8d2392) into master (dd3fcae) will decrease coverage by 0.03%.
The diff coverage is 85.00%.

@@            Coverage Diff             @@
##           master    #1343      +/-   ##
==========================================
- Coverage   68.85%   68.82%   -0.04%     
==========================================
  Files         287      287              
  Lines       26453    26530      +77     
  Branches     2497     2506       +9     
==========================================
+ Hits        18215    18260      +45     
- Misses       7750     7782      +32     
  Partials      488      488              

Impacted Files	Coverage Δ
flytekit/core/base_task.py	46.88% <ø> (ø)
flytekit/core/context_manager.py	39.61% <ø> (ø)
flytekit/core/python_function_task.py	53.27% <70.83%> (+0.63%)	⬆️
tests/flytekit/unit/core/test_dynamic.py	94.21% <91.07%> (+0.20%)	⬆️
tests/flytekit/unit/cli/pyflyte/workflow.py	86.95% <0.00%> (-13.05%)	⬇️
flytekit/clis/sdk_in_container/run.py	79.86% <0.00%> (-4.30%)	⬇️
flytekit/configuration/__init__.py	36.87% <0.00%> (-0.67%)	⬇️
flytekit/models/literals.py	40.28% <0.00%> (-0.58%)	⬇️
tests/flytekit/unit/core/test_node_creation.py	95.60% <0.00%> (-0.20%)	⬇️
tests/flytekit/unit/cli/pyflyte/test_run.py	99.16% <0.00%> (-0.04%)	⬇️
... and 3 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.