-
Notifications
You must be signed in to change notification settings - Fork 305
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Pickle remote task for Jupyter Notebook Environment (#2733)
* Use Jupyter notebooks to execute tasks Signed-off-by: Ketan Umare <[email protected]> * More updates Signed-off-by: Ketan Umare <[email protected]> * map task update Signed-off-by: Ketan Umare <[email protected]> * entrypoint fix Signed-off-by: Ketan Umare <[email protected]> * Working map task Signed-off-by: Ketan Umare <[email protected]> * updated for map tasks and removed unnecessary prints Signed-off-by: Ketan Umare <[email protected]> * Almost working, dynamic needs work in remote Signed-off-by: Ketan Umare <[email protected]> * Dynamic task trial Signed-off-by: Ketan Umare <[email protected]> * dynamic update Signed-off-by: Ketan Umare <[email protected]> * Updates Signed-off-by: Ketan Umare <[email protected]> * map task fix Signed-off-by: Ketan Umare <[email protected]> * create interface for task & workflow for jupyter notebook Signed-off-by: Mecoli1219 <[email protected]> * lint & docstring Signed-off-by: Mecoli1219 <[email protected]> * enable computed version Signed-off-by: Mecoli1219 <[email protected]> * add version for workflow Signed-off-by: Mecoli1219 <[email protected]> * disable file upload for workflow Signed-off-by: Mecoli1219 <[email protected]> * lint Signed-off-by: Mecoli1219 <[email protected]> * Add testing jupyter interaction Signed-off-by: Mecoli1219 <[email protected]> * lint & fix import script error Signed-off-by: Mecoli1219 <[email protected]> * Update unit test Signed-off-by: Mecoli1219 <[email protected]> * Make it thread safe Signed-off-by: Mecoli1219 <[email protected]> * Enable options for both init_remote() and remote() & restrict one-time init_remote() & update doc Signed-off-by: Mecoli1219 <[email protected]> * [Still Trying...] Store the interactive_mode_enabled to avoid using ipython_check Signed-off-by: Mecoli1219 <[email protected]> * Add interactive mode enable for init_remote & fix unit test Signed-off-by: Mecoli1219 <[email protected]> * Fix intergration test Signed-off-by: Mecoli1219 <[email protected]> * Fix unit-test Signed-off-by: Mecoli1219 <[email protected]> * Update integration test names & skip the unsupported test Signed-off-by: Mecoli1219 <[email protected]> * lint Signed-off-by: Mecoli1219 <[email protected]> * enable verbosity & small update & update docstring Signed-off-by: Mecoli1219 <[email protected]> * Enable map_task with partial_fn & Add unit test Signed-off-by: Mecoli1219 <[email protected]> * save from merge Signed-off-by: Mecoli1219 <[email protected]> * Add translator unit test for interactive Signed-off-by: Mecoli1219 <[email protected]> * Remove fast_register_file_uploader Signed-off-by: Mecoli1219 <[email protected]> * Fix merge error Signed-off-by: Mecoli1219 <[email protected]> * Remove future Signed-off-by: Mecoli1219 <[email protected]> * Fix unit-test Signed-off-by: Mecoli1219 <[email protected]> * Fix entrypoint Signed-off-by: Mecoli1219 <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * update tracker Signed-off-by: Kevin Su <[email protected]> * Fix I/O on closed file error Signed-off-by: Mecoli1219 <[email protected]> * Update integration test Signed-off-by: Mecoli1219 <[email protected]> * Update integration test Signed-off-by: Mecoli1219 <[email protected]> * Remove unused try & error for event loop Signed-off-by: Mecoli1219 <[email protected]> * Set size limit to pickled file Signed-off-by: Mecoli1219 <[email protected]> * Remove nest_asyncio Signed-off-by: Mecoli1219 <[email protected]> * Remove some args for pyflyte-execute Signed-off-by: Mecoli1219 <[email protected]> * Remove changes to entrypoint Signed-off-by: Mecoli1219 <[email protected]> * Fix map task Signed-off-by: Mecoli1219 <[email protected]> * Remove unnecessary code Signed-off-by: Mecoli1219 <[email protected]> * Remove unnecessary code Signed-off-by: Mecoli1219 <[email protected]> * Add more test Signed-off-by: Mecoli1219 <[email protected]> * fix unit test Signed-off-by: Mecoli1219 <[email protected]> * notebook task resolver Signed-off-by: Kevin Su <[email protected]> * set resolver Signed-off-by: Kevin Su <[email protected]> * Update test Signed-off-by: Mecoli1219 <[email protected]> * Update test Signed-off-by: Mecoli1219 <[email protected]> * Update test Signed-off-by: Mecoli1219 <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * fix unit test Signed-off-by: Kevin Su <[email protected]> * fix unit test Signed-off-by: Kevin Su <[email protected]> * interactive_mode_enabled=True Signed-off-by: Kevin Su <[email protected]> * display_ipython_warning Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Mecoli1219 <[email protected]> * fix unit test Signed-off-by: Mecoli1219 <[email protected]> * Move pkl.gz to global variable Signed-off-by: Mecoli1219 <[email protected]> * change logging level Signed-off-by: Mecoli1219 <[email protected]> * Fix map task not using notebook resolver Signed-off-by: Mecoli1219 <[email protected]> * Add map task serialization unit test Signed-off-by: Mecoli1219 <[email protected]> * moving some codes & fix map_task problem Signed-off-by: Mecoli1219 <[email protected]> * Fix map task extract pickled task module error Signed-off-by: Mecoli1219 <[email protected]> * wrap the ipython check into a function Signed-off-by: Mecoli1219 <[email protected]> * Fix test Signed-off-by: Thomas J. Fan <[email protected]> * Adds integration test for jupyter Signed-off-by: Thomas J. Fan <[email protected]> * Install kernel before using KernelManager Signed-off-by: Thomas J. Fan <[email protected]> * Add a module scope kernel install Signed-off-by: Thomas J. Fan <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Mecoli1219 <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Thomas J. Fan <[email protected]> Co-authored-by: Ketan Umare <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Thomas J. Fan <[email protected]>
- Loading branch information
1 parent
fdf93da
commit a3131f2
Showing
17 changed files
with
552 additions
and
26 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -55,6 +55,8 @@ pyarrow | |
scikit-learn | ||
types-requests | ||
prometheus-client | ||
jupyter-client | ||
ipykernel | ||
|
||
orjson | ||
kubernetes>=12.0.1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.