Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data subsystem yt #1518

Merged
merged 45 commits into from
Feb 28, 2023
Merged

Data subsystem yt #1518

merged 45 commits into from
Feb 28, 2023

Conversation

wild-endeavor
Copy link
Contributor

ignore

pbrogan12 and others added 30 commits January 4, 2023 12:03
@pbrogan12
docs: update register help, non-fast version is supported (#1402)
48bab48 
Signed-off-by: Patrick Brogan <[email protected]>
@pingsutw
Update log level for structured dataset (#1394)
be32ac7 
Signed-off-by: Kevin Su <[email protected]>
@pingsutw
Add Niels to code owners (#1404)
0dcda1d 
Signed-off-by: Kevin Su <[email protected]>
@wild-endeavor
Signal use (#1398)
d008917 
Signed-off-by: Yee Hing Tong <[email protected]>
@pingsutw
User Documentation Proposal (#1200)
8c72082 
Signed-off-by: Kevin Su <[email protected]>
@kumare3 @pingsutw @cosmicBboy
Add support MLFlow plugin (#1274)
55b6602 
* MLFlow plugin in progress

Signed-off-by: Ketan Umare <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* update test

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* update readme

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* dwip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* change experiment name

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* Add mlflow to index.rst

Signed-off-by: Kevin Su <[email protected]>

* use experiment name that user provided

Signed-off-by: Kevin Su <[email protected]>

* update doc-requirements.txt

Signed-off-by: Kevin Su <[email protected]>

* Add backend plugin deployment

Signed-off-by: Kevin Su <[email protected]>

* generate doc for method

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

* update docstring

Signed-off-by: Niels Bantilan <[email protected]>

* update docstring

Signed-off-by: Niels Bantilan <[email protected]>

* Update tracking.py

Signed-off-by: Niels Bantilan <[email protected]>

Signed-off-by: Ketan Umare <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Niels Bantilan <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Co-authored-by: Niels Bantilan <[email protected]>
@cosmicBboy
fix remote API reference (#1405)
fcf6dce 
Signed-off-by: Niels Bantilan <[email protected]>

Signed-off-by: Niels Bantilan <[email protected]>
@pingsutw @wild-endeavor
Read structured dataset from a folder (#1406)
1311ea4 
* Read polars dataframe in a folder

Signed-off-by: Kevin Su <[email protected]>

* Read polars dataframe in a folder

Signed-off-by: Kevin Su <[email protected]>

* Load huggingface and spark plugin implicitly

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* remove _pyspark alias

Signed-off-by: Yee Hing Tong <[email protected]>

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Co-authored-by: Yee Hing Tong <[email protected]>
@cosmicBboy
Update default config to work out-of-the-box with flytectl demo (#1384)
581a5c6 
Signed-off-by: Niels Bantilan <[email protected]>
@bstadlbauer @eapolinario
Add dask plugin #patch (#1366)
531db3c 
* Add dummy task type to test backend plugin

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add docs page

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add dask models

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add function to convert resources

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add tests to `dask` task

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Remove namespace

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update setup.py

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add dask to `plugin/README.md`

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add README.md for `dask`

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Top level export of `JopPodSpec` and `DaskCluster`

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update docs for images

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update README.md

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update models after `flyteidl` change

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update task after `flyteidl` change

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Raise error when less than 1 worker

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update flyteidl to >= 1.3.2

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update doc requirements

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update doc-requirements.txt

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Re-lock dependencies on linux

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update dask API docs

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Fix documentation links

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Default optional model constructor arguments to `None`

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Refactor `convert_resources_to_resource_model` to `core.resources`

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Use `convert_resources_to_resource_model` in `core.node`

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Incorporate review feedback

Signed-off-by: Eduardo Apolinario <[email protected]>

* Lint

Signed-off-by: Eduardo Apolinario <[email protected]>

Signed-off-by: Bernhard Stadlbauer <[email protected]>
Signed-off-by: Bernhard Stadlbauer <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
@pingsutw
Add support for overriding task configurations (#1410)
4b1675f 
Signed-off-by: Kevin Su <[email protected]>
@pingsutw
Warning if git is not installed (#1414)
905cabc 
* warning if git is not installed

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: Kevin Su <[email protected]>
@wild-endeavor
Flip the settings for channel and logger (#1415)
5dd887c 
Signed-off-by: Yee Hing Tong <[email protected]>
@kumare3
Preserving Exception in the LazyEntity fetch (#1412)
310aea3 
* Preserving Exception in the LazyEntity fetch

Signed-off-by: Ketan Umare <[email protected]>

* updated lint error

Signed-off-by: Ketan Umare <[email protected]>

* more tests

Signed-off-by: Ketan Umare <[email protected]>

Signed-off-by: Ketan Umare <[email protected]>
@ppiegaze @EngHabu
[Docs] SynchronousFlyteClient API reference #3095 (#1416)
b6605bc 
Signed-off-by: Peeter Piegaze <[email protected]>

Signed-off-by: Peeter Piegaze <[email protected]>
Co-authored-by: Peeter Piegaze <[email protected]>
Co-authored-by: Haytham Abuelfutuh <[email protected]>
@pingsutw
Return error code on fail (#1408)
99722d5 
* AWS batch return error code once it fails

Signed-off-by: Kevin Su <[email protected]>

* AWS batch return error code once it fails

Signed-off-by: Kevin Su <[email protected]>

* update tests

Signed-off-by: Kevin Su <[email protected]>

* Update tests

Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
@wild-endeavor
wrapping flyte entity in a task node in call to flyte node constructo…
5126b2a 
…r, not sure if integration tests are actually running (#1422)

Signed-off-by: Yee Hing Tong <[email protected]>

Signed-off-by: Yee Hing Tong <[email protected]>
@cosmicBboy
Sqlalchemy multiline query (#1421)
7b48fff 
* SQLAlchemyTask should handle multiline strings for query template

Signed-off-by: Niels Bantilan <[email protected]>

* sqlalchemy supports multi-line query

Signed-off-by: Niels Bantilan <[email protected]>

* update base sql task

Signed-off-by: Niels Bantilan <[email protected]>

* remove space

Signed-off-by: Niels Bantilan <[email protected]>

* fix snowflake tests

Signed-off-by: Niels Bantilan <[email protected]>

* fix lint

Signed-off-by: Niels Bantilan <[email protected]>

* fix test

Signed-off-by: Niels Bantilan <[email protected]>

Signed-off-by: Niels Bantilan <[email protected]>
@cosmicBboy
Sklearn type transformer should be automatically loaded with import f…
1513773 
…lytekit (#1423)

* add flytekit.extras.sklearn to main __init__ import

Signed-off-by: Niels Bantilan <[email protected]>

* fix docs

Signed-off-by: Niels Bantilan <[email protected]>

* add temporary docs/requirements.txt for onnx plugins

Signed-off-by: Niels Bantilan <[email protected]>

---------

Signed-off-by: Niels Bantilan <[email protected]>
@eapolinario
Bump isort to 5.12.0 (#1427)
88c6bba 
Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
@kumare3 @eapolinario
Fixes guess type bug in UnionTransformer (#1426)
ea39054 
Signed-off-by: Ketan Umare <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
@ByronHsu @eapolinario
Add pod_template and pod_template_name arguments for `PythonAutoC…
9dc4d4d 
…ontainerTask`, its downstream tasks, and `@task`. (#1425)

* Add `pod_template` and `pod_template_name` arguments for `PythonAutoContainerTask`, its downstream tasks, and `@task`

Signed-off-by: byhsu <[email protected]>

* clean

Signed-off-by: byhsu <[email protected]>

* fix test

Signed-off-by: byhsu <[email protected]>

* Fix taskmetadata

Signed-off-by: byhsu <[email protected]>

* add kubernetes in setup.py

Signed-off-by: byhsu <[email protected]>

* address comments

Signed-off-by: byhsu <[email protected]>

* Regenerate requirements using python 3.7

Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: byhsu <[email protected]>

* keep container validation

Signed-off-by: byhsu <[email protected]>

* bump idl version

Signed-off-by: byhsu <[email protected]>

* Regenerate requirements using python 3.7

Signed-off-by: Eduardo Apolinario <[email protected]>

* Regenerate doc-requirements.txt

Signed-off-by: Eduardo Apolinario <[email protected]>

* fix

Signed-off-by: byhsu <[email protected]>

---------

Signed-off-by: byhsu <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: byhsu <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
@kumare3
Auto Backfill workflow (#1420)
d006df6
@samhita-alla
Fix primitive decoder when evaluating Promise (#1432)
a5c9970 
Signed-off-by: Samhita Alla <[email protected]>
@cosmicBboy
set maximum python version to 3.10 (#1433)
dee4804 
* set maximum python version to 3.10

Signed-off-by: Niels Bantilan <[email protected]>

* remove unneeded python version check

Signed-off-by: Niels Bantilan <[email protected]>

* fix lint

Signed-off-by: Niels Bantilan <[email protected]>

---------

Signed-off-by: Niels Bantilan <[email protected]>
@eapolinario
Revert "Remove project/domain from being overridden with execution va…
47ac6ac 
…lues in serialized context (#1378)" (#1460)

* Revert "Remove project/domain from being overridden with execution values in serialized context (#1378)"

This reverts commit b3bfef5.

* Import os

Signed-off-by: Eduardo Apolinario <[email protected]>

* Lint

Signed-off-by: Eduardo Apolinario <[email protected]>

---------

Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
@bethebunny @stef-stripe @eapolinario
Support checkpointing in local mode from cached tasks (#1457)
35b1fa6 
* support checkpointing in local mode from cached tasks

* clear cache before tests

---------

Co-authored-by: Stef Nelson-Lindall <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
@pingsutw @eapolinario
Deprecate FlyteSchema (#1418)
5087d38 
* Deprecate FlyteSchema

Signed-off-by: Kevin Su <[email protected]>

* Remove version

Signed-off-by: Kevin Su <[email protected]>

---------

Signed-off-by: Kevin Su <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
@eapolinario
Use scarf images (#1434)
8e5bf6c 
* Use scarf images

Signed-off-by: Eduardo Apolinario <[email protected]>

* Use scarf names in tests.

Signed-off-by: Eduardo Apolinario <[email protected]>

---------

Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
@cosmicBboy @eapolinario @samhita-alla
add undocumented objects/functions to flytekit api ref (#1502)
ecded3e 
* add reference_launch_plan to flytekit api ref

Signed-off-by: Niels Bantilan <[email protected]>

* import in init, add docstrings

Signed-off-by: Niels Bantilan <[email protected]>

* add more to references

Signed-off-by: Niels Bantilan <[email protected]>

* fix lint

Signed-off-by: Niels Bantilan <[email protected]>

* update

Signed-off-by: Niels Bantilan <[email protected]>

* fix up docstrings

Signed-off-by: Niels Bantilan <[email protected]>

---------

Signed-off-by: Niels Bantilan <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Samhita Alla <[email protected]>
pingsutw and others added 15 commits February 14, 2023 09:23
@pingsutw
Use non-root user in default flytekit image (#1417)
606e0c4 
Signed-off-by: Kevin Su <[email protected]>
@samhita-alla
Fix PyTorch transformer (#1510)
03d9f62 
Signed-off-by: Samhita Alla <[email protected]>
@pingsutw @wild-endeavor
Fix mypy errors (#1313)
c11362f 
* wip

Signed-off-by: Kevin Su <[email protected]>

* Fix mypy errors

Signed-off-by: Kevin Su <[email protected]>

* Fix mypy errors

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix test

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* Update type

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* update dev-requirements.txt

Signed-off-by: Kevin Su <[email protected]>

* Address comment

Signed-off-by: Kevin Su <[email protected]>

* upgrade torch

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

---------

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Co-authored-by: Yee Hing Tong <[email protected]>
@pingsutw
Compile the workflow only at compile time (#1311)
014eea9 
* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* add tests

Signed-off-by: Kevin Su <[email protected]>

* add tests

Signed-off-by: Kevin Su <[email protected]>

* support dynamic task

Signed-off-by: Kevin Su <[email protected]>

* test

Signed-off-by: Kevin Su <[email protected]>

* test

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* lazy compile

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

* add tests

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* test

Signed-off-by: Kevin Su <[email protected]>

* test

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

* test

Signed-off-by: Kevin Su <[email protected]>

* test

Signed-off-by: Kevin Su <[email protected]>

* test

Signed-off-by: Kevin Su <[email protected]>

* test

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* update test

Signed-off-by: Kevin Su <[email protected]>

---------

Signed-off-by: Kevin Su <[email protected]>
@pingsutw @cosmicBboy
Get the origin type when serializing dataclass (#1508)
74cbe58 
* Get the origin type when serializing dataclass

Signed-off-by: Kevin Su <[email protected]>

* test

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* update test

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

---------

Signed-off-by: Kevin Su <[email protected]>
Co-authored-by: Niels Bantilan <[email protected]>
@eapolinario
Remove requirements files (#1511)
99d3d50 
* Remove mentions to spark

Signed-off-by: Eduardo Apolinario <[email protected]>

* Remove mentions to requirements.txt and dev-requirements.txt

Signed-off-by: Eduardo Apolinario <[email protected]>

* Remove plugins requirements.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Remove all_requirements target from plugins makefile

Signed-off-by: Eduardo Apolinario <[email protected]>

* Use dev-requirements.in in lint gh action job

Signed-off-by: Eduardo Apolinario <[email protected]>

* Revert "Remove all_requirements target from plugins makefile"

This reverts commit 50cbb4d.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Revert "Remove plugins requirements."

This reverts commit eae945c.

Signed-off-by: Eduardo Apolinario <[email protected]>

---------

Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
@eapolinario
Delay initialization of SynchronousFlyteClient in FlyteRemote (#1514)
707fc03 
* Delay initialization of SynchronousFlyteClient in FlyteRemote

Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix spark plugin flyteremote test.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Lint

Signed-off-by: Eduardo Apolinario <[email protected]>

---------

Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
@wild-endeavor
start testing
b4064fc 
Signed-off-by: Yee Hing Tong <[email protected]>
@wild-endeavor
merge master and resolve
e0c48da 
Signed-off-by: Yee Hing Tong <[email protected]>
@wild-endeavor
fix a couple tests
16ff0ae 
Signed-off-by: Yee Hing Tong <[email protected]>
@wild-endeavor
steadily cleaning up tests
b7dcfa1 
Signed-off-by: Yee Hing Tong <[email protected]>
@wild-endeavor
tmp
abbb65a 
Signed-off-by: Yee Hing Tong <[email protected]>
@wild-endeavor
fmt and some test fixes
22c6a6c 
Signed-off-by: Yee Hing Tong <[email protected]>
@wild-endeavor
special case local dir copy
ff32e3a 
Signed-off-by: Yee Hing Tong <[email protected]>
@wild-endeavor
move import up
027d3fe 
Signed-off-by: Yee Hing Tong <[email protected]>
@wild-endeavor wild-endeavor marked this pull request as ready for review February 28, 2023 19:30
@wild-endeavor wild-endeavor merged commit e69317f into data-subsystem Feb 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kumare3 kumare3 Awaiting requested review from kumare3 kumare3 is a code owner

@eapolinario eapolinario Awaiting requested review from eapolinario eapolinario is a code owner

@pingsutw pingsutw Awaiting requested review from pingsutw pingsutw is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.