-
Notifications
You must be signed in to change notification settings - Fork 307
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Creating Whylogs transformer + renderers #1100
Conversation
Thank you for opening this pull request! 🙌 These tips will help get your PR across the finish line:
|
Codecov Report
@@ Coverage Diff @@
## master #1100 +/- ##
==========================================
+ Coverage 86.41% 86.92% +0.51%
==========================================
Files 255 275 +20
Lines 24387 25453 +1066
Branches 2774 2862 +88
==========================================
+ Hits 21073 22125 +1052
- Misses 2842 2850 +8
- Partials 472 478 +6
Continue to review full report at Codecov.
|
* Add support deck in jupyter notebook Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * fmt Signed-off-by: Kevin Su <[email protected]> * fmt Signed-off-by: Kevin Su <[email protected]> * fmt Signed-off-by: Kevin Su <[email protected]> * fix lint Signed-off-by: Kevin Su <[email protected]> * fix test Signed-off-by: Kevin Su <[email protected]> * More tests Signed-off-by: Kevin Su <[email protected]> * Updated ipython Signed-off-by: Kevin Su <[email protected]> * add logger Signed-off-by: Kevin Su <[email protected]> * Fix lint Signed-off-by: Kevin Su <[email protected]> * Fix lint Signed-off-by: Kevin Su <[email protected]> * refine the code Signed-off-by: Kevin Su <[email protected]> * Refactor Signed-off-by: Kevin Su <[email protected]> * lint fix Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * Add display deck function Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * update Signed-off-by: Kevin Su <[email protected]> Signed-off-by: murilommen <[email protected]>
I've got several TODOs in the code where things need to change but this is a good first step. Next is to take a stab at the rendering plugin. Signed-off-by: murilommen <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: murilommen <[email protected]>
We published our rc to pypi so I can actually reference it directly without the wheel. Signed-off-by: murilommen <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: murilommen <[email protected]>
The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-PILLOW-2838873 Co-authored-by: snyk-bot <[email protected]> Signed-off-by: murilommen <[email protected]>
…bworkflow order (#1011) * update get_serializable_workflow Signed-off-by: Vrinda Vasavada <[email protected]> * make fmt Signed-off-by: Vrinda Vasavada <[email protected]> Signed-off-by: murilommen <[email protected]>
* Fix the tests on windows Signed-off-by: Kevin Su <[email protected]> * Fix the tests on windows Signed-off-by: Kevin Su <[email protected]> * Add comment Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * pin pip Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> Signed-off-by: murilommen <[email protected]>
Signed-off-by: SmritiSatyanV <[email protected]> Signed-off-by: murilommen <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: murilommen <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: murilommen <[email protected]>
* Bump requirements files to bring in ipython>7.31 Signed-off-by: Eduardo Apolinario <[email protected]> * Bump plugins requirements Signed-off-by: Eduardo Apolinario <[email protected]> * delete flytekit-deck-standard/__init__.py Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: murilommen <[email protected]>
* Fix importing code in script mode * Store _filename using an absolute path and comment the use of it Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: murilommen <[email protected]>
…in imperative (#1000) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: murilommen <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: murilommen <[email protected]>
* Added support for insecureSkipVerify Signed-off-by: Roberto Ruiz <[email protected]> * Extract port from endpoint before obtaining unverified SSL certificate Signed-off-by: Roberto Ruiz <[email protected]> * Code reformat using black Signed-off-by: Roberto Ruiz <[email protected]> * Added pyopenssl requirement Signed-off-by: Roberto Ruiz <[email protected]> * Corrected error from typo Signed-off-by: Roberto Ruiz <[email protected]> * Added corrections on how the endpoint is split into server and port Signed-off-by: Roberto Ruiz <[email protected]> * Use the correct split method to handle multiple occurrences of ':' Signed-off-by: Roberto Ruiz <[email protected]> * Regenerated requirements-spark2.txt and doc-requirements.txt Signed-off-by: Roberto Ruiz <[email protected]> * Generated all requirements files together Signed-off-by: Roberto Ruiz <[email protected]> Signed-off-by: murilommen <[email protected]>
* Allow struct/dataclass types to be used as default arguments Signed-off-by: Roberto Ruiz <[email protected]> * Add test case Signed-off-by: Eduardo Apolinario <[email protected]> * Lint Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: murilommen <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: murilommen <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: murilommen <[email protected]>
* add polars plugin Signed-off-by: Robin Kahlow <[email protected]> * support for older polars versions, add info about what polars is Signed-off-by: Robin Kahlow <[email protected]> * run make fmt Signed-off-by: Robin Kahlow <[email protected]> * structured dataset instead of schema transformer Signed-off-by: Robin Kahlow <[email protected]> * polars html describe only Signed-off-by: Robin Kahlow <[email protected]> * set polars min to 0.7.13 (.describe() added) Signed-off-by: Robin Kahlow <[email protected]> * set polars min to 0.8.27 (.transpose() added) Signed-off-by: Robin Kahlow <[email protected]> * add gcs, fix encode local/remote dir Signed-off-by: Robin Kahlow <[email protected]> * black and isort Signed-off-by: Robin Kahlow <[email protected]> * add polars plugin to pythonbuild.yml Signed-off-by: Robin Kahlow <[email protected]> Signed-off-by: murilommen <[email protected]>
…resh_credentials_from_command (#1065) * add support for custom auth from env Signed-off-by: Sonja Ericsson <[email protected]> * Update flytekit/clis/flyte_cli/main.py Signed-off-by: Sonja Ericsson <[email protected]> * fix Signed-off-by: Sonja Ericsson <[email protected]> * fix Signed-off-by: Sonja Ericsson <[email protected]> * fix Signed-off-by: Sonja Ericsson <[email protected]> * fix Signed-off-by: Sonja Ericsson <[email protected]> * fix Signed-off-by: Sonja Ericsson <[email protected]> * Update test_raw.py Signed-off-by: Sonja Ericsson <[email protected]> * fix Signed-off-by: Sonja Ericsson <[email protected]> * fix Signed-off-by: Sonja Ericsson <[email protected]> * fix Signed-off-by: Sonja Ericsson <[email protected]> * fix Signed-off-by: Sonja Ericsson <[email protected]> * fix Signed-off-by: Sonja Ericsson <[email protected]> * fix Signed-off-by: Sonja Ericsson <[email protected]> * fix Signed-off-by: Sonja Ericsson <[email protected]> * fix Signed-off-by: Sonja Ericsson <[email protected]> * fix Signed-off-by: Sonja Ericsson <[email protected]> Signed-off-by: murilommen <[email protected]>
Signed-off-by: Jeev B <[email protected]> Signed-off-by: murilommen <[email protected]>
* Add support union type for pyflyte run Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * More tests Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> Signed-off-by: murilommen <[email protected]>
* Support optional input Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * set default value for optional inputs when compiling Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * add tests Signed-off-by: Kevin Su <[email protected]> * More tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> Signed-off-by: murilommen <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: murilommen <[email protected]>
Signed-off-by: Kevin Su <[email protected]> Signed-off-by: murilommen <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: Kevin Su <[email protected]> Signed-off-by: murilommen <[email protected]>
* Add details on fsspec data plugin * fixed lint error * Review based changes Signed-off-by: SmritiSatyanV <[email protected]> Signed-off-by: murilommen <[email protected]>
Signed-off-by: Kevin Su <[email protected]> Signed-off-by: murilommen <[email protected]>
They show up in the pypi page: https://pypi.org/project/flytekit/ Signed-off-by: Kevin Su <[email protected]> Signed-off-by: murilommen <[email protected]>
Signed-off-by: Kevin Su <[email protected]> Signed-off-by: murilommen <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: murilommen <[email protected]>
Signed-off-by: Madhur Tandon <[email protected]> Signed-off-by: murilommen <[email protected]>
The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-LXML-2940874 Co-authored-by: snyk-bot <[email protected]> Signed-off-by: murilommen <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: murilommen <[email protected]>
* fix: plugins/flytekit-greatexpectations/requirements.txt to reduce vulnerabilities The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-MISTUNE-2940625 * Fixed tests Signed-off-by: Kevin Su <[email protected]> Co-authored-by: snyk-bot <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: murilommen <[email protected]>
* fix: plugins/flytekit-papermill/requirements.txt to reduce vulnerabilities The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-MISTUNE-2940625 * bump nbconvert to 7.0.0rc2 Signed-off-by: Yee Hing Tong <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Signed-off-by: murilommen <[email protected]>
* TypeTransformers for PyTorch Tensor and Module Signed-off-by: Samhita Alla <[email protected]> * add torch to requirements Signed-off-by: Samhita Alla <[email protected]> * add module as a native type and PyTorchCheckpoint Signed-off-by: Samhita Alla <[email protected]> * update requirements Signed-off-by: Samhita Alla <[email protected]> * procedural to OOP approach Signed-off-by: Samhita Alla <[email protected]> * nit Signed-off-by: Samhita Alla <[email protected]> * verify device conversion Signed-off-by: Samhita Alla <[email protected]> * verify device conversion Signed-off-by: Samhita Alla <[email protected]> * hyperparameters can be None Signed-off-by: Samhita Alla <[email protected]> * device conversion Signed-off-by: Samhita Alla <[email protected]> * device conversion Signed-off-by: Samhita Alla <[email protected]> * checkpoint code cleanup Signed-off-by: Samhita Alla <[email protected]> * resolve merge conflict Signed-off-by: Samhita Alla <[email protected]> * fix pytorch api reference; resolve merge conflict Signed-off-by: Samhita Alla <[email protected]> * fix pytorch import Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: murilommen <[email protected]>
…1096) * Correctly fetch metadata in Signed-off-by: Samhita Alla <[email protected]> * fix import issue Signed-off-by: Samhita Alla <[email protected]> * add unit test Signed-off-by: Samhita Alla <[email protected]> * fix low codecov Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: murilommen <[email protected]>
Signed-off-by: murilommen <[email protected]>
* fix isort errors Signed-off-by: Samhita Alla <[email protected]> * modified the plugin structure Signed-off-by: Samhita Alla <[email protected]> * resolve merge conflicts Signed-off-by: Samhita Alla <[email protected]> * added more parameters and cleaned up code Signed-off-by: Samhita Alla <[email protected]> * add readme Signed-off-by: Samhita Alla <[email protected]> * modified package names Signed-off-by: Samhita Alla <[email protected]> * update pythonbuild; add ONNXFile Signed-off-by: Samhita Alla <[email protected]> * wip Signed-off-by: Samhita Alla <[email protected]> * update requirements Signed-off-by: Samhita Alla <[email protected]> * exclude tests on python 3.10 Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: murilommen <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add the new plugin to this list so that tests run in CI?
plugins/flytekit-deck-standard/flytekitplugins/deck/renderer.py
Outdated
Show resolved
Hide resolved
Signed-off-by: murilommen <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: murilommen <[email protected]>
TL;DR
It creates the whylogs plugin to extend Flyte's usage.
Type
Are all requirements met?
Complete description
It creates a schema for "transforming" whylogs'
DatasetProfileView
objects to and from the FlyteSchema. This PR also creates two renderers with FlyteDeck, one for the Constraints report and the other for the SummaryDriftReport.Tracking Issue
N/A
Follow-up issue
NA