Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Creating Whylogs transformer + renderers #1100

Closed
wants to merge 64 commits into from
Closed

Creating Whylogs transformer + renderers #1100

wants to merge 64 commits into from

Conversation

murilommen
Copy link
Contributor

TL;DR

It creates the whylogs plugin to extend Flyte's usage.

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Complete description

It creates a schema for "transforming" whylogs' DatasetProfileView objects to and from the FlyteSchema. This PR also creates two renderers with FlyteDeck, one for the Constraints report and the other for the SummaryDriftReport.

Tracking Issue

N/A

Follow-up issue

NA

@welcome
Copy link

welcome bot commented Jul 13, 2022

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

  • Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
  • Sign off your commits (Reference: DCO Guide).

@codecov
Copy link

codecov bot commented Jul 13, 2022

Codecov Report

Merging #1100 (bfd1987) into master (5d6c3dd) will increase coverage by 0.51%.
The diff coverage is 90.18%.

❗ Current head bfd1987 differs from pull request most recent head e9a40b0. Consider uploading reports for the commit e9a40b0 to get more accurate results

@@            Coverage Diff             @@
##           master    #1100      +/-   ##
==========================================
+ Coverage   86.41%   86.92%   +0.51%     
==========================================
  Files         255      275      +20     
  Lines       24387    25453    +1066     
  Branches     2774     2862      +88     
==========================================
+ Hits        21073    22125    +1052     
- Misses       2842     2850       +8     
- Partials      472      478       +6     
Impacted Files Coverage Δ
flytekit/core/base_task.py 90.90% <ø> (-0.12%) ⬇️
flytekit/core/schedule.py 89.28% <ø> (ø)
flytekit/types/structured/__init__.py 75.00% <ø> (ø)
flytekit/types/structured/structured_dataset.py 92.89% <0.00%> (-1.40%) ⬇️
...ekit/unit/core/flyte_functools/decorator_source.py 63.63% <ø> (ø)
...tekit/unit/core/flyte_functools/decorator_usage.py 83.33% <ø> (ø)
...tekit/unit/core/flyte_functools/test_decorators.py 100.00% <ø> (ø)
tests/flytekit/unit/tools/test_module_loader.py 100.00% <ø> (ø)
flytekit/clients/friendly.py 55.39% <16.66%> (-1.75%) ⬇️
flytekit/clients/raw.py 64.46% <28.57%> (-2.65%) ⬇️
... and 80 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5d6c3dd...e9a40b0. Read the comment docs.

pingsutw and others added 24 commits July 13, 2022 15:49
* Add support deck in jupyter notebook

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* fmt

Signed-off-by: Kevin Su <[email protected]>

* fmt

Signed-off-by: Kevin Su <[email protected]>

* fmt

Signed-off-by: Kevin Su <[email protected]>

* fix lint

Signed-off-by: Kevin Su <[email protected]>

* fix test

Signed-off-by: Kevin Su <[email protected]>

* More tests

Signed-off-by: Kevin Su <[email protected]>

* Updated ipython

Signed-off-by: Kevin Su <[email protected]>

* add logger

Signed-off-by: Kevin Su <[email protected]>

* Fix lint

Signed-off-by: Kevin Su <[email protected]>

* Fix lint

Signed-off-by: Kevin Su <[email protected]>

* refine the code

Signed-off-by: Kevin Su <[email protected]>

* Refactor

Signed-off-by: Kevin Su <[email protected]>

* lint fix

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* Add display deck function

Signed-off-by: Kevin Su <[email protected]>

* Fixed tests

Signed-off-by: Kevin Su <[email protected]>

* update

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: murilommen <[email protected]>
I've got several TODOs in the code where things need to change but this
is a good first step. Next is to take a stab at the rendering plugin.

Signed-off-by: murilommen <[email protected]>
Signed-off-by: murilommen <[email protected]>
We published our rc to pypi so I can actually reference it directly
without the wheel.

Signed-off-by: murilommen <[email protected]>
Signed-off-by: murilommen <[email protected]>
The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-PILLOW-2838873

Co-authored-by: snyk-bot <[email protected]>
Signed-off-by: murilommen <[email protected]>
…bworkflow order (#1011)

* update get_serializable_workflow

Signed-off-by: Vrinda Vasavada <[email protected]>

* make fmt

Signed-off-by: Vrinda Vasavada <[email protected]>
Signed-off-by: murilommen <[email protected]>
* Fix the tests on windows

Signed-off-by: Kevin Su <[email protected]>

* Fix the tests on windows

Signed-off-by: Kevin Su <[email protected]>

* Add comment

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* pin pip

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: SmritiSatyanV <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>

Co-authored-by: Eduardo Apolinario <[email protected]>
Signed-off-by: murilommen <[email protected]>
* Bump requirements files to bring in ipython>7.31

Signed-off-by: Eduardo Apolinario <[email protected]>

* Bump plugins requirements

Signed-off-by: Eduardo Apolinario <[email protected]>

* delete flytekit-deck-standard/__init__.py

Signed-off-by: Eduardo Apolinario <[email protected]>

Co-authored-by: Eduardo Apolinario <[email protected]>
Signed-off-by: murilommen <[email protected]>
* Fix importing code in script mode

* Store _filename using an absolute path and comment the use of it

Signed-off-by: Eduardo Apolinario <[email protected]>

Co-authored-by: Eduardo Apolinario <[email protected]>
Signed-off-by: murilommen <[email protected]>
…in imperative (#1000)

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: murilommen <[email protected]>
* Added support for insecureSkipVerify

Signed-off-by: Roberto Ruiz <[email protected]>

* Extract port from endpoint before obtaining unverified SSL certificate

Signed-off-by: Roberto Ruiz <[email protected]>

* Code reformat using black

Signed-off-by: Roberto Ruiz <[email protected]>

* Added pyopenssl requirement

Signed-off-by: Roberto Ruiz <[email protected]>

* Corrected error from typo

Signed-off-by: Roberto Ruiz <[email protected]>

* Added corrections on how the endpoint is split into server and port

Signed-off-by: Roberto Ruiz <[email protected]>

* Use the correct split method to handle multiple occurrences of ':'

Signed-off-by: Roberto Ruiz <[email protected]>

* Regenerated requirements-spark2.txt and doc-requirements.txt

Signed-off-by: Roberto Ruiz <[email protected]>

* Generated all requirements files together

Signed-off-by: Roberto Ruiz <[email protected]>
Signed-off-by: murilommen <[email protected]>
* Allow struct/dataclass types to be used as default arguments

Signed-off-by: Roberto Ruiz <[email protected]>

* Add test case

Signed-off-by: Eduardo Apolinario <[email protected]>

* Lint

Signed-off-by: Eduardo Apolinario <[email protected]>

Co-authored-by: Eduardo Apolinario <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: murilommen <[email protected]>
RobinKa and others added 24 commits July 13, 2022 15:49
* add polars plugin

Signed-off-by: Robin Kahlow <[email protected]>

* support for older polars versions, add info about what polars is

Signed-off-by: Robin Kahlow <[email protected]>

* run make fmt

Signed-off-by: Robin Kahlow <[email protected]>

* structured dataset instead of schema transformer

Signed-off-by: Robin Kahlow <[email protected]>

* polars html describe only

Signed-off-by: Robin Kahlow <[email protected]>

* set polars min to 0.7.13 (.describe() added)

Signed-off-by: Robin Kahlow <[email protected]>

* set polars min to 0.8.27 (.transpose() added)

Signed-off-by: Robin Kahlow <[email protected]>

* add gcs, fix encode local/remote dir

Signed-off-by: Robin Kahlow <[email protected]>

* black and isort

Signed-off-by: Robin Kahlow <[email protected]>

* add polars plugin to pythonbuild.yml

Signed-off-by: Robin Kahlow <[email protected]>
Signed-off-by: murilommen <[email protected]>
…resh_credentials_from_command (#1065)

* add support for custom auth from env

Signed-off-by: Sonja Ericsson <[email protected]>

* Update flytekit/clis/flyte_cli/main.py

Signed-off-by: Sonja Ericsson <[email protected]>

* fix

Signed-off-by: Sonja Ericsson <[email protected]>

* fix

Signed-off-by: Sonja Ericsson <[email protected]>

* fix

Signed-off-by: Sonja Ericsson <[email protected]>

* fix

Signed-off-by: Sonja Ericsson <[email protected]>

* fix

Signed-off-by: Sonja Ericsson <[email protected]>

* Update test_raw.py

Signed-off-by: Sonja Ericsson <[email protected]>

* fix

Signed-off-by: Sonja Ericsson <[email protected]>

* fix

Signed-off-by: Sonja Ericsson <[email protected]>

* fix

Signed-off-by: Sonja Ericsson <[email protected]>

* fix

Signed-off-by: Sonja Ericsson <[email protected]>

* fix

Signed-off-by: Sonja Ericsson <[email protected]>

* fix

Signed-off-by: Sonja Ericsson <[email protected]>

* fix

Signed-off-by: Sonja Ericsson <[email protected]>

* fix

Signed-off-by: Sonja Ericsson <[email protected]>

* fix

Signed-off-by: Sonja Ericsson <[email protected]>
Signed-off-by: murilommen <[email protected]>
* Add support union type for pyflyte run

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* More tests

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: murilommen <[email protected]>
* Support optional input

Signed-off-by: Kevin Su <[email protected]>

* Fixed tests

Signed-off-by: Kevin Su <[email protected]>

* Fixed tests

Signed-off-by: Kevin Su <[email protected]>

* Fixed tests

Signed-off-by: Kevin Su <[email protected]>

* Fixed tests

Signed-off-by: Kevin Su <[email protected]>

* set default value for optional inputs when compiling

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* add tests

Signed-off-by: Kevin Su <[email protected]>

* More tests

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: murilommen <[email protected]>
* Add details on fsspec data plugin
* fixed lint error
* Review based changes
Signed-off-by: SmritiSatyanV <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: murilommen <[email protected]>
They show up in the pypi page: https://pypi.org/project/flytekit/
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: murilommen <[email protected]>
The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-LXML-2940874

Co-authored-by: snyk-bot <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: murilommen <[email protected]>
* fix: plugins/flytekit-greatexpectations/requirements.txt to reduce vulnerabilities

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-MISTUNE-2940625

* Fixed tests

Signed-off-by: Kevin Su <[email protected]>

Co-authored-by: snyk-bot <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Signed-off-by: murilommen <[email protected]>
* fix: plugins/flytekit-papermill/requirements.txt to reduce vulnerabilities

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-MISTUNE-2940625

* bump nbconvert to 7.0.0rc2

Signed-off-by: Yee Hing Tong <[email protected]>

Co-authored-by: Yee Hing Tong <[email protected]>
Signed-off-by: murilommen <[email protected]>
* TypeTransformers for PyTorch Tensor and Module

Signed-off-by: Samhita Alla <[email protected]>

* add torch to requirements

Signed-off-by: Samhita Alla <[email protected]>

* add module as a native type and PyTorchCheckpoint

Signed-off-by: Samhita Alla <[email protected]>

* update requirements

Signed-off-by: Samhita Alla <[email protected]>

* procedural to OOP approach

Signed-off-by: Samhita Alla <[email protected]>

* nit

Signed-off-by: Samhita Alla <[email protected]>

* verify device conversion

Signed-off-by: Samhita Alla <[email protected]>

* verify device conversion

Signed-off-by: Samhita Alla <[email protected]>

* hyperparameters can be None

Signed-off-by: Samhita Alla <[email protected]>

* device conversion

Signed-off-by: Samhita Alla <[email protected]>

* device conversion

Signed-off-by: Samhita Alla <[email protected]>

* checkpoint code cleanup

Signed-off-by: Samhita Alla <[email protected]>

* resolve merge conflict

Signed-off-by: Samhita Alla <[email protected]>

* fix pytorch api reference; resolve merge conflict

Signed-off-by: Samhita Alla <[email protected]>

* fix pytorch import

Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: murilommen <[email protected]>
…1096)

* Correctly fetch  metadata in

Signed-off-by: Samhita Alla <[email protected]>

* fix import issue

Signed-off-by: Samhita Alla <[email protected]>

* add unit test

Signed-off-by: Samhita Alla <[email protected]>

* fix low codecov

Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: murilommen <[email protected]>
Signed-off-by: murilommen <[email protected]>
* fix isort errors

Signed-off-by: Samhita Alla <[email protected]>

* modified the plugin structure

Signed-off-by: Samhita Alla <[email protected]>

* resolve merge conflicts

Signed-off-by: Samhita Alla <[email protected]>

* added more parameters and cleaned up code

Signed-off-by: Samhita Alla <[email protected]>

* add readme

Signed-off-by: Samhita Alla <[email protected]>

* modified package names

Signed-off-by: Samhita Alla <[email protected]>

* update pythonbuild; add ONNXFile

Signed-off-by: Samhita Alla <[email protected]>

* wip

Signed-off-by: Samhita Alla <[email protected]>

* update requirements

Signed-off-by: Samhita Alla <[email protected]>

* exclude tests on python 3.10

Signed-off-by: Samhita Alla <[email protected]>
Copy link
Collaborator

@eapolinario eapolinario left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add the new plugin to this list so that tests run in CI?

plugins/flytekit-deck-standard/setup.py Outdated Show resolved Hide resolved
plugins/flytekit-whylogs/flytekitplugins/whylogs/schema.py Outdated Show resolved Hide resolved
plugins/flytekit-whylogs/tests/test_schema.py Outdated Show resolved Hide resolved
plugins/flytekit-whylogs/tests/test_schema.py Outdated Show resolved Hide resolved
.gitignore Outdated Show resolved Hide resolved
Signed-off-by: murilommen <[email protected]>
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.