Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lint docs #679

Merged
merged 1 commit into from
Sep 28, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<html>
<p align="center">
<p align="center">
<img src="https://github.com/flyteorg/flyte/blob/master/rsts/images/flyte_circle_gradient_1_4x4.png" alt="Flyte Logo" width="100">
</p>
<h1 align="center">
Expand All @@ -25,15 +25,15 @@
[![Docs](https://readthedocs.org/projects/flytekit/badge/?version=latest&style=plastic)](https://flytekit.rtfd.io)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

Flytekit Python is the Python Library for easily authoring, testing, deploying, and interacting with Flyte tasks, workflows, and launch plans.
Flytekit Python is the Python Library for easily authoring, testing, deploying, and interacting with Flyte tasks, workflows, and launch plans.

If you haven't explored Flyte yet, please refer to:
- [Flyte homepage](https://flyte.org)
- [Flyte core repository](https://github.com/flyteorg/flyte)

## 🚀 Quick Start

Flytekit is the core extensible library to author Flyte workflows and tasks and interact with Flyte backend services.
Flytekit is the core extensible library to author Flyte workflows and tasks and interact with Flyte backend services.

### Installation

Expand Down Expand Up @@ -71,5 +71,5 @@ You can find the detailed contribution guide [here](https://docs.flyte.org/proje
Refer to the [issues](https://docs.flyte.org/en/latest/community/contribute.html#issues) section in the contribution guide if you'd like to file an issue.

## 🔌 Flytekit Plugins
Refer to [plugins/README.md](plugins/README.md) for a list of available plugins.
Refer to [plugins/README.md](plugins/README.md) for a list of available plugins.
There may be plugins outside of this list, but this list is maintained by the core maintainers.
8 changes: 4 additions & 4 deletions docs/source/contributing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ This will install flytekit dependencies and also install flytekit itself in edit
******************
Plugin Development
******************
As discussed in the design component, Flytekit plugins currently live in this flytekit repo, but under a different top level folder ``plugins``.
As discussed in the design component, Flytekit plugins currently live in this flytekit repo, but under a different top level folder ``plugins``.
In the future, this will be separated out into a different repo. These plugins follow a `microlib <https://medium.com/@jherreras/python-microlibs-5be9461ad979>`__ structure, which will persist even if we move repos. ::

source ~/.virtualenvs/flytekit/bin/activate
Expand Down Expand Up @@ -69,7 +69,7 @@ Running unit tests: ::

Cookbook Testing
----------------
Please see the `cookbook <https://github.com/flyteorg/flytesnacks/tree/master/cookbook>`__ and the generated `docs <https://flytecookbook.readthedocs.io/en/latest/>`__ for more information.
Please see the `cookbook <https://github.com/flyteorg/flytesnacks/tree/master/cookbook>`__ and the generated `docs <https://flytecookbook.readthedocs.io/en/latest/>`__ for more information.
This example repo can be cloned and run on a local Flyte cluster, or just in your IDE or other Python environment.

Follow the setup instructions for the cookbook and then override it with the version of Flytekit you're interested in testing by running something like: ::
Expand All @@ -85,15 +85,15 @@ End-to-end Testing

.. TODO: Replace this with actual instructions

The Flyte developer experience team has put together an end-to-end testing framework that will spin up a K8s cluster, install Flyte onto it, and run through a series of workflows.
The Flyte developer experience team has put together an end-to-end testing framework that will spin up a K8s cluster, install Flyte onto it, and run through a series of workflows.
Please contact us if you reach this stage and would like more information on this.


****************
Pre-commit hooks
****************

We use `pre-commit <https://pre-commit.com/>`__ to automate linting and code formatting on every commit.
We use `pre-commit <https://pre-commit.com/>`__ to automate linting and code formatting on every commit.
Configured hooks include `black <https://github.com/psf/black>`__, `isort <https://github.com/PyCQA/isort>`__, and `flake8 <https://github.com/PyCQA/flake8>`__ and also linters to check for the validity of YAML files and ensuring that newlines are added to the end of files.

We run all those hooks in CI, but if you want to run them locally on every commit, run `pre-commit install` after installing the dev environment requirements. In case you want to disable `pre-commit` hooks locally, for example, while you're iterating on some feature, run `pre-commit uninstall`. More info in https://pre-commit.com/.
Expand Down
7 changes: 3 additions & 4 deletions plugins/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ All Flytekit plugins maintained by the core team are added here. It is not neces
Please [file an issue](https://github.com/flyteorg/flyte/issues/new?assignees=&labels=untriaged%2Cplugins&template=backend-plugin-request.md&title=%5BPlugin%5D).

## Development 💻
Flytekit plugins are structured as micro-libs and can be authored in an independent repository.
Flytekit plugins are structured as micro-libs and can be authored in an independent repository.

> Refer to the [Python microlibs](https://medium.com/@jherreras/python-microlibs-5be9461ad979) blog to understand the idea of microlibs.

Expand All @@ -38,7 +38,7 @@ Plugins should have their own unit tests.
Some guidelines to help you write the Flytekit plugins better.

1. The folder name has to be `flytekit-*`, e.g., `flytekit-hive`. In case you want to group for a specific service, then use `flytekit-aws-athena`.
2. Flytekit plugins use a concept called [Namespace packages](https://packaging.python.org/guides/creating-and-discovering-plugins/#using-namespace-packages), and thus, the package structure is essential.
2. Flytekit plugins use a concept called [Namespace packages](https://packaging.python.org/guides/creating-and-discovering-plugins/#using-namespace-packages), and thus, the package structure is essential.

Please use the following Python package structure:
```
Expand Down Expand Up @@ -113,7 +113,7 @@ setup(
6. Each plugin should have its own tests' package. *NOTE:* `tests` folder should have an `__init__.py` file.

7. There may be some cases where you might want to auto-load some of your modules when the plugin is installed. This is especially true for `data-plugins` and `type-plugins`.
In such a case, you can add a special directive in the `setup.py` which will instruct Flytekit to automatically load the prescribed modules.
In such a case, you can add a special directive in the `setup.py` which will instruct Flytekit to automatically load the prescribed modules.

Following shows an excerpt from the `flytekit-data-fsspec` plugin's setup.py file.

Expand All @@ -133,4 +133,3 @@ In such a case, you can add a special directive in the `setup.py` which will ins
- Example of a plugin that modifies the execution command: [flytekit-spark](./flytekit-spark/) OR [flytekit-aws-sagemaker](./flytekit-aws-sagemaker/)
- Example that allows executing the user container with some other context modifications: [flytekit-kf-tensorflow](./flytekit-kf-tensorflow/)
- Example of a Persistence Plugin that allows data to be stored to different persistence layers: [flytekit-data-fsspec](./flytekit-data-fsspec/)

2 changes: 1 addition & 1 deletion plugins/flytekit-greatexpectations/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,4 +71,4 @@ def simple_task(
@workflow
def simple_wf(directory: str = "my_assets") -> str:
return simple_task(directory=directory)
```
```