Updated README to use new flytekit Beta (flyteorg#350)

README.md

@@ -14,86 +14,44 @@ Python Library for easily authoring, testing, deploying, and interacting with Fl
 
 ## Installation
 
-Flytekit is designed for minimal footprint, and thus some features must be installed as extras.
+Flytekit is the core extensible library to author Flyte workflows and tasks and interact with Flyte Backend services. Flyte plugins can be installed separately. 
 
 ### Base Installation
 
-This is the lightest-weight SDK install. This installation includes everything you need to interact with Flyte.
-
-Modules include:
-1. The full Flyte IDL and an additional model layer for easier extension of the data model.
-2. gRPC client for communicating with the platform.
-3. Implementations for authoring and extending all Flyte entities (including tasks, workflows, and launch plans).
-
-Tools include:
-1. flyte-cli (Command-Line Interface for Interacting with the Flyte Platform)
-2. pyflyte (Command-Line tool for easing the registration of Flyte entities)
-
-```bash
-pip install flytekit
-```
-
-### Plugin Installation
-#### Spark
-
-If `@spark_task` is to be used, one should install the `spark` plugin.
-
 ```bash
-pip install "flytekit[spark]" for Spark 2.4.x
-pip install "flytekit[spark3]" for Spark 3.x
+pip install flytekit==0.16.0b1
 ```
 
-Please note that Spark 2.4 support is deprecated and will be removed in a future release.
+### Simple getting started
 
-#### Schema 
+```python
+from flytekit import task, workflow
 
-If `Types.Schema()` is to be used for computations involving large dataframes, one should install the `schema` extension.
+@task(cache=True, cache_version="1", retries=3)
+def sum(x: int, y: int) -> int:
+    return x + y
 
-```bash
-pip install "flytekit[schema]"
-```
-
-#### Sidecar
-
-If `@sidecar_task` is to be used, one should install the `sidecar` plugin.
+@task(cache=True, cache_version="1", retries=3)
+def square(x: int) -> int:
+    return x*x
 
-```bash
-pip install "flytekit[sidecar]"
+@workflow
+def my_workflow(x: int, y: int) -> int:
+    return sum(x=square(x=x),y=square(y=y))
 ```
 
-### Pytorch
-
-If `@pytorch_task` is to be used, one should install the `pytorch` plugin.
+### Learn Flytekit by example using
+- [Learn flytekit by examples](https://flytecookbook.readthedocs.io/)
+- [Flytekit API documentation](http://flytekit.readthedocs.io/)
+- [Flyte documentation Hub](http://flytekit.readthedocs.io/)
 
-```bash
-pip install "flytekit[pytorch]"
-```
-
-### TensorFlow
-
-If `@tensorflow_task` is to be used, one should install the `tensorflow` plugin.
-
-```bash
-pip install flytekit[tensorflow]
-```
-
-### Full Installation
-
-To install all or multiple available plugins, one can specify them individually:
-
-```bash
-pip install "flytekit[sidecar,spark3,schema]"
-```
-
-Or install them with the `all` or `all-spark2.4` or `all-spark3` directives which will install all the plugins and a specific Spark version.
- Please note that `all` defaults to Spark 3.0 and Spark 2.4 support will be fully removed in a future release.
-
-
-```bash
-pip install "flytekit[all]"
-```
+### Plugins:
+Refer to [plugins/README.md](plugins/README.md) for a list of available
+plugins. There may be plugins outside of this list, but this list is maintained
+by the core maintainers.
 
 ## Development
+Guide to contribute and develop flytekit
 
 ### Recipes
 

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+SPHINXPROJ    = flytekit
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/make.bat

@@ -0,0 +1,36 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=source
+set BUILDDIR=build
+set SPHINXPROJ=simpleble
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.http://sphinx-doc.org/
+	exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
+
+:end
+popd

docs/source/conditions.rst

@@ -0,0 +1,9 @@
+###########################
+Branches and Conditionals
+###########################
+
+Branches and conditionals can be expressed explicitly in Flyte. These conditions are evaluated
+in the flyte engine and hence should be used for control flow. ``dynamic workflows`` can be used to perform custom conditional logic not supported by flytekit
+
+.. automodule:: flytekit.annotated.condition
+    :members:

docs/source/conf.py

@@ -0,0 +1,187 @@
+# -*- coding: utf-8 -*-
+#
+# Configuration file for the Sphinx documentation builder.
+#
+# This file does only contain a selection of the most common options. For a
+# full list see the documentation:
+# http://www.sphinx-doc.org/en/stable/config
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import os
+import sys
+
+sys.path.insert(0, os.path.abspath("../../flytekit/"))
+
+
+# -- Project information -----------------------------------------------------
+
+project = "Flyte Cookbook [Python]"
+copyright = "2021, Flyte"
+author = "Flyte"
+
+# The full version, including alpha/beta/rc tags
+release = "0.16.0b1"
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    "sphinx.ext.autodoc",
+    "sphinx.ext.autosummary",
+    "sphinx.ext.autosectionlabel",
+    "sphinx.ext.napoleon",
+    "sphinx.ext.todo",
+    "sphinx.ext.viewcode",
+    "sphinx.ext.doctest",
+    "sphinx.ext.intersphinx",
+    "sphinx.ext.coverage",
+    "sphinx-prompt",
+    "sphinx_copybutton",
+]
+
+# build the templated autosummary files
+autosummary_generate = True
+
+# autosectionlabel throws warnings if section names are duplicated.
+# The following tells autosectionlabel to not throw a warning for
+# duplicated section names that are in different documents.
+autosectionlabel_prefix_document = True
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ["_templates"]
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+#
+# source_suffix = ['.rst', '.md']
+source_suffix = ".rst"
+
+# The master toctree document.
+master_doc = "index"
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path .
+exclude_patterns = []
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = "sphinx"
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = "sphinx_material"
+html_theme_options = {
+    # Set the name of the project to appear in the navigation.
+    "nav_title": "Flyte",
+    # Set you GA account ID to enable tracking
+    "google_analytics_account": "G-YQL24L5CKY",
+    # Specify a base_url used to generate sitemap.xml. If not
+    # specified, then no sitemap will be built.
+    "base_url": "https://github.com/lyft/flytekit",
+    # Set the color and the accent color
+    "color_primary": "deep-purple",
+    "color_accent": "blue",
+    # Set the repo location to get a badge with stats
+    "repo_url": "https://github.com/lyft/flyte/",
+    "repo_name": "flyte",
+    # Visible levels of the global TOC; -1 means unlimited
+    "globaltoc_depth": 1,
+    # If False, expand all TOC entries
+    "globaltoc_collapse": False,
+    # If True, show hidden TOC entries
+    "globaltoc_includehidden": False,
+}
+
+# The default sidebars (for documents that don't match any pattern) are
+# defined by theme itself.  Builtin themes are using these templates by
+# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
+# 'searchbox.html']``.
+html_sidebars = {"**": ["logo-text.html", "globaltoc.html", "localtoc.html", "searchbox.html"]}
+
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ["_static"]
+
+# Custom sidebar templates, must be a dictionary that maps document names
+# to template names.
+#
+
+
+# -- Options for HTMLHelp output ---------------------------------------------
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = "flytekitdoc"
+
+
+# -- Options for LaTeX output ------------------------------------------------
+
+latex_elements = {
+    # The paper size ('letterpaper' or 'a4paper').
+    #
+    # 'papersize': 'letterpaper',
+    # The font size ('10pt', '11pt' or '12pt').
+    #
+    # 'pointsize': '10pt',
+    # Additional stuff for the LaTeX preamble.
+    #
+    # 'preamble': '',
+    # Latex figure (float) alignment
+    #
+    # 'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+#  author, documentclass [howto, manual, or own class]).
+latex_documents = [
+    (master_doc, "flytekit.tex", "Flytekit Documentation", "Flyte", "manual"),
+]
+
+
+# -- Options for manual page output ------------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [(master_doc, "flytekit", "Flytekit Documentation", [author], 1)]
+
+
+# -- Options for Texinfo output ----------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+    (
+        master_doc,
+        "flytekit",
+        "Flytekit Documentation",
+        author,
+        "flytekit",
+        "Python SDK for Flyte (https://flyte.org).",
+        "Miscellaneous",
+    ),
+]
+
+
+# -- Extension configuration -------------------------------------------------

docs/source/context.rst

@@ -0,0 +1,18 @@
+###################
+Context Parameters
+###################
+
+.. automodule:: flytekit
+
+
+Execution Parameters
+----------------------
+
+.. autoclass:: ExecutionParameters
+
+
+Retrieve Active context in a task
+----------------------------------
+
+.. autofunction:: current_context
+