[Merged by Bors] - Connectors (beta) Overview and first pass through cdk

content/cli/connectors/_index.md

@@ -0,0 +1,6 @@
+---
+title: CONNECTORS 
+weight: 40
+_build:
+  render: never
+---

content/cli/connectors/cdk.md

@@ -0,0 +1,27 @@
+---
+title: Connector Development Kit (CDK)
+menu: CDK 
+weight: 10
+---
+
+Check out the [Connector Development Kit]({{<ref "/connectors-beta/cdk/overview.md">}}) page for more information.
+
+{{% inline-embed file="embeds/cli/help/cdk.md" %}}
+## `cdk build`
+{{% inline-embed file="embeds/cli/help/cdk-build.md" %}}
+## `cdk generate`
+{{% inline-embed file="embeds/cli/help/cdk-generate.md" %}}
+## `cdk test`
+{{% inline-embed file="embeds/cli/help/cdk-test.md" %}}
+## `cdk deploy`
+{{% inline-embed file="embeds/cli/help/cdk-deploy.md" %}}
+## `cdk deploy start`
+{{% inline-embed file="embeds/cli/help/cdk-deploy-start.md" %}}
+## `cdk deploy shutdown`
+{{% inline-embed file="embeds/cli/help/cdk-deploy-shutdown.md" %}}
+## `cdk deploy list`
+{{% inline-embed file="embeds/cli/help/cdk-deploy-list.md" %}}
+## `cdk deploy log`
+{{% inline-embed file="embeds/cli/help/cdk-deploy-log.md" %}}
+## `cdk publish`
+{{% inline-embed file="embeds/cli/help/cdk-publish.md" %}}

content/cli/connectors/hub.md

@@ -0,0 +1,70 @@
+---
+title: Connectors Hub
+menu: Hub
+weight: 20
+---
+
+## Introduction to the Connectors Hub Service
+
+The SmartConnectors Hub allows users to start building their data streaming pipeline with SmartConnectors which have been uploaded into the Hub as ready to use packages. The Hub allows Fluvio pipeline construction focusing on the business logic by providing already built, reliable and tested modules. The modules contain high level configurable inputs which can be applied without needing to rebuild the module. Pipelines can employ a mix of Hub downloaded packages as well as custom developed SmartModules.
+
+A view of available and installed SmartModules is accessible from both the `fluvio` cli as well as on the Fluvio cloud dashboard.
+
+
+## Fluvio Hub Commands
+
+Commands for users to interact with the SmartModule hub are under the **fluvio hub connector** set of commands.
+
+- `fluvio hub connector download group/package@version`
+- `fluvio hub connector list`
+
+### `fluvio hub connector download`
+
+{{% inline-embed file="embeds/cli/help/fluvio-hub-connector-download.md" %}}
+
+The command **fluvio hub connector download <group/package@version>** will download a hub SmartConnector `.ipkg` locally, which can be used with `cdk`.
+
+%copy first-line%
+```bash
+$ % fluvio hub connector download infinyon/[email protected]
+downloading infinyon/[email protected] to infinyon-mqtt-source-0.1.2.ipkg
+... downloading complete
+```
+
+### `fluvio hub connector list`
+
+The command **fluvio hub connector list** will list Connectors and versions available for download
+
+{{% inline-embed file="embeds/cli/help/fluvio-hub-connector-list.md" %}}
+
+The list command shows Connectors by **group/package@version** in the CONNECTOR column. The Visibility column describes if the package is accessible to public or private views.  If package visibility is **private** only your user login can list or download the package. Packages uploaded by InfinyOn will be in the infinyon group.
+
+
+%copy first-line%
+```bash
+$ fluvio hub connector list
+Using hub https://hub.infinyon.cloud
+  CONNECTOR                    Visibility 
+  infinyon/[email protected]   public     
+  infinyon/[email protected]    public     
+  infinyon/[email protected]  public     
+  infinyon/[email protected]   public     
+  infinyon/[email protected]      public
+```
+
+## Connector Packages
+
+Connector Packages are listed in the hub as in the form of `<group>/<package name>@version`. 
+
+Group `infinyon` packages are published by InfinyOn, the maker of Fluvio.  Third-party developers may also upload public packages which appear in other groups. The Hub service enforces signing and publishing restrictions on uploaded packages. Once a signed package of given group has been published, only the group owner may update that package or add new packages within that group.
+
+### Private and Public Packages
+
+Published packages are by default private. In order to make a published package publicly available, the package should be set to public. The cli method to do this is below:
+
+```bash
+cd connector-dev-dir
+cdk set-public
+```
+
+After the package is set to public, any publish will ask to verify the publish operation. Once a package version is publicly published, the package cannot revoked.

content/cli/smartmodules/hub.md

@@ -6,9 +6,9 @@ weight: 20
 
 ## Introduction to the SmartModule Hub Service
 
-The SmartModule Hub allows users to start building their data streaming pipline with SmartModules which have been uploaded into the Hub as ready to use packages. The Hub allows Fluvio pipeline construction focusing on the business logic by providing already built, reliable and tested modules. The modules contain high level configurable inputs which can be applied without needing to rebuild the module. Pipelines can employ a mix of Hub downloaded packages as well as custom developed SmartModules.
+The SmartModule Hub allows users to start building their data streaming pipeline with SmartModules which have been uploaded into the Hub as ready to use packages. The Hub allows Fluvio pipeline construction focusing on the business logic by providing already built, reliable and tested modules. The modules contain high level configurable inputs which can be applied without needing to rebuild the module. Pipelines can employ a mix of Hub downloaded packages as well as custom developed SmartModules.
 
-A view of available and installed SmartModules is accessible from both the fluvio cli as well as on the Fluvio cloud dashboard.
+A view of available and installed SmartModules is accessible from both the `fluvio` cli as well as on the Fluvio cloud dashboard.
 
 
 ## Fluvio Hub Commands
@@ -79,14 +79,14 @@ Group `infinyon` packages are published by InfinyOn, the maker of Fluvio.  Third
 
 ### Private and Public Packages
 
-Published packages are by default private. In order to make a published package publically available, the package should be set to public. The cli method to do this is below:
+Published packages are by default private. In order to make a published package publicly available, the package should be set to public. The cli method to do this is below:
 
 ```bash
 cd smartmodule-dev-dir
 smdk set-public
 ```
 
-After the package is set to public, any publish will ask to verify the publish operation. Once a package version is publically published, the package cannot revoked.
+After the package is set to public, any publish will ask to verify the publish operation. Once a package version is publicly published, the package cannot revoked.
 
 
 

content/connectors-beta/_index.md

@@ -9,3 +9,15 @@ toc: true
 This version of connectors is in Beta
 {{</caution>}}
 
+Smart Connectors make the process of importing or exporting data simple.
+You can import data with an `Inbound` connector and export data with an `Outbound` connector.
+
+Inbound and outbound connectors fundamentally work in the same way. The only difference is the direction your data is streaming with respect to a Fluvio topic.
+
+<img src="./images/smart-connectors-extra.svg"
+     alt="Smart Connectors"
+     style="justify: center; max-width: 600px" />
+
+* Learn more about the connector's 4 steps of transformation in [core concepts]({{<ref "core-concepts.md">}})
+* To get started with using connectors, check out the requirements first before browsing through one of our Inbound or Outbound connectors.
+* For the development of your own connectors, continue on to the [Connector Development Kit docs]({{<ref "/connectors-beta/cdk/overview" >}})

content/connectors-beta/cdk/_index.md

@@ -0,0 +1,6 @@
+---
+title: DEVELOPMENT 
+weight: 40
+_build:
+  render: never
+---

content/connectors-beta/cdk/build.md

@@ -0,0 +1,25 @@
+---
+title: Build & Test 
+weight: 40
+---
+### 1.1. Build
+
+Use `cdk build` to build a new connector:
+
+```bash
+$ cdk build --help
+Build the Connector in the current working directory
+
+Usage: cdk build [OPTIONS] [-- <EXTRA_ARGUMENTS>...]
+
+Arguments:
+  [EXTRA_ARGUMENTS]...  Extra arguments to be passed to cargo
+
+Options:
+      --release <RELEASE>            Release profile name [default: release]
+  -p, --package-name <PACKAGE_NAME>  Optional package/project name
+  -h, --help                         Print help
+```
+
+It compiles and builds a binary of the new Connector. The built binary is located in the default Cargo’s target directory.
+

content/connectors-beta/cdk/deploy.md

@@ -0,0 +1,4 @@
+---
+title:  Deploy 
+weight: 70
+---

content/connectors-beta/cdk/generate.md

@@ -0,0 +1,11 @@
+---
+title: Generate 
+weight: 30
+---
+
+Use `cdk generate` to create a new connector project:
+
+```jsx
+$ cdk generate my-connector
+```
+

content/connectors-beta/cdk/install.md

@@ -0,0 +1,39 @@
+---
+title: Install 
+weight: 20
+---
+
+### Install
+
+CDK is an independent executable downloadable via [`Fluvio CLI`](https://www.fluvio.io/cli/utilities/install/):
+
+```bash
+$ fluvio install cdk
+
+# Until it is officially released
+# fluvio install --hub cdk
+
+# Alternative, recommended only for dev purposes:
+# cargo install --git https://github.com/infinyon/fluvio cdk
+```
+
+The executable is installed in `~/.fluvio/bin` directory, the same directory as `fluvio`. The command should be immediately accessible at the prompt:
+
+```bash
+$ cdk
+
+Connector Development Kit
+
+Usage: cdk <COMMAND>
+
+Commands:
+  build       Build the Connector in the current working directory
+  test        Build and run the Connector in the current working directory
+  deploy      Deploy the Connector from the current working directory
+  publish     Publish Connector package to the Hub
+  set-public  Set connector visibility to public
+  help        Print this message or the help of the given subcommand(s)
+
+Options:
+  -h, --help  Print help
+```

content/connectors-beta/cdk/list.md

@@ -0,0 +1,4 @@
+---
+title: List 
+weight: 80
+---

content/connectors-beta/cdk/logs.md

@@ -0,0 +1,4 @@
+---
+title: Logs 
+weight: 90
+---

content/connectors-beta/cdk/overview.md

@@ -0,0 +1,18 @@
+---
+title: Overview 
+weight: 10
+---
+
+## Intro
+
+Fluvio Connector Development Kit (CDK) is a command-line tool whose primary goal is to help developers build, test, deploy SmartConnectors, and publish them to the Hub.
+
+[Source code](https://github.com/infinyon/fluvio/tree/master/crates/cdk)
+
+## Installation
+
+CDK only supports connectors developed in the Rust programming language. Other languages will be added in future releases.
+
+### Requirements
+
+Install Rust compiler and Cargo. See [https://www.rust-lang.org/tools/install](https://www.rust-lang.org/tools/install) for installation instructions.

content/connectors-beta/cdk/publish.md

@@ -0,0 +1,45 @@
+---
+title:  Publish 
+weight: 60
+---
+### 1.3. Publish
+
+The final step in this scenario would be publishing the SmartModule to the Hub - where other users can download and integrate it into their data pipelines.
+
+```bash
+$ cdk publish --help
+Publish Connector package to the Hub
+
+Usage: cdk publish [OPTIONS] [PACKAGE_META]
+
+Arguments:
+  [PACKAGE_META]
+
+Options:
+      --public-yes
+          don't ask for confirmation of public package publish
+
+      --pack
+          do only the pack portion
+
+      --push
+          given a packed file do only the push
+
+      --remote <REMOTE>
+
+  -h, --help
+```
+
+If run without arguments, it will pack everything needed into a package (`cdk build`must be executed beforehand) and push the package to the Hub.
+
+If you need to inspect the package before the push:
+
+```bash
+$ cdk publish --pack
+Package hub/http-source-0.1.0.ipkg created
+
+# check the file and then push. The file is a Tar Gzip archive.
+$ cdk publish --push hub/http-source-0.1.0.ipkg
+```
+
+`publish` command uses `hub/package-meta.yml` file with the metadata needed for the Hub. If it doesn’t exist, `cdk` creates it for you. Then, you can modify it, and the changes will be picked up on the subsequent command execution. 

content/connectors-beta/cdk/shutdown.md

@@ -0,0 +1,4 @@
+---
+title: Shutdown 
+weight: 100
+---

content/connectors-beta/cdk/test.md

@@ -0,0 +1,34 @@
+---
+title: Test 
+weight: 50
+---
+## 1.2. Test
+
+If the connector builds successfully, it’s time to `cdk test`:
+
+```bash
+$ cdk test --help
+Build and run the Connector in the current working directory
+
+Usage: cdk test [OPTIONS] --config <PATH> [-- <EXTRA_ARGUMENTS>...]
+
+Arguments:
+  [EXTRA_ARGUMENTS]...  Extra arguments to be passed to cargo
+
+Options:
+      --release <RELEASE>            Release profile name [default: release]
+  -p, --package-name <PACKAGE_NAME>  Optional package/project name
+  -c, --config <PATH>                Path to configuration file in YAML format
+  -s, --secrets <PATH>               Path to file with secrets. Secrets are 'key=value' pairs separated by the new line character. Optional
+  -h, --help                         Print help
+```
+
+The `test` command runs the connector’s binary (if it is not built, it builds it). 
+
+The `--config <PATH>` argument is required. It is a path to the configuration file in YAML format. 
+
+The configuration might have `secrets`. By default, `secrets` are resolved from environment variables unless a secrets file is provided via `-s, --secrets <PATH>` argument. The expected format of this file is `key=value` pairs separated by the new line character.
+
+SmartConnector output will be redirected to the current terminal output.
+
+To stop running SmartConnector in test mode, press Ctrl+C.