Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add platform engineer tutorials to the docs #16123

Draft
wants to merge 16 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 16 additions & 3 deletions docs/mint.json
Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,22 @@
{
"group": "Tutorials",
"pages": [
"v3/tutorials/schedule",
"v3/tutorials/pipelines",
"v3/tutorials/scraping"
{
"group": "For data engineers",
"pages": [
"v3/tutorials/schedule",
"v3/tutorials/pipelines",
"v3/tutorials/scraping"
]
},
{
"group": "For platform engineers",
"pages": [
"v3/tutorials/platform",
"v3/tutorials/debug",
"v3/tutorials/alert"
]
}
],
"version": "v3"
},
Expand Down
Binary file added docs/v3/img/ui/deployments-page.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
12 changes: 12 additions & 0 deletions docs/v3/tutorials/alert.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
title: Alert on failure
description: Send alerts to Slack when flows fail to run
---

1. Continuing from [Debug a flow run](/v3/tutorials/debug), choose a flow and describe what a persistent failure looks like and how to automatically send alerts. Emphasize that alerts are a way to communicate and collaborate as a team instead of having to debug flows individually.
- If the user did not complete the quickstart, give them a repo they can clone and run to perform those steps quickly first.
2. Once the alert automation is set up, make a change to the flow (ideally in the UI using a parameter so that they don't have to manually edit code) and/or show how failing flows will trigger the alert.
3. Navigate to Slack and show how the alert appears in the channel.
4. Make another change to the parameters which resolves the alert, then re-run the flow.
5. Show that this change has resolved the condition which triggered the alert.
6. See if there's a way to send the resolution to Slack so that the team knows the issue is fixed.
14 changes: 14 additions & 0 deletions docs/v3/tutorials/debug.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
title: Debug a flow run
description: Find out when flow runs go wrong and how to troubleshoot them.
---

1. Continuing from [Set up a data platform](/v3/tutorials/platform), show how to view failing runs in the UI
- If the reader hasn't completed the quickstart, give them links to:
1. a Terraform repo they can clone and run to provision Prefect Cloud
2. a Prefect repo they can clone and run to populate their workspace with the flows needed demonstrate debugging (this can be the same as the one used in the operator quickstart).
2. Show them how to verify which flows failed, and how to go about debugging them.
- This will involve navigating to the run detail page.
3. Once they've fixed the issues, show them how to run the failed flows again manually and then verify that they ran successfully.

Next steps: [Alert on failure](/v3/tutorials/alert)
157 changes: 157 additions & 0 deletions docs/v3/tutorials/platform.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
---
title: Set up a data platform
description: Give your data team the ability to deploy and run workflows.
---

In this tutorial, you'll learn how to set up a data platform with Prefect Cloud.
We'll show you how to create teams, work pools, workers, and then deploy flows to the infrastructure you've provisioned.

## Prerequisites

To complete this tutorial, you'll need:

- Git
- Docker
- Python 3.9 or newer
- A forever free [Prefect Cloud](https://app.prefect.cloud/) account

You'll also need to clone the Prefect demo repository and install the Prefect SDK and CLI:

```bash
# Clone the demo repository
git clone https://github.com/PrefectHQ/demos.git
cd demos
```

```bash
# Create a virtual environment and install dependencies
python -m venv env # You may need to use `python3` instead
source env/bin/activate
pip install -r requirements.txt
```

```bash
# Show that the Prefect CLI is installed
prefect --version
```

## Create a workspace

To create a workspace on Prefect Cloud, you'll need a forever free account.

1. Head to https://app.prefect.cloud/ and sign in or create an account.
1. If you haven't created a workspace yet, you'll be prompted to choose a name for your workspace.

<Note>
You can also set up a self-managed instance with [Prefect server](/v3/manage/self-host).
</Note>

## Define your team structure

Most data platforms serve multiple teams.
In Prefect Cloud, you can create teams to control access to your infrastructure.

### Create teams

First, you need to create teams in your Prefect account.

1. Click the workspace switcher in the sidebar, and then click **Account settings**.
1. Click **Teams** in the sidebar.
1. Create two teams, named `team-a` and `team-b`.
1. Optional: Choose the team you want to add members to, and then click **Add team members**.

<Note>
Learn how to set up [user accounts](/v3/manage/cloud/manage-users/index), [service accounts](/v3/manage/cloud/manage-users/service-accounts), or [SSO](/v3/manage/cloud/manage-users/configure-sso).
</Note>

### Grant workspace access

You need to explicitly grant teams access to workspaces.

1. Click the workspace switcher in the sidebar, and then click **Invite and manage members**.
1. In the **Teams** section, add the two teams you created (`team-a` and `team-b`) with the **Developer** role.

Your teams now have developer access to your workspace.

## Provision infrastructure

Since this tutorial uses Docker, you must provision infrastructure before you can deploy and run flows on Prefect Cloud.

### Create Docker work pools

Work pools let you dynamically provision infrastructure for your flows.

1. Click **Work pools** in the sidebar.
1. Create a new **Docker** hybrid work pool, named `team-a-pool` and using the default configuration.
1. Repeat and create a second work pool named `team-b-pool`.

<Note>
See [Deploy overview](/v3/deploy/index) for options other than Docker.
</Note>

### Optional: Grant team access to work pools

By default, your teams have access to all work pools in the workspace.
However, you can partition your infrastructure by granting teams access to only specific work pools.

1. Click **Work pools** in the sidebar.
1. Open the actions menu for the `team-a-pool` work pool, and then click **Manage Access**.
1. Click **Add Users, Service Accounts, & Teams**.
1. Click the **Actors** dropdown, and then select the `team-a` team.
1. Click the **Access Level** dropdown, and then select **Run**.
1. Click **Add**.

Repeat these steps to grant `team-b` access to the `team-b-pool` work pool.
You've now set up your infrastructure so that `team-a` can only run flows on `team-a-pool` and `team-b` can only run flows on `team-b-pool`.

## Start Docker workers

Because you're using hybrid work pools, you need to start at least one worker for each work pool to process flow runs.
On your local machine, run two workers in the background:

```bash
prefect worker start --pool team-a-pool &
prefect worker start --pool team-b-pool &
```

## Deploy and run flows

Run the following script from the demo repository to create flow runs:

```bash
# Run the script to create deployments and flow runs
python simulate_failures.py
```

Wait a few minutes for the script to finish.
You will see multiple flow runs for each team you created.

<Note>
Check out the [Quickstart](/v3/get-started/quickstart) for how to write and deploy a flow from scratch.
</Note>

## View flow run activity

Now that you've deployed and run your flows, you can view the run details in the UI.
If you go to the **Deployments** page, you can see run activity broken out by deployment.

![Deployments page](/v3/img/ui/deployments-page.png)

Here's what happened:

- The script created two deployments of the flow, one deployment per work pool/team.
- The script then generated multiple flow runs for each deployment.
- The activity graph on the **Deployments** page shows successful runs as green and failed runs as red.
- Notice that the flow runs on the `team-a-pipeline` deployment succeeded, but some of the flow runs on the `team-b-pipeline` deployment failed.

This output is expected, you'll learn how to debug these failures in the next tutorial.

## Next steps

In this tutorial, you successfully set up a data platform for multiple teams, and then deployed and ran flows on the infrastructure you provisioned.

Next, learn how to [debug a flow run](/v3/tutorials/debug) when things go wrong.

<Tip>
Need help? [Book a meeting](https://calendly.com/prefect-experts/prefect-product-advocates?utm_campaign=prefect_docs_cloud&utm_content=prefect_docs&utm_medium=docs&utm_source=docs) with a Prefect Product Advocate to get your questions answered.
</Tip>