Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs/update deploy dagster #1761

Merged
merged 22 commits into from
Sep 6, 2024
Merged
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -50,13 +50,7 @@ capabilities for handling data extraction and load and Dagster's orchestration f

### Orchestrating `dlt` pipeline on Dagster

Here's a concise guide to orchestrating a `dlt` pipeline with Dagster, creating a pipeline which ingests GitHub issues data from a repository and loading

using the project "Ingesting
GitHub issues data from a repository and storing it in BigQuery" as an example.

More details can be found in the article
[“Orchestrating unstructured data pipelines with dagster and dlt."](https://dagster.io/blog/dagster-dlt)
Here's a concise guide to orchestrating a `dlt` pipeline with Dagster, creating a pipeline which ingests GitHub issues data from a repository and loads it to DuckDB.

**The steps are as follows:**

Expand Down Expand Up @@ -102,13 +96,14 @@ More details can be found in the article
```
1. Create a dlt_assets definition:

The @dlt_assets decorator takes a dlt_source and dlt_pipeline parameter. In this example, we used the github_source source and created a dlt_pipeline to ingest data from Github to DuckDB.
The @dlt_assets decorator takes a 'dlt_source' and 'dlt_pipeline' parameter. In this example, we used the 'github_source' source and created a 'dlt_pipeline' to ingest data from Github to DuckDB.
mariarice15 marked this conversation as resolved.
Show resolved Hide resolved

Here’s an example of how to define assets (github_source/assets.py):

```py
from dagster import AssetExecutionContext
from dagster_embedded_elt.dlt import DagsterDltResource, dlt_assets
from dlt import pipeline
mariarice15 marked this conversation as resolved.
Show resolved Hide resolved
from .github_pipeline import github_source

@dlt_assets(
Expand All @@ -131,7 +126,7 @@ More details can be found in the article

1. Create the Definitions object

The last step is to include the assets and resource in a [Definitions](https://docs.dagster.io/_apidocs/definitions#dagster.Definitions) object (github_source/definitions.py). This enables Dagster tools to load everything we've defined:
The last step is to include the assets and resource in a [Definitions](https://docs.dagster.io/_apidocs/definitions#dagster.Definitions) object ('github_source/definitions.py'). This enables Dagster tools to load everything we have defined:

```py
from dagster import Definitions, load_assets_from_modules
Expand Down
Loading