Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

updates to ReadMe to include Azure DevOps and dbt Cloud #46

Merged
merged 3 commits into from
Nov 5, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 27 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@
![test](https://github.com/cagov/caldata-infrastructure-template/actions/workflows/test.yml/badge.svg?branch=main)

This repository is a [copier](https://copier.readthedocs.io/en/stable/) template
which can be used to quickly seed a modern data stack project.
It consists of:
which can be used to quickly seed a modern data stack project. Instructions may vary depending
on if the repo is hosted via GitHub or Azure DevOps, so we make some distinctions below.

This repo consists of:

1. A [dbt](https://docs.getdbt.com/) project.
1. [pre-commit](https://pre-commit.com/) checks for enforcing code-quality.
Expand All @@ -24,9 +26,14 @@ Create a directory into which the project will be rendered:
```bash
mkdir <your-project-name>
cd <your-project-name>
git init
```

Create a new project using the copier command-line tool, with HTTPS:
Create a new project using the copier command-line tool, with...


### GitHub repo
HTTPS:
```bash
copier copy https://github.com/cagov/caldata-infrastructure-template .
```
Expand All @@ -35,12 +42,22 @@ OR with SSH:
copier copy [email protected]:cagov/caldata-infrastructure-template.git .
```

### Azure DevOps repo
Install git credential manager (with [Homebrew](https://brew.sh/) if on a mac, if on a windows you should have it by default.) Then run the following three commands:
britt-allen marked this conversation as resolved.
Show resolved Hide resolved

```bash
brew install git-credential-manager

git remote add origin <Azure DevOps repo url e.g. https://[email protected]/caldata-sandbox/mdsa-test/_git/mdsa-test>

copier copy https://github.com/cagov/caldata-infrastructure-template .
```

This will ask you a series of questions, the answers to which will be used to populate the project.

Once the project is rendered, you should initialize it as a git repository:

```bash
git init
git add .
git commit -m "Initial commit"
```
Expand All @@ -53,7 +70,12 @@ git add poetry.lock
git commit -m "Add poetry.lock"
```

### Snowflake setup
### dbt Cloud setup
For Azure DevOps repos you'll follow the instructions [here](https://docs.getdbt.com/docs/cloud/git/setup-azure#register-an-azure-ad-app).

For GitHub repos you'll follow the instructions [here](https://docs.getdbt.com/docs/cloud/git/connect-github).

### GitHub-based Snowflake setup

The projects generated from our infrastructure template need read access to the
Snowflake account in order to do two things from GitHub actions:
Expand Down
2 changes: 2 additions & 0 deletions {{project_name}}/transform/models/sample_data.sql.jinja
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
{% if execute %}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this if block needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

was experimenting to try to solve this error. the error message points me to a dbt article that makes use of such a block

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may actually be due to some of the work @ram-kishore-odi is doing in cagov/data-infrastructure#413. Our dev roles are currently a bit messed up, and that can break CI here.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes @britt-allen, Ian is right. terraform snowflake provider upgrade to v 0.97 is in progress and I am trying to wrap it up asap. But we are facing some unexpected situations which is causing the delays to upgrade dev.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So that error can be ignored for now then? @ian-r-rose @ram-kishore-odi

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think so.

select
randstr(uniform(10, 30, random(1)), uniform(1, 100000, random(1)))::varchar(
30
Expand All @@ -10,3 +11,4 @@ select
30
) as email
from table(generator(rowcount => 1000))
{% endif %}
Loading