Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update docs on Spark version #897

Merged
merged 4 commits into from
Oct 11, 2023
Merged

Update docs on Spark version #897

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2b5adcf
Bump to Spark 3.4 and update docs
Fokko Sep 29, 2023
0da818f
Update docker/Dockerfile
Fokko Sep 29, 2023
44118a3
Update README.md
Fokko Sep 29, 2023
fd3a6f4
Merge branch 'main' into fd-update-dockerfile
colin-rogers-dbt Oct 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,18 +26,20 @@ more information, consult [the docs](https://docs.getdbt.com/docs/profile-spark)

## Running locally
A `docker-compose` environment starts a Spark Thrift server and a Postgres database as a Hive Metastore backend.
Note: dbt-spark now supports Spark 3.1.1 (formerly on Spark 2.x).
Note: dbt-spark now supports Spark 3.3.2.

The following command would start two docker containers
```
The following command starts two docker containers:

```sh
docker-compose up -d
```

It will take a bit of time for the instance to start, you can check the logs of the two containers.
If the instance doesn't start correctly, try the complete reset command listed below and then try start again.

Create a profile like this one:

```
```yaml
spark_testing:
target: local
outputs:
Expand All @@ -60,7 +62,7 @@ Connecting to the local spark instance:

Note that the Hive metastore data is persisted under `./.hive-metastore/`, and the Spark-produced data under `./.spark-warehouse/`. To completely reset you environment run the following:

```
```sh
docker-compose down
rm -rf ./.hive-metastore/
rm -rf ./.spark-warehouse/
Expand Down