All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
- dbt version detection: palm can now detect the version of dbt used in a project and will add the version to the plugin config. This can then be used to support changes between dbt versions.
- support for dbt's yaml selectors via the
--selector
option on relevant commands.
- Requires palm v2.6.0 or higher
- env vars are updated for projects running dbt v1.5.0 and later. This resolves deprecation warnings for people using more recent versions of dbt.
palm test
now supports the--defer
option, making it easy to run the tests for models you have changed in your current branch.
- All palm dbt commands have been updated to have a consistent interface with
palm run
. - Short options are added to many commands to improve usability.
dbt seed
command uses the proper decorator to pass the env object.
palm seed
command for working with dbt seeds
dbt seed
runs with--full-refresh
by default inpalm run
and other places.- PR template updated to make contributing easier
palm snapshot
command now persists by default, similar topalm run
- Automated release to pypi via GH actions
- Support for containerizing dbt projects with dbt v1+
- Validation of plugin config where prod manifest is not used
- Default dbt run options do not include stateful selection unless --defer is used.
- Support for Pyyaml v6 (removed upper version pin)
- Interactive column doc descriptions when generating a model-doc
- New dbt passthrough command. Running
palm dbt <command>
allows users to run bespoke or complex dbt commands without having to usepalm shell
for all those poweruser commands! - PluginConfig - the plugin is now configurable, currently this is used to store the path for local and production assets, enabling another new feature. note: this change requires palm v2.5.1 or higher
- Added a no-op command for downloading artifacts.
note to use
--defer
mode, you will need to override this function in your repo and implement the necessary logic to download production artifacts from your cloud provider.
-
run & test commands simplified and improved!
- Persist by default, no need to specify
--persist
on every run! - New
run defer
modes allow you to move more quickly, no need to explicitly select the model you've been working on! - New
run iterative
mode allows you to iterate over failures in your models quickly, without exiting and re-running the same command over-and-over. - Re-wrote some legacy code that made these operations more complicated than necessary.
- Persist by default, no need to specify
- Update model-doc command to populate existing columns when a docs_path config is present
- model-doc command: now supports advanced dbt project structures, parsese the dbt_project and includes CLI prompts to ensure model doc is created in the appropriate directory.
- dbt test no longer runs seed: palm test no longer runs seeds before executing tests. This was a strange decision and didn't really make any sense, caused slower dev/test cycles and added no value.
- model-doc: New command to auto-generate docs for a new model!
Palm can now generate .yml and .md files based on the SQL in your model,
to use palm model-doc run
palm model-doc models/path/to/model.sql
in your dbt project!
- dbt deps: palm-dbt now assumes dbt deps are installed when the docker image is built. Projects containerized by palm-dbt will be set up with this functionality, other projects may need to adjust their Dockerfile to RUN dbt deps and implement the volume mount in their docker-compose.yaml. See README for full details.
- pin version of black: The version of black used to lint this project has been pinned to ensure the local version matches the version used in CI
- Containerize: There is an explicit prompt to enter the dbt version number when containerizing a dbt project. Issue #34
- General: Improved testing, documentation, CI, issue templates, and linting.
- Bug Fixes: Removed profiles.yml reference in Dockerfile template. Palm-dbt detects profiles.yml automatically.
- Type hints were causing containerize to fail in python 3.8 and below. Issue #30
The plugin officially supports dbt back TWO minor versions (this will march forward), currently supporting back through v0.19.
- Containerize: NEW command for containerizing dbt,
containerize
, has been implemented! - Shell: NEW command to shell into the project container and execute arbitrary commands (run dbt directly!)
- Version validation: Added version detection and validation of supported dbt versions in a dbt project.
- Issue templates: Issue templates have been enabled for the repository!
- General: Removed hard-coded path names referring to Palmetto's own dbt project information and generalized pathing for code generation output.
- PEP8: Ran linting on the whole project.
- Fixed the enumeration of objects passed into the
--select
and--models
flags when passed to dbt.
- The palmcli package is being renamed to palm, updated dependency naming for compatibility
- Palm snapshot is missing dbt deps