diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md deleted file mode 100644 index 9207294b2e0..00000000000 --- a/.github/ISSUE_TEMPLATE.md +++ /dev/null @@ -1,34 +0,0 @@ -Please make sure to fill out either the issue template or the feature template and delete the other one! - -## Issue - -### Issue description -In general terms, please describe the issue. What command did you run? - -### Results -What happened? What did you expect to happen? - -### System information -The output of `dbt --version`: -``` - -``` - -The operating system you're running on: - -The python version you're using (probably the output of `python --version`) - -### Steps to reproduce -In as much detail as possible, please provide steps to reproduce the issue. Sample data that triggers the issue, example models, etc are all very helpful here. - - - - - -## Feature - -### Feature description -Please describe the feature you would like dbt to have. Please provide any details, relevant documentation links, StackOverflow links, etc here. - -### Who will this benefit? -What kind of use case will this feature be useful for? Please be specific and provide examples, this will help us prioritize properly. diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 00000000000..c65287e071a --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,41 @@ +--- +name: Bug report +about: Report a bug or an issue you've found with dbt +title: '' +labels: bug, triage +assignees: '' + +--- + +### Describe the bug +A clear and concise description of what the bug is. What command did you run? What happened? + +### Steps To Reproduce +In as much detail as possible, please provide steps to reproduce the issue. Sample data that triggers the issue, example model code, etc is all very helpful here. + +### Expected behavior +A clear and concise description of what you expected to happen. + +### Screenshots and log output +If applicable, add screenshots or log output to help explain your problem. + +### System information +**Which database are you using dbt with?** +- [ ] postgres +- [ ] redshift +- [ ] bigquery +- [ ] snowflake +- [ ] other (specify: ____________) + + +**The output of `dbt --version`:** +``` + +``` + +**The operating system you're using:** + +**The output of `python --version`:** + +### Additional context +Add any other context about the problem here. diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 00000000000..2a359196836 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,20 @@ +--- +name: Feature request +about: Suggest an idea for dbt +title: '' +labels: enhancement, triage +assignees: '' + +--- + +### Describe the feature +A clear and concise description of what you want to happen. + +### Describe alternatives you've considered +A clear and concise description of any alternative solutions or features you've considered. + +### Additional context +Is this feature database-specific? Which database(s) is/are relevant? Please include any other relevant context here. + +### Who will this benefit? +What kind of use case will this feature be useful for? Please be specific and provide examples, this will help us prioritize properly. diff --git a/CHANGELOG.md b/CHANGELOG.md index cca0d32d6fe..b94644aedac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,40 @@ -## dbt 0.14.1 +## dbt 0.14.1 (Currently unreleased) + +### Overview + +This is primarily a bugfix release which contains a few minor improvements too. Note: this release includes an important change in how the `check` snapshot strategy works. See [#1614](https://github.com/fishtown-analytics/dbt/pull/1614) for more information. If you are using snapshots with the `check` strategy on dbt v0.14.0, it is strongly recommended that you upgrade to 0.14.1 at your soonest convenience. ### Breaking changes - - the undocumented "graph" variable's "macros" field was removed from the parsing context ([#1615](https://github.com/fishtown-analytics/dbt/pull/1615)) + - The undocumented `macros` attribute was removed from the `graph` context variable ([#1615](https://github.com/fishtown-analytics/dbt/pull/1615)) + +### Features: + - Summarize warnings at the end of dbt runs ([#1597](https://github.com/fishtown-analytics/dbt/issues/1597), [#1654](https://github.com/fishtown-analytics/dbt/pull/1654)) + - Speed up catalog generation on postgres by using avoiding use of the `information_schema` ([#1540](https://github.com/fishtown-analytics/dbt/pull/1540)) + - Docs site updates ([#1621](https://github.com/fishtown-analytics/dbt/issues/1621)) + - Fix for incorrect node selection logic in DAG view ([docs#38](https://github.com/fishtown-analytics/dbt-docs/pull/38)) + - Update page title, meta tags, and favicon ([docs#39](https://github.com/fishtown-analytics/dbt-docs/pull/39)) + - Bump the version of `dbt-styleguide`, changing file tree colors from orange to black :) + - Add environment variables for macro debugging flags ([#1628](https://github.com/fishtown-analytics/dbt/issues/1628), [#1629](https://github.com/fishtown-analytics/dbt/pull/1629)) + - Speed up node selection by making it linear, rather than quadratic, in complexity ([#1611](https://github.com/fishtown-analytics/dbt/issues/1611), [#1615](https://github.com/fishtown-analytics/dbt/pull/1615)) + - Specify the `application` field in Snowflake connections ([#1622](https://github.com/fishtown-analytics/dbt/issues/1622), [#1623](https://github.com/fishtown-analytics/dbt/pull/1623)) + +### Fixes: + - Fix for reused `check_cols` values in snapshots ([#1614](https://github.com/fishtown-analytics/dbt/pull/1614)) + - Fix for rendering column descriptions in sources ([#1619](https://github.com/fishtown-analytics/dbt/issues/1619), [#1633](https://github.com/fishtown-analytics/dbt/pull/1633)) + - Fix for `is_incremental()` returning True for models that are not materialized as incremental models ([#1249](https://github.com/fishtown-analytics/dbt/issues/1249), [#1608](https://github.com/fishtown-analytics/dbt/pull/1608)) + - Fix for serialization of BigQuery results which contain nested or repeated records ([#1626](https://github.com/fishtown-analytics/dbt/issues/1626), [#1638](https://github.com/fishtown-analytics/dbt/pull/1638)) + - Fix for loading seed files which contain non-ascii characters ([#1632](https://github.com/fishtown-analytics/dbt/issues/1632), [#1644](https://github.com/fishtown-analytics/dbt/pull/1644)) + - Fix for creation of user cookies in incorrect directories when `--profile-dir` or `$DBT_PROFILES_DIR` is provided ([#1645](https://github.com/fishtown-analytics/dbt/issues/1645), [#1656](https://github.com/fishtown-analytics/dbt/pull/1656)) + - Fix for error handling when transactions are being rolled back ([#1647](https://github.com/fishtown-analytics/dbt/pull/1647)) + - Fix for incorrect references to `dbt.exceptions` in jinja code ([#1569](https://github.com/fishtown-analytics/dbt/issues/1569), [#1609](https://github.com/fishtown-analytics/dbt/pull/1609)) + + ### Contributors: +Thanks for your contributions to dbt! + +- [@levimalott](https://github.com/levimalott) ([#1647](https://github.com/fishtown-analytics/dbt/pull/1647)) +- [@aminamos](https://github.com/aminamos) ([#1609](https://github.com/fishtown-analytics/dbt/pull/1609)) +- [@elexisvenator](https://github.com/elexisvenator) ([#1540](https://github.com/fishtown-analytics/dbt/pull/1540)) + ## dbt 0.14.0 - Wilt Chamberlain (July 10, 2019) diff --git a/core/dbt/adapters/base/connections.py b/core/dbt/adapters/base/connections.py index b8ed37c3048..fecbf55bf36 100644 --- a/core/dbt/adapters/base/connections.py +++ b/core/dbt/adapters/base/connections.py @@ -245,7 +245,13 @@ def commit(self): @classmethod def _rollback_handle(cls, connection): """Perform the actual rollback operation.""" - connection.handle.rollback() + try: + connection.handle.rollback() + except Exception: + logger.debug( + 'Failed to rollback {}'.format(connection.name), + exc_info=True + ) @classmethod def _close_handle(cls, connection): diff --git a/core/dbt/clients/agate_helper.py b/core/dbt/clients/agate_helper.py index 189b2774692..8139a554853 100644 --- a/core/dbt/clients/agate_helper.py +++ b/core/dbt/clients/agate_helper.py @@ -1,6 +1,8 @@ from codecs import BOM_UTF8 import agate +import json + BOM = BOM_UTF8.decode('utf-8') # '\ufeff' @@ -31,6 +33,22 @@ def table_from_data(data, column_names): return table.select(column_names) +def table_from_data_flat(data, column_names): + "Convert list of dictionaries into an Agate table" + + rows = [] + for _row in data: + row = [] + for value in list(_row.values()): + if isinstance(value, (dict, list, tuple)): + row.append(json.dumps(value)) + else: + row.append(value) + rows.append(row) + + return agate.Table(rows, column_names) + + def empty_table(): "Returns an empty Agate table. To be used in place of None" diff --git a/core/dbt/clients/jinja.py b/core/dbt/clients/jinja.py index 4433726c6d6..e4fb6b0e932 100644 --- a/core/dbt/clients/jinja.py +++ b/core/dbt/clients/jinja.py @@ -1,6 +1,7 @@ import codecs import linecache import os +import tempfile import jinja2 import jinja2._compat @@ -17,6 +18,34 @@ from dbt.logger import GLOBAL_LOGGER as logger # noqa +def _linecache_inject(source, write): + if write: + # this is the only reliable way to accomplish this. Obviously, it's + # really darn noisy and will fill your temporary directory + tmp_file = tempfile.NamedTemporaryFile( + prefix='dbt-macro-compiled-', + suffix='.py', + delete=False, + mode='w+', + encoding='utf-8', + ) + tmp_file.write(source) + filename = tmp_file.name + else: + filename = codecs.encode(os.urandom(12), 'hex').decode('ascii') + + # encode, though I don't think this matters + filename = jinja2._compat.encode_filename(filename) + # put ourselves in the cache + linecache.cache[filename] = ( + len(source), + None, + [line + '\n' for line in source.splitlines()], + filename + ) + return filename + + class MacroFuzzParser(jinja2.parser.Parser): def parse_macro(self): node = jinja2.nodes.Macro(lineno=next(self.stream).lineno) @@ -42,22 +71,16 @@ def _parse(self, source, name, filename): def _compile(self, source, filename): """Override jinja's compilation to stash the rendered source inside - the python linecache for debugging. + the python linecache for debugging when the appropriate environment + variable is set. + + If the value is 'write', also write the files to disk. + WARNING: This can write a ton of data if you aren't careful. """ - if filename == '