Merge pull request #1668 from fishtown-analytics/merge-0.14.1-into-lma

Merge 0.14.1 into louisa-may-alcott

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,41 @@
+---
+name: Bug report
+about: Report a bug or an issue you've found with dbt
+title: ''
+labels: bug, triage
+assignees: ''
+
+---
+
+### Describe the bug
+A clear and concise description of what the bug is. What command did you run? What happened?
+
+### Steps To Reproduce
+In as much detail as possible, please provide steps to reproduce the issue. Sample data that triggers the issue, example model code, etc is all very helpful here.
+
+### Expected behavior
+A clear and concise description of what you expected to happen.
+
+### Screenshots and log output
+If applicable, add screenshots or log output to help explain your problem.
+
+### System information
+**Which database are you using dbt with?**
+- [ ] postgres
+- [ ] redshift
+- [ ] bigquery
+- [ ] snowflake
+- [ ] other (specify: ____________)
+
+
+**The output of `dbt --version`:**
+```
+<output goes here>
+```
+
+**The operating system you're using:**
+
+**The output of `python --version`:**
+
+### Additional context
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for dbt
+title: ''
+labels: enhancement, triage
+assignees: ''
+
+---
+
+### Describe the feature
+A clear and concise description of what you want to happen.
+
+### Describe alternatives you've considered
+A clear and concise description of any alternative solutions or features you've considered.
+
+### Additional context
+Is this feature database-specific? Which database(s) is/are relevant? Please include any other relevant context here.
+
+### Who will this benefit?
+What kind of use case will this feature be useful for? Please be specific and provide examples, this will help us prioritize properly.

CHANGELOG.md

@@ -1,7 +1,40 @@
-## dbt 0.14.1
+## dbt 0.14.1 (Currently unreleased)
+
+### Overview
+
+This is primarily a bugfix release which contains a few minor improvements too. Note: this release includes an important change in how the `check` snapshot strategy works. See [#1614](https://github.com/fishtown-analytics/dbt/pull/1614) for more information. If you are using snapshots with the `check` strategy on dbt v0.14.0, it is strongly recommended that you upgrade to 0.14.1 at your soonest convenience.
 
 ### Breaking changes
- - the undocumented "graph" variable's "macros" field was removed from the parsing context ([#1615](https://github.com/fishtown-analytics/dbt/pull/1615))
+ - The undocumented `macros` attribute was removed from the `graph` context variable ([#1615](https://github.com/fishtown-analytics/dbt/pull/1615))
+
+### Features:
+ - Summarize warnings at the end of dbt runs ([#1597](https://github.com/fishtown-analytics/dbt/issues/1597), [#1654](https://github.com/fishtown-analytics/dbt/pull/1654))
+ - Speed up catalog generation on postgres by using avoiding use of the `information_schema` ([#1540](https://github.com/fishtown-analytics/dbt/pull/1540))
+ - Docs site updates ([#1621](https://github.com/fishtown-analytics/dbt/issues/1621))
+   - Fix for incorrect node selection logic in DAG view ([docs#38](https://github.com/fishtown-analytics/dbt-docs/pull/38))
+   - Update page title, meta tags, and favicon ([docs#39](https://github.com/fishtown-analytics/dbt-docs/pull/39))
+   - Bump the version of `dbt-styleguide`, changing file tree colors from orange to black :)
+ - Add environment variables for macro debugging flags ([#1628](https://github.com/fishtown-analytics/dbt/issues/1628), [#1629](https://github.com/fishtown-analytics/dbt/pull/1629))
+ - Speed up node selection by making it linear, rather than quadratic, in complexity ([#1611](https://github.com/fishtown-analytics/dbt/issues/1611), [#1615](https://github.com/fishtown-analytics/dbt/pull/1615))
+ - Specify the `application` field in Snowflake connections ([#1622](https://github.com/fishtown-analytics/dbt/issues/1622), [#1623](https://github.com/fishtown-analytics/dbt/pull/1623))
+
+### Fixes:
+ - Fix for reused `check_cols` values in snapshots ([#1614](https://github.com/fishtown-analytics/dbt/pull/1614))
+ - Fix for rendering column descriptions in sources ([#1619](https://github.com/fishtown-analytics/dbt/issues/1619), [#1633](https://github.com/fishtown-analytics/dbt/pull/1633))
+ - Fix for `is_incremental()` returning True for models that are not materialized as incremental models ([#1249](https://github.com/fishtown-analytics/dbt/issues/1249), [#1608](https://github.com/fishtown-analytics/dbt/pull/1608))
+ - Fix for serialization of BigQuery results which contain nested or repeated records ([#1626](https://github.com/fishtown-analytics/dbt/issues/1626), [#1638](https://github.com/fishtown-analytics/dbt/pull/1638))
+ - Fix for loading seed files which contain non-ascii characters ([#1632](https://github.com/fishtown-analytics/dbt/issues/1632), [#1644](https://github.com/fishtown-analytics/dbt/pull/1644))
+ - Fix for creation of user cookies in incorrect directories when `--profile-dir` or `$DBT_PROFILES_DIR` is provided ([#1645](https://github.com/fishtown-analytics/dbt/issues/1645), [#1656](https://github.com/fishtown-analytics/dbt/pull/1656))
+ - Fix for error handling when transactions are being rolled back ([#1647](https://github.com/fishtown-analytics/dbt/pull/1647))
+ - Fix for incorrect references to `dbt.exceptions` in jinja code ([#1569](https://github.com/fishtown-analytics/dbt/issues/1569), [#1609](https://github.com/fishtown-analytics/dbt/pull/1609))
+
+ ### Contributors:
+Thanks for your contributions to dbt!
+
+- [@levimalott](https://github.com/levimalott) ([#1647](https://github.com/fishtown-analytics/dbt/pull/1647))
+- [@aminamos](https://github.com/aminamos) ([#1609](https://github.com/fishtown-analytics/dbt/pull/1609))
+- [@elexisvenator](https://github.com/elexisvenator) ([#1540](https://github.com/fishtown-analytics/dbt/pull/1540))
+
 
 
 ## dbt 0.14.0 - Wilt Chamberlain (July 10, 2019)

core/dbt/adapters/base/connections.py

@@ -245,7 +245,13 @@ def commit(self):
     @classmethod
     def _rollback_handle(cls, connection):
         """Perform the actual rollback operation."""
-        connection.handle.rollback()
+        try:
+            connection.handle.rollback()
+        except Exception:
+            logger.debug(
+                'Failed to rollback {}'.format(connection.name),
+                exc_info=True
+            )
 
     @classmethod
     def _close_handle(cls, connection):

core/dbt/clients/agate_helper.py

@@ -1,6 +1,8 @@
 from codecs import BOM_UTF8
 
 import agate
+import json
+
 
 BOM = BOM_UTF8.decode('utf-8')  # '\ufeff'
 
@@ -31,6 +33,22 @@ def table_from_data(data, column_names):
         return table.select(column_names)
 
 
+def table_from_data_flat(data, column_names):
+    "Convert list of dictionaries into an Agate table"
+
+    rows = []
+    for _row in data:
+        row = []
+        for value in list(_row.values()):
+            if isinstance(value, (dict, list, tuple)):
+                row.append(json.dumps(value))
+            else:
+                row.append(value)
+        rows.append(row)
+
+    return agate.Table(rows, column_names)
+
+
 def empty_table():
     "Returns an empty Agate table. To be used in place of None"
 

core/dbt/clients/jinja.py

@@ -1,6 +1,7 @@
 import codecs
 import linecache
 import os
+import tempfile
 
 import jinja2
 import jinja2._compat
@@ -17,6 +18,34 @@
 from dbt.logger import GLOBAL_LOGGER as logger  # noqa
 
 
+def _linecache_inject(source, write):
+    if write:
+        # this is the only reliable way to accomplish this. Obviously, it's
+        # really darn noisy and will fill your temporary directory
+        tmp_file = tempfile.NamedTemporaryFile(
+            prefix='dbt-macro-compiled-',
+            suffix='.py',
+            delete=False,
+            mode='w+',
+            encoding='utf-8',
+        )
+        tmp_file.write(source)
+        filename = tmp_file.name
+    else:
+        filename = codecs.encode(os.urandom(12), 'hex').decode('ascii')
+
+    # encode, though I don't think this matters
+    filename = jinja2._compat.encode_filename(filename)
+    # put ourselves in the cache
+    linecache.cache[filename] = (
+        len(source),
+        None,
+        [line + '\n' for line in source.splitlines()],
+        filename
+    )
+    return filename
+
+
 class MacroFuzzParser(jinja2.parser.Parser):
     def parse_macro(self):
         node = jinja2.nodes.Macro(lineno=next(self.stream).lineno)
@@ -42,22 +71,16 @@ def _parse(self, source, name, filename):
 
     def _compile(self, source, filename):
         """Override jinja's compilation to stash the rendered source inside
-        the python linecache for debugging.
+        the python linecache for debugging when the appropriate environment
+        variable is set.
+
+        If the value is 'write', also write the files to disk.
+        WARNING: This can write a ton of data if you aren't careful.
         """
-        if filename == '<template>':
-            # make a better filename
-            filename = 'dbt-{}'.format(
-                codecs.encode(os.urandom(12), 'hex').decode('ascii')
-            )
-            # encode, though I don't think this matters
-            filename = jinja2._compat.encode_filename(filename)
-            # put ourselves in the cache
-            linecache.cache[filename] = (
-                len(source),
-                None,
-                [line + '\n' for line in source.splitlines()],
-                filename
-            )
+        macro_compile = os.environ.get('DBT_MACRO_DEBUGGING')
+        if filename == '<template>' and macro_compile:
+            write = macro_compile == 'write'
+            filename = _linecache_inject(source, write)
 
         return super()._compile(source, filename)
 

core/dbt/compilation.py

@@ -34,21 +34,21 @@ def _compiled_type_for(model: ParsedNode):
 
 def print_compile_stats(stats):
     names = {
-        NodeType.Model: 'models',
-        NodeType.Test: 'tests',
-        NodeType.Snapshot: 'snapshots',
-        NodeType.Analysis: 'analyses',
-        NodeType.Macro: 'macros',
-        NodeType.Operation: 'operations',
-        NodeType.Seed: 'seed files',
-        NodeType.Source: 'sources',
+        NodeType.Model: 'model',
+        NodeType.Test: 'test',
+        NodeType.Snapshot: 'snapshot',
+        NodeType.Analysis: 'analyse',
+        NodeType.Macro: 'macro',
+        NodeType.Operation: 'operation',
+        NodeType.Seed: 'seed file',
+        NodeType.Source: 'source',
     }
 
     results = {k: 0 for k in names.keys()}
     results.update(stats)
 
     stat_line = ", ".join(
-        ["{} {}".format(ct, names.get(t)) for t, ct in results.items()])
+        [dbt.utils.pluralize(ct, names.get(t)) for t, ct in results.items()])
 
     logger.notice("Found {}".format(stat_line))
 

core/dbt/config/profile.py

@@ -108,7 +108,6 @@ def __eq__(self, other):
         if not (isinstance(other, self.__class__) and
                 isinstance(self, other.__class__)):
             return False
-            return False
         return self.to_profile_info() == other.to_profile_info()
 
     def validate(self):

core/dbt/context/common.py

@@ -194,6 +194,13 @@ def call(name):
     return call
 
 
+def _debug_here():
+    import sys
+    import ipdb
+    frame = sys._getframe(3)
+    ipdb.set_trace(frame)
+
+
 def _add_sql_handlers(context):
     sql_results = {}
     return dbt.utils.merge(context, {
@@ -418,6 +425,8 @@ def generate_base(model, model_dict, config, manifest, source_config,
         "target": target,
         "try_or_compiler_error": try_or_compiler_error(model)
     })
+    if os.environ.get('DBT_MACRO_DEBUGGING'):
+        context['debug'] = _debug_here
 
     return context
 

core/dbt/contracts/graph/parsed.py

@@ -412,7 +412,7 @@ class ParsedSourceDefinition(
     resource_type: SourceType
     quoting: Quoting = field(default_factory=Quoting)
     loaded_at_field: Optional[str] = None
-    freshness: FreshnessThreshold = field(default_factory=FreshnessThreshold)
+    freshness: Optional[FreshnessThreshold] = None
     docrefs: List[Docref] = field(default_factory=list)
     description: str = ''
     columns: Dict[str, ColumnInfo] = field(default_factory=dict)

core/dbt/contracts/graph/unparsed.py

@@ -114,6 +114,9 @@ def status(self, age: float) -> FreshnessStatus:
         else:
             return FreshnessStatus.Pass
 
+    def __bool__(self):
+        return self.warn_after is not None or self.error_after is not None
+
 
 @dataclass
 class Quoting(JsonSchemaMixin, Mergeable):
@@ -127,7 +130,9 @@ class UnparsedSourceTableDefinition(ColumnDescription, NodeDescription):
     loaded_at_field: Optional[str] = None
     identifier: Optional[str] = None
     quoting: Quoting = field(default_factory=Quoting)
-    freshness: FreshnessThreshold = field(default_factory=FreshnessThreshold)
+    freshness: Optional[FreshnessThreshold] = field(
+        default_factory=FreshnessThreshold
+    )
 
     def __post_init__(self):
         NodeDescription.__post_init__(self)
@@ -142,7 +147,9 @@ class UnparsedSourceDefinition(JsonSchemaMixin, Replaceable):
     schema: Optional[str] = None
     loader: str = ''
     quoting: Quoting = field(default_factory=Quoting)
-    freshness: FreshnessThreshold = field(default_factory=FreshnessThreshold)
+    freshness: Optional[FreshnessThreshold] = field(
+        default_factory=FreshnessThreshold
+    )
     loaded_at_field: Optional[str] = None
     tables: List[UnparsedSourceTableDefinition] = field(default_factory=list)
 

core/dbt/contracts/results.py

@@ -47,6 +47,7 @@ class PartialResult(JsonSchemaMixin, Writable):
     thread_id: Optional[int] = 0
     timing: List[TimingInfo] = field(default_factory=list)
     fail: Optional[bool] = None
+    warn: Optional[bool] = None
 
     # if the result got to the point where it could be skipped/failed, we would
     # be returning a real result, not a partial.

core/dbt/contracts/util.py

@@ -15,8 +15,8 @@ def merged(self, *args):
         """
         replacements = {}
         cls = type(self)
-        for field in dataclasses.fields(cls):
-            for arg in args:
+        for arg in args:
+            for field in dataclasses.fields(cls):
                 value = getattr(arg, field.name)
                 if value is not None:
                     replacements[field.name] = value