Add a "severity" for tests (#1005)

[beckjake]

Fixes #1005

Add an optional severity config for tests that defaults to ERROR.

A test with severity=WARN that fails looks like this:

Running with dbt=0.13.0
Found 1 models, 2 tests, 0 archives, 0 analyses, 95 macros, 0 operations, 1 seed files, 1 sources

18:28:20 | Concurrency: 4 threads (target='default2')
18:28:20 |
18:28:20 | 1 of 2 START test not_null_model_email............................... [RUN]
18:28:20 | 2 of 2 START test source_not_null_source_nulls_email................. [RUN]
18:28:20 | 1 of 2 WARN 2 not_null_model_email................................... [WARN 2 in 0.04s]
18:28:20 | 2 of 2 WARN 2 source_not_null_source_nulls_email..................... [WARN 2 in 0.04s]
18:28:20 |
18:28:20 | Finished running 2 tests in 0.57s.

Those trailing WARN 2s are yellow, like in source freshness.

A test with severity=WARN that fails with the strict flag set looks like this:

Running with dbt=0.13.0
Found 1 models, 2 tests, 0 archives, 0 analyses, 95 macros, 0 operations, 1 seed files, 1 sources

18:28:24 | Concurrency: 4 threads (target='default2')
18:28:24 |
18:28:24 | 1 of 2 START test not_null_model_email............................... [RUN]
18:28:24 | 2 of 2 START test source_not_null_source_nulls_email................. [RUN]
18:28:24 | 2 of 2 FAIL 2 source_not_null_source_nulls_email..................... [FAIL 2 in 0.04s]
18:28:24 | 1 of 2 FAIL 2 not_null_model_email................................... [FAIL 2 in 0.04s]
18:28:24 |
18:28:24 | Finished running 2 tests in 0.62s.

Those trailing FAIL 2s are red.

I also refactored some of the schema test parsing code and made a TestBuilder class to encapsulate some of the kind of hairy logic that's currently involved in the source/ref test split. I think that came out super nicely.

I think this is mostly done, just want to trigger Windows tests to make sure everything went ok there.

[drewbanin]

This is really cool - I like the work that you did around TestBuilders and the code works super well.

In playing around with this, I think I have some reservations about parsing the severity out of the test name. While it's really slick for string-type tests, I don't think it holds up for dict-type tests. Eg:

version: 2

models:
  - name: my_model
    columns:
      - name: id
        tests:
          - unique, severity=ERROR
          - relationships, severity=WARN:
              from: id
              to: ref('other_model')
              field: name

The unique example is great and intuitive, but the relationships example feels pretty strange to me. Having played around with this for a few minutes, I think I'd prefer supplying the severity config as an argument to the test, eg:

models:
  - name: my_model
    columns:
      - name: id
        tests:
          - unique:
              severity: ERROR
          - relationships:
              from: id
              to: ref('other_model')
              field: name
              severity: WARN

This will definitely mask any macro args named severity, which is unpleasant, but I think we could work around it by supporting args like __severity__, or dbt.severity, or similar if need be.

The - unique, severity=ERROR case is super nice, and a lot more pleasant than adding an arg to the unique test config. I don't think it would make sense to support both syntaxes, but I just want to make it clear that I need to sleep on this one.

Let's chat about what our options are here

[drewbanin]

I don't think this works as expected, the following spec compiles and runs whereas I would expect to see a compiler error:

models:
    - name: my_model
      columns:
          - name: id
            tests:
                - unique, invalid

[drewbanin]

One comment here, but otherwise super happy with where we ended up! I tested this a bunch and it's rock-solid -- I think a lot of folks are going to be excited to deploy this in their projects!