diff --git a/docs/guide/advanced/local_compilation.mdx b/docs/guide/advanced/local_compilation.mdx new file mode 100644 index 00000000..64fca7a3 --- /dev/null +++ b/docs/guide/advanced/local_compilation.mdx @@ -0,0 +1,118 @@ +--- +title: "Local Compilation" +description: + "Compile your SDF workspace offline without running any queries against your database." +--- + +# Overview + +Do you have a need for speed? Is the Rust performance of SDF not enough for you? Do you want to compile your SDF project without running any queries against your database? If you answered yes to any of these questions, then you're in the right place! +This guide will show you how to compile your SDF project locally without running any queries against your database. The key to accomplishing this is hydrating your SDF workspace with schemas for the remote sources locally. + + + Managing a locally compilable SDF workspace is significantly more maintenance than a standard SDF workspace. We recommend only advanced users with a strong understanding of SDF and general software engineering principles attempt this. + + +# Architecture + +The architecture of a locally compilable SDF workspace is similar to a standard SDF workspace. The primary difference is that the locally compilable workspace will have a local copy of the schemas for the remote sources. This local copy of the schemas will be used to compile the workspace without running any queries against the database. +We recommend storing these in a directory called `sources` in the root of your workspace. As such, a typical directory structure for this might be: + +```shell +. +├── checks +├── macros +├── models +├── seeds +├── sources +├── tests +└── workspace.sdf.yml +``` + +For compilation speeds, we then recommend structuring your source YML declaration using the [catalog-schema-table-name](/guide/advanced/index) index. + +So if I had sources coming from a database called `financials`, in a schema `public`, the file structure would optimally look like: + +```shell +sources +└── financials + └── public + ├── raw_customers.sdf.yml + ├── raw_orderitems.sdf.yml + └── raw_orders.sdf.yml +``` + +# Source Declarations + +In order to compile locally, we need the column names and datatypes for the tables our queries are pulling data from (i.e. sources). +These can be declared as **Local Schema Files**. You can create a YML file for each source table in the `sources` directory. These files should contain the column names and datatypes for the source table. (example below) + + + Since these source declarations are just SDF yml files - descriptions, tests, and other metadata like classifiers can be easily stored alongside the schema. + + +Let's imagine we have a source table called `raw_customers` (as seen above) with columns like an `customerid` and `name`. An example of a schema for this source table might look like: + +```yml financials/public/raw_customers.sdf.yml +table: + name: raw_customers + origin: remote # Important! + columns: + - name: CUSTOMERID + datatype: decimal(38, 0) + - name: NAME + description: (SENSITIVE) Customer full name + datatype: varchar + - name: PHONE + description: (SENSITIVE) Customer phone number + datatype: varchar + - name: EMAIL + description: (SENSITIVE) Customer email + datatype: varchar + - name: ADDRESS + datatype: varchar + - name: REGION + datatype: varchar + - name: POSTALZIP + datatype: varchar + - name: COUNTRY + datatype: varchar + - name: CREATEDAT + datatype: timestamp + - name: UPDATEDAT + datatype: timestamp +``` + +Let's take note of a few things about this source declaration: + +- The `origin` field is set to `remote` to indicate that this is a remote source. **This is critical**, as SDF will still fetch the remote schema unless this attribute is set. +- The `columns` field contains a list of column objects, each with a `name` and `datatype` field. The `datatype` field must be a valid SQL datatype that corresponds to the column's datatype in the remote source. +- Other metadata like classifiers and descriptions can be added alongside the datatype. These will propagate downstream using our column-level lineage. + + + The easiest way to manually create these is by looking into the `sdftarget` after compiling use remote sources. In the example above, I can simply copy the SDF yml file produced from compilation located at + `sdftarget/dbg/table/financials/public/raw_customers.sdf.yml`. Note it's recommended to remove lots of extra generated metadata like the `purpose`, `case-policy`, `materialization`, and more if using this method. + + +We've seen users take a variety of methods to generate these YML files programmatically. The most common is to use a protobuf representation of the sources, likely generated from the production db (like Postgres), and convert that into SDF yml as a [pre-compile script](/guide/advanced/custom_scripts#reserved-script-keywords-pre-compile-pre-run-post-compile-and-post-run). + +# Incremental Models and Snapshots + +When SDF compiles incremental models and snapshots, the compilation results are often modified by the incremental or snapshot mode. By default, this mode is set by simply checking to see if this model exists in the remote database. Therefore, we need to overwrite this default behavior if we'd like to compile incremental models and snapshots entirely locally. + +This can be accomplished by passing in the flag `--prefer-local` to the `sdf compile` command. This flag will force SDF to compile the model entirely locally, without checking the remote database. +The `--prefer-local` flag works by simply setting the incremental mode and snapshot mode variables to true during compilation, thereby replacing the need to check the remote database. + +However, let's say we wanted to compile these models locally but with incremental and snapshot mode off. We can do this by passing extra parameters to the `sdf compile` command, specifically `--no-incremental-mode` and `--no-snapshot-mode`. + +Here are four examples and their expected results: + +1. `sdf compile --prefer-local`: All incremental models and snapshots will not require a database request to compile. They will compile with incremental mode to `true` and snapshot mode to `true` +2. `sdf compile --prefer-local --no-incremental-mode`: All incremental models and snapshots will not require a database request to compile. They will be compile with incremental mode to `false` and snapshot mode to `true` +3. `sdf compile --prefer-local --no-snapshot-mode`: All incremental models and snapshots will not require a database request to compile. They will be compile with incremental mode to `true` and snapshot mode to `false` +4. `sdf compile --prefer-local --no-incremental-mode --no-snapshot-mode`: All incremental models and snapshots will not require a database request to compile. They will be compile with incremental mode to `false` and snapshot mode to `false` + + + `--no-incremental-mode` and `--no-snapshot-mode` will not work to compile locally without `--prefer-local`. `--prefer-local` is required to prevent a request to the remote database. + + diff --git a/docs/guide/advanced/logging.mdx b/docs/guide/advanced/logging.mdx index f90a8b0b..f556185c 100644 --- a/docs/guide/advanced/logging.mdx +++ b/docs/guide/advanced/logging.mdx @@ -32,16 +32,16 @@ The following guide will go through the different logging options available in S 
 
-{"_ts":"2024-12-04T23:41:59.391363471+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/main.rs:105","ev":"sdf"}
-{"_ts":"2024-12-04T23:41:59.391410439+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/main.rs:106","ev":"sdf","set":"job_id","val_s":"61f53a8d-d8ea-4938-a805-74f6bdb0e778"}
-{"_ts":"2024-12-04T23:41:59.391418414+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/main.rs:107","ev":"sdf","set":"args","val_as":["compile","moms_flower_shop.raw.raw_inapp_events","--log-level","info","--show","none"]}
-{"_ts":"2024-12-04T23:41:59.392667+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/utils/initializing.rs:119","ev":"initialize"}
-{"_ts":"2024-12-04T23:41:59.392836297+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/utils/initializing.rs:298","ev":"initialize.load_workspace"}
-{"_ts":"2024-12-04T23:41:59.409612520+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/utils/initializing.rs:367","ev":"initialize.load_workspace","st_dur_ms":16}
-{"_ts":"2024-12-04T23:41:59.409625875+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/utils/initializing.rs:369","ev":"initialize.load_dependencies"}
-{"_ts":"2024-12-04T23:41:59.409632347+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/utils/initializing.rs:500","ev":"initialize.load_dependencies","st_dur_ms":0}
-{"_ts":"2024-12-04T23:41:59.409641263+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/utils/initializing.rs:502","ev":"initialize.load_prelude"}
-{"_ts":"2024-12-04T23:41:59.412179329+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/utils/initializing.rs:538","ev":"initialize.load_prelude","st_dur_ms":2}
+{"_ts":"2024-12-05T01:16:05.586827753+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/main.rs:105","ev":"sdf"}
+{"_ts":"2024-12-05T01:16:05.586879159+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/main.rs:106","ev":"sdf","set":"job_id","val_s":"6f931fec-bb45-4f7e-9139-e3e958d38acf"}
+{"_ts":"2024-12-05T01:16:05.586887555+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/main.rs:107","ev":"sdf","set":"args","val_as":["compile","moms_flower_shop.raw.raw_inapp_events","--log-level","info","--show","none"]}
+{"_ts":"2024-12-05T01:16:05.588098982+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/utils/initializing.rs:119","ev":"initialize"}
+{"_ts":"2024-12-05T01:16:05.588268888+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/utils/initializing.rs:298","ev":"initialize.load_workspace"}
+{"_ts":"2024-12-05T01:16:05.605455095+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/utils/initializing.rs:367","ev":"initialize.load_workspace","st_dur_ms":17}
+{"_ts":"2024-12-05T01:16:05.605468800+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/utils/initializing.rs:369","ev":"initialize.load_dependencies"}
+{"_ts":"2024-12-05T01:16:05.605475012+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/utils/initializing.rs:500","ev":"initialize.load_dependencies","st_dur_ms":0}
+{"_ts":"2024-12-05T01:16:05.605483518+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/utils/initializing.rs:502","ev":"initialize.load_prelude"}
+{"_ts":"2024-12-05T01:16:05.608032837+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/utils/initializing.rs:538","ev":"initialize.load_prelude","st_dur_ms":2}
@@ -63,58 +63,58 @@ The following guide will go through the different logging options available in S 
 
-{"_ts":"2024-12-04T23:42:01.097430282+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/utils/initializing.rs:577","ev":"initialize.prep_env.resolve_env.build_env","st_dur_ms":502}
-{"_ts":"2024-12-04T23:42:01.097498469+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/context/env.rs:907","message":"Reading jinja file: sdf_test at sdftarget/dbg/.lib/sdf_test/macros/test.jinja"}
-{"_ts":"2024-12-04T23:42:01.097765459+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/context/env.rs:907","message":"Reading jinja file: sdf_utils at sdftarget/dbg/.lib/sdf_utils/macros/date_spine.jinja"}
-{"_ts":"2024-12-04T23:42:01.097838997+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/context/env.rs:907","message":"Reading jinja file: sdf_utils at sdftarget/dbg/.lib/sdf_utils/macros/generate_integer_values.jinja"}
-{"_ts":"2024-12-04T23:42:01.097894641+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/context/env.rs:907","message":"Reading jinja file: sdf_utils at sdftarget/dbg/.lib/sdf_utils/macros/generate_surrogate_key.jinja"}
-{"_ts":"2024-12-04T23:42:01.097935508+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/context/env.rs:907","message":"Reading jinja file: sdf_utils at sdftarget/dbg/.lib/sdf_utils/macros/group_by.jinja"}
-{"_ts":"2024-12-04T23:42:01.097974481+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/context/env.rs:907","message":"Reading jinja file: sdf_utils at sdftarget/dbg/.lib/sdf_utils/macros/generate_date_values.jinja"}
-{"_ts":"2024-12-04T23:42:01.098013835+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/context/env.rs:907","message":"Reading jinja file: sdf_mat at sdftarget/dbg/.lib/sdf_mat/macros/dialect/trino/materialize.jinja"}
-{"_ts":"2024-12-04T23:42:01.098384659+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/utils/initializing.rs:776","ev":"initialize.prep_env.resolve_env","st_dur_ms":503}
-
-{"_ts":"2024-12-04T23:42:01.099044927+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:518","ev":"cmd.prepare.read_makefile","set":"verdict","val_s":"full-rebuild"}
-{"_ts":"2024-12-04T23:42:01.099062259+00:00","_ll":"DEBUG","_fn":"crates/sdf-cli/src/subcommands/build.rs:574","ev":"cmd.prepare.read_makefile","set":"analyze_with_sources","val_b":false}
-{"_ts":"2024-12-04T23:42:01.099070965+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:579","ev":"cmd.prepare.read_makefile","set":"resolved_table_names","val_s":"[]"}
-{"_ts":"2024-12-04T23:42:01.099077117+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:585","ev":"cmd.prepare.read_makefile","set":"seen_these_files","val_s":"[]"}
-{"_ts":"2024-12-04T23:42:01.099083459+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:591","ev":"cmd.prepare.read_makefile","set":"sources","val_s":"[]"}
-{"_ts":"2024-12-04T23:42:01.099088979+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:602","ev":"cmd.prepare.read_makefile","st_dur_ms":0}
-
-{"_ts":"2024-12-04T23:42:01.105666275+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:1605","ev":"cmd.schedule.prep"}
-{"_ts":"2024-12-04T23:42:01.105695791+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:1627","ev":"cmd.schedule","set":"deps","val_as":{"moms_flower_shop.raw.raw_inapp_events":[]}}
-{"_ts":"2024-12-04T23:42:01.105726659+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:1707","ev":"cmd.schedule.prep","st_dur_ms":0}
-{"_ts":"2024-12-04T23:42:01.105735585+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:1804","ev":"cmd.schedule","set":"resolved_tables","val_as":{}}
-{"_ts":"2024-12-04T23:42:01.105741647+00:00","_ll":"DEBUG","_fn":"crates/sdf-cli/src/subcommands/build.rs:1810","ev":"cmd.schedule","set":"sources_defined_by_providers","val_as":[]}
-{"_ts":"2024-12-04T23:42:01.105751145+00:00","_ll":"DEBUG","_fn":"crates/sdf-cli/src/subcommands/build.rs:1856","ev":"cmd.schedule","set":"to_be_downloaded","val_as":[]}
-{"_ts":"2024-12-04T23:42:01.105787683+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:2034","ev":"cmd.schedule.dag"}
-{"_ts":"2024-12-04T23:42:01.105797421+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2036","ev":"cmd.schedule.dag","set":"dag.deps","val_s":"{\"moms_flower_shop.raw.raw_inapp_events\":[]}"}
-{"_ts":"2024-12-04T23:42:01.105812790+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2050","ev":"cmd.schedule.dag","set":"dag.initial","val_s":"{\"moms_flower_shop.raw.raw_inapp_events\":[]}"}
-{"_ts":"2024-12-04T23:42:01.105847522+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2062","ev":"cmd.schedule.dag","set":"dag.cleaned","val_s":"{\"moms_flower_shop.raw.raw_inapp_events\":[]}"}
-{"_ts":"2024-12-04T23:42:01.105863081+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2068","ev":"cmd.schedule.dag","set":"missing_defs","val_s":"[]"}
-{"_ts":"2024-12-04T23:42:01.105884912+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2146","ev":"cmd.schedule.dag","set":"dag.resolved_targets","val_s":"[\"moms_flower_shop.raw.raw_inapp_events\"]"}
-{"_ts":"2024-12-04T23:42:01.105906856+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2174","ev":"cmd.schedule.dag","set":"dag.targeted","val_s":"{\"moms_flower_shop.raw.raw_inapp_events\":[]}"}
-{"_ts":"2024-12-04T23:42:01.105943054+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2233","ev":"cmd.schedule.dag","set":"dag.acyclic","val_s":"{\"moms_flower_shop.raw.raw_inapp_events\":[]}"}
-{"_ts":"2024-12-04T23:42:01.105951801+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2239","ev":"cmd.schedule.dag","set":"cycles","val_s":"[]"}
-{"_ts":"2024-12-04T23:42:01.105957661+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2245","ev":"cmd.schedule.dag","set":"cut_points","val_s":"[]"}
-{"_ts":"2024-12-04T23:42:01.105968171+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2251","ev":"cmd.schedule.dag","set":"order.initial","val_s":"[\"moms_flower_shop.raw.raw_inapp_events\"]"}
-{"_ts":"2024-12-04T23:42:01.105991655+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2520","ev":"cmd.schedule.dag","set":"showable_root_tables","val_s":"[]"}
-{"_ts":"2024-12-04T23:42:01.106013216+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2809","ev":"cmd.schedule.dag","set":"order.without_providers","val_s":"[\"moms_flower_shop.raw.raw_inapp_events\"]"}
-{"_ts":"2024-12-04T23:42:01.106020619+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:2835","ev":"cmd.schedule.dag","st_dur_ms":0}
-
-{"_ts":"2024-12-04T23:42:01.106031610+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:2842","ev":"cmd.do"}
-{"_ts":"2024-12-04T23:42:01.106046979+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2868","ev":"cmd.do","set":"order.without_providers_or_cached","val_s":"[\"moms_flower_shop.raw.raw_inapp_events\"]"}
-{"_ts":"2024-12-04T23:42:01.106055836+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:2885","ev":"cmd.do.sources"}
-
-{"_ts":"2024-12-04T23:42:01.106618557+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:4314","ev":"cmd.do.derived.wave_1.mat","ev_tb":"moms_flower_shop.raw.raw_inapp_events","ev_tb_catalog":"moms_flower_shop","ev_tb_schema":"raw","ev_tb_table":"raw_inapp_events"}
-{"_ts":"2024-12-04T23:42:01.107401277+00:00","_ll":"DEBUG","_fn":"crates/sdf-cli/src/subcommands/describe.rs:262","ev":"cmd.do.derived.wave_1.meta_analysis","ev_tb":"moms_flower_shop.raw.raw_inapp_events","ev_tb_catalog":"moms_flower_shop","ev_tb_schema":"raw","ev_tb_table":"raw_inapp_events"}
-{"_ts":"2024-12-04T23:42:01.107447463+00:00","_ll":"DEBUG","_fn":"crates/sdf-cli/src/subcommands/describe.rs:307","ev":"cmd.do.derived.wave_1.meta_analysis","ev_tb":"moms_flower_shop.raw.raw_inapp_events","ev_tb_catalog":"moms_flower_shop","ev_tb_schema":"raw","ev_tb_table":"raw_inapp_events","st_dur_ms":0}
-{"_ts":"2024-12-04T23:42:01.107466379+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:4438","ev":"cmd.do.derived.wave_1.mat","ev_tb":"moms_flower_shop.raw.raw_inapp_events","ev_tb_catalog":"moms_flower_shop","ev_tb_purpose":"model","ev_tb_schema":"raw","ev_tb_table":"raw_inapp_events","st_code":"succeeded","st_done":"executed","st_dur_ms":0}
-
-{"_ts":"2024-12-04T23:42:01.107503989+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:4298","ev":"cmd.do.derived","st_dur_ms":0}
-{"_ts":"2024-12-04T23:42:01.107859462+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/utils/dep_cache.rs:1888","message":"Saving table cache: moms_flower_shop.raw.raw_inapp_events, sdftarget/dbg/table/moms_flower_shop/raw/raw_inapp_events.sdf.yml"}
-{"_ts":"2024-12-04T23:42:01.108200303+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:3520","ev":"cmd.do.gen_makefile"}
-{"_ts":"2024-12-04T23:42:01.108441125+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/utils/dep_cache.rs:2298","ev":"written_makefile","set":"makefile","val_m":{"contents":{"cli_vars":"99914b932bd37a50b983c5e7c90ae93b","credentials":"99914b932bd37a50b983c5e7c90ae93b","env_vars":"d751713988987e9331980363e24189ce"},"prelude":[{"file":"workspace.sdf.yml","mtime":"2024-12-04T23:41:59.110736425Z"},{"file":"metadata/raw/raw_addresses.sdf.yml","mtime":"2024-12-04T23:41:59.106736402Z"},{"file":"metadata/raw/raw_customers.sdf.yml","mtime":"2024-12-04T23:41:59.106736402Z"},{"file":"metadata/staging/app_installs.sdf.yml","mtime":"2024-12-04T23:41:59.110736425Z"},{"file":"metadata/analytics/agg_installs_and_campaigns.sdf.yml","mtime":"2024-12-04T23:41:59.106736402Z"},{"file":"metadata/analytics/dim_marketing_campaigns.sdf.yml","mtime":"2024-12-04T23:41:59.106736402Z"},{"file":"metadata/raw/raw_marketing_campaign_events.sdf.yml","mtime":"2024-12-04T23:41:59.106736402Z"},{"file":"metadata/raw/raw_inapp_events.sdf.yml","mtime":"2024-12-04T23:41:59.106736402Z"},{"file":"metadata/staging/inapp_events.sdf.yml","mtime":"2024-12-04T23:41:59.110736425Z"},{"file":"classifications/column_classifiers.sdf.yml","mtime":"2024-12-04T23:41:59.106736402Z"},{"file":"classifications/table_classifiers.sdf.yml","mtime":"2024-12-04T23:41:59.106736402Z"},{"file":"metadata/raw/raw_addresses.sdf.yml","mtime":"2024-12-04T23:41:59.106736402Z"},{"file":"metadata/raw/raw_customers.sdf.yml","mtime":"2024-12-04T23:41:59.106736402Z"},{"file":"metadata/staging/app_installs.sdf.yml","mtime":"2024-12-04T23:41:59.110736425Z"},{"file":"metadata/analytics/agg_installs_and_campaigns.sdf.yml","mtime":"2024-12-04T23:41:59.106736402Z"},{"file":"metadata/analytics/dim_marketing_campaigns.sdf.yml","mtime":"2024-12-04T23:41:59.106736402Z"},{"file":"metadata/raw/raw_marketing_campaign_events.sdf.yml","mtime":"2024-12-04T23:41:59.106736402Z"},{"file":"metadata/raw/raw_inapp_events.sdf.yml","mtime":"2024-12-04T23:41:59.106736402Z"},{"file":"metadata/staging/inapp_events.sdf.yml","mtime":"2024-12-04T23:41:59.110736425Z"},{"file":"sdftarget/dbg/.lib/sdf_test/workspace.sdf.yml","mtime":"2024-12-04T23:42:00.590745080Z"},{"file":"sdftarget/dbg/.lib/sdf_utils/workspace.sdf.yml","mtime":"2024-12-04T23:42:00.590745080Z"},{"file":"sdftarget/dbg/.lib/sdf_mat/workspace.sdf.yml","mtime":"2024-12-04T23:42:00.590745080Z"},{"file":"sdftarget/dbg/.lib/sdf_test/macros/test.jinja","mtime":"2024-12-04T23:42:00.590745080Z"},{"file":"sdftarget/dbg/.lib/sdf_utils/macros/date_spine.jinja","mtime":"2024-12-04T23:42:00.590745080Z"},{"file":"sdftarget/dbg/.lib/sdf_utils/macros/generate_integer_values.jinja","mtime":"2024-12-04T23:42:00.590745080Z"},{"file":"sdftarget/dbg/.lib/sdf_utils/macros/generate_surrogate_key.jinja","mtime":"2024-12-04T23:42:00.590745080Z"},{"file":"sdftarget/dbg/.lib/sdf_utils/macros/group_by.jinja","mtime":"2024-12-04T23:42:00.590745080Z"},{"file":"sdftarget/dbg/.lib/sdf_utils/macros/generate_date_values.jinja","mtime":"2024-12-04T23:42:00.590745080Z"},{"file":"sdftarget/dbg/.lib/sdf_mat/macros/dialect/trino/materialize.jinja","mtime":"2024-12-04T23:42:00.590745080Z"}],"deps":[{"target":"moms_flower_shop.raw.raw_inapp_events","deps":[]}],"source_files":{"values":{"moms_flower_shop.raw.raw_inapp_events":[{"file":"metadata/raw/raw_inapp_events.sdf.yml","mtime":"2024-12-04T23:41:59.106736402Z"},{"file":"models/raw/raw_inapp_events.sql","mtime":"2024-12-04T23:41:59.110736425Z"},{"file":"seeds/parquet/inapp_events.parquet","mtime":"2024-12-04T23:41:59.110736425Z"}]}},"incomplete_source_files":{"values":[]},"target_status":{"values":{"moms_flower_shop.raw.raw_inapp_events":{"purpose":"model","status":"succeeded","draft":false}}}}}
-{"_ts":"2024-12-04T23:42:01.108599137+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:3534","ev":"cmd.do.gen_makefile","st_dur_ms":0}
+{"_ts":"2024-12-05T01:16:07.288835430+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/utils/initializing.rs:577","ev":"initialize.prep_env.resolve_env.build_env","st_dur_ms":505}
+{"_ts":"2024-12-05T01:16:07.288901553+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/context/env.rs:907","message":"Reading jinja file: sdf_test at sdftarget/dbg/.lib/sdf_test/macros/test.jinja"}
+{"_ts":"2024-12-05T01:16:07.289158933+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/context/env.rs:907","message":"Reading jinja file: sdf_utils at sdftarget/dbg/.lib/sdf_utils/macros/date_spine.jinja"}
+{"_ts":"2024-12-05T01:16:07.289231940+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/context/env.rs:907","message":"Reading jinja file: sdf_utils at sdftarget/dbg/.lib/sdf_utils/macros/generate_integer_values.jinja"}
+{"_ts":"2024-12-05T01:16:07.289275962+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/context/env.rs:907","message":"Reading jinja file: sdf_utils at sdftarget/dbg/.lib/sdf_utils/macros/generate_surrogate_key.jinja"}
+{"_ts":"2024-12-05T01:16:07.289317429+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/context/env.rs:907","message":"Reading jinja file: sdf_utils at sdftarget/dbg/.lib/sdf_utils/macros/group_by.jinja"}
+{"_ts":"2024-12-05T01:16:07.289357874+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/context/env.rs:907","message":"Reading jinja file: sdf_utils at sdftarget/dbg/.lib/sdf_utils/macros/generate_date_values.jinja"}
+{"_ts":"2024-12-05T01:16:07.289399762+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/context/env.rs:907","message":"Reading jinja file: sdf_mat at sdftarget/dbg/.lib/sdf_mat/macros/dialect/trino/materialize.jinja"}
+{"_ts":"2024-12-05T01:16:07.289766316+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/utils/initializing.rs:776","ev":"initialize.prep_env.resolve_env","st_dur_ms":506}
+
+{"_ts":"2024-12-05T01:16:07.290415145+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:518","ev":"cmd.prepare.read_makefile","set":"verdict","val_s":"full-rebuild"}
+{"_ts":"2024-12-05T01:16:07.290432438+00:00","_ll":"DEBUG","_fn":"crates/sdf-cli/src/subcommands/build.rs:574","ev":"cmd.prepare.read_makefile","set":"analyze_with_sources","val_b":false}
+{"_ts":"2024-12-05T01:16:07.290441535+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:579","ev":"cmd.prepare.read_makefile","set":"resolved_table_names","val_s":"[]"}
+{"_ts":"2024-12-05T01:16:07.290448167+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:585","ev":"cmd.prepare.read_makefile","set":"seen_these_files","val_s":"[]"}
+{"_ts":"2024-12-05T01:16:07.290454188+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:591","ev":"cmd.prepare.read_makefile","set":"sources","val_s":"[]"}
+{"_ts":"2024-12-05T01:16:07.290459187+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:602","ev":"cmd.prepare.read_makefile","st_dur_ms":0}
+
+{"_ts":"2024-12-05T01:16:07.297273351+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:1605","ev":"cmd.schedule.prep"}
+{"_ts":"2024-12-05T01:16:07.297305732+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:1627","ev":"cmd.schedule","set":"deps","val_as":{"moms_flower_shop.raw.raw_inapp_events":[]}}
+{"_ts":"2024-12-05T01:16:07.297339504+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:1707","ev":"cmd.schedule.prep","st_dur_ms":0}
+{"_ts":"2024-12-05T01:16:07.297350094+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:1804","ev":"cmd.schedule","set":"resolved_tables","val_as":{}}
+{"_ts":"2024-12-05T01:16:07.297356696+00:00","_ll":"DEBUG","_fn":"crates/sdf-cli/src/subcommands/build.rs:1810","ev":"cmd.schedule","set":"sources_defined_by_providers","val_as":[]}
+{"_ts":"2024-12-05T01:16:07.297365984+00:00","_ll":"DEBUG","_fn":"crates/sdf-cli/src/subcommands/build.rs:1856","ev":"cmd.schedule","set":"to_be_downloaded","val_as":[]}
+{"_ts":"2024-12-05T01:16:07.297403904+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:2034","ev":"cmd.schedule.dag"}
+{"_ts":"2024-12-05T01:16:07.297413803+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2036","ev":"cmd.schedule.dag","set":"dag.deps","val_s":"{\"moms_flower_shop.raw.raw_inapp_events\":[]}"}
+{"_ts":"2024-12-05T01:16:07.297428240+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2050","ev":"cmd.schedule.dag","set":"dag.initial","val_s":"{\"moms_flower_shop.raw.raw_inapp_events\":[]}"}
+{"_ts":"2024-12-05T01:16:07.297464730+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2062","ev":"cmd.schedule.dag","set":"dag.cleaned","val_s":"{\"moms_flower_shop.raw.raw_inapp_events\":[]}"}
+{"_ts":"2024-12-05T01:16:07.297479828+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2068","ev":"cmd.schedule.dag","set":"missing_defs","val_s":"[]"}
+{"_ts":"2024-12-05T01:16:07.297501618+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2146","ev":"cmd.schedule.dag","set":"dag.resolved_targets","val_s":"[\"moms_flower_shop.raw.raw_inapp_events\"]"}
+{"_ts":"2024-12-05T01:16:07.297519522+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2174","ev":"cmd.schedule.dag","set":"dag.targeted","val_s":"{\"moms_flower_shop.raw.raw_inapp_events\":[]}"}
+{"_ts":"2024-12-05T01:16:07.297554717+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2233","ev":"cmd.schedule.dag","set":"dag.acyclic","val_s":"{\"moms_flower_shop.raw.raw_inapp_events\":[]}"}
+{"_ts":"2024-12-05T01:16:07.297563754+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2239","ev":"cmd.schedule.dag","set":"cycles","val_s":"[]"}
+{"_ts":"2024-12-05T01:16:07.297569745+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2245","ev":"cmd.schedule.dag","set":"cut_points","val_s":"[]"}
+{"_ts":"2024-12-05T01:16:07.297581607+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2251","ev":"cmd.schedule.dag","set":"order.initial","val_s":"[\"moms_flower_shop.raw.raw_inapp_events\"]"}
+{"_ts":"2024-12-05T01:16:07.297605452+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2520","ev":"cmd.schedule.dag","set":"showable_root_tables","val_s":"[]"}
+{"_ts":"2024-12-05T01:16:07.297623025+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2809","ev":"cmd.schedule.dag","set":"order.without_providers","val_s":"[\"moms_flower_shop.raw.raw_inapp_events\"]"}
+{"_ts":"2024-12-05T01:16:07.297630509+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:2835","ev":"cmd.schedule.dag","st_dur_ms":0}
+
+{"_ts":"2024-12-05T01:16:07.297643894+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:2842","ev":"cmd.do"}
+{"_ts":"2024-12-05T01:16:07.297656818+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/subcommands/build.rs:2868","ev":"cmd.do","set":"order.without_providers_or_cached","val_s":"[\"moms_flower_shop.raw.raw_inapp_events\"]"}
+{"_ts":"2024-12-05T01:16:07.297665664+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:2885","ev":"cmd.do.sources"}
+
+{"_ts":"2024-12-05T01:16:07.298190102+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:4314","ev":"cmd.do.derived.wave_1.mat","ev_tb":"moms_flower_shop.raw.raw_inapp_events","ev_tb_catalog":"moms_flower_shop","ev_tb_schema":"raw","ev_tb_table":"raw_inapp_events"}
+{"_ts":"2024-12-05T01:16:07.299061835+00:00","_ll":"DEBUG","_fn":"crates/sdf-cli/src/subcommands/describe.rs:262","ev":"cmd.do.derived.wave_1.meta_analysis","ev_tb":"moms_flower_shop.raw.raw_inapp_events","ev_tb_catalog":"moms_flower_shop","ev_tb_schema":"raw","ev_tb_table":"raw_inapp_events"}
+{"_ts":"2024-12-05T01:16:07.299115365+00:00","_ll":"DEBUG","_fn":"crates/sdf-cli/src/subcommands/describe.rs:307","ev":"cmd.do.derived.wave_1.meta_analysis","ev_tb":"moms_flower_shop.raw.raw_inapp_events","ev_tb_catalog":"moms_flower_shop","ev_tb_schema":"raw","ev_tb_table":"raw_inapp_events","st_dur_ms":0}
+{"_ts":"2024-12-05T01:16:07.299133769+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:4438","ev":"cmd.do.derived.wave_1.mat","ev_tb":"moms_flower_shop.raw.raw_inapp_events","ev_tb_catalog":"moms_flower_shop","ev_tb_purpose":"model","ev_tb_schema":"raw","ev_tb_table":"raw_inapp_events","st_code":"succeeded","st_done":"executed","st_dur_ms":0}
+
+{"_ts":"2024-12-05T01:16:07.299179204+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:4298","ev":"cmd.do.derived","st_dur_ms":1}
+{"_ts":"2024-12-05T01:16:07.299535119+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/utils/dep_cache.rs:1888","message":"Saving table cache: moms_flower_shop.raw.raw_inapp_events, sdftarget/dbg/table/moms_flower_shop/raw/raw_inapp_events.sdf.yml"}
+{"_ts":"2024-12-05T01:16:07.299827683+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:3520","ev":"cmd.do.gen_makefile"}
+{"_ts":"2024-12-05T01:16:07.300044307+00:00","_ll":"TRACE","_fn":"crates/sdf-cli/src/utils/dep_cache.rs:2298","ev":"written_makefile","set":"makefile","val_m":{"contents":{"cli_vars":"99914b932bd37a50b983c5e7c90ae93b","credentials":"99914b932bd37a50b983c5e7c90ae93b","env_vars":"d751713988987e9331980363e24189ce"},"prelude":[{"file":"workspace.sdf.yml","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"metadata/raw/raw_customers.sdf.yml","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"metadata/staging/inapp_events.sdf.yml","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"metadata/analytics/agg_installs_and_campaigns.sdf.yml","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"metadata/raw/raw_inapp_events.sdf.yml","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"metadata/raw/raw_marketing_campaign_events.sdf.yml","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"metadata/raw/raw_addresses.sdf.yml","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"metadata/analytics/dim_marketing_campaigns.sdf.yml","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"metadata/staging/app_installs.sdf.yml","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"classifications/column_classifiers.sdf.yml","mtime":"2024-12-05T01:16:05.254445432Z"},{"file":"classifications/table_classifiers.sdf.yml","mtime":"2024-12-05T01:16:05.254445432Z"},{"file":"metadata/raw/raw_customers.sdf.yml","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"metadata/staging/inapp_events.sdf.yml","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"metadata/analytics/agg_installs_and_campaigns.sdf.yml","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"metadata/raw/raw_inapp_events.sdf.yml","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"metadata/raw/raw_marketing_campaign_events.sdf.yml","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"metadata/raw/raw_addresses.sdf.yml","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"metadata/analytics/dim_marketing_campaigns.sdf.yml","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"metadata/staging/app_installs.sdf.yml","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"sdftarget/dbg/.lib/sdf_test/workspace.sdf.yml","mtime":"2024-12-05T01:16:06.778445018Z"},{"file":"sdftarget/dbg/.lib/sdf_utils/workspace.sdf.yml","mtime":"2024-12-05T01:16:06.778445018Z"},{"file":"sdftarget/dbg/.lib/sdf_mat/workspace.sdf.yml","mtime":"2024-12-05T01:16:06.778445018Z"},{"file":"sdftarget/dbg/.lib/sdf_test/macros/test.jinja","mtime":"2024-12-05T01:16:06.778445018Z"},{"file":"sdftarget/dbg/.lib/sdf_utils/macros/date_spine.jinja","mtime":"2024-12-05T01:16:06.778445018Z"},{"file":"sdftarget/dbg/.lib/sdf_utils/macros/generate_integer_values.jinja","mtime":"2024-12-05T01:16:06.778445018Z"},{"file":"sdftarget/dbg/.lib/sdf_utils/macros/generate_surrogate_key.jinja","mtime":"2024-12-05T01:16:06.778445018Z"},{"file":"sdftarget/dbg/.lib/sdf_utils/macros/group_by.jinja","mtime":"2024-12-05T01:16:06.778445018Z"},{"file":"sdftarget/dbg/.lib/sdf_utils/macros/generate_date_values.jinja","mtime":"2024-12-05T01:16:06.778445018Z"},{"file":"sdftarget/dbg/.lib/sdf_mat/macros/dialect/trino/materialize.jinja","mtime":"2024-12-05T01:16:06.778445018Z"}],"deps":[{"target":"moms_flower_shop.raw.raw_inapp_events","deps":[]}],"source_files":{"values":{"moms_flower_shop.raw.raw_inapp_events":[{"file":"metadata/raw/raw_inapp_events.sdf.yml","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"models/raw/raw_inapp_events.sql","mtime":"2024-12-05T01:16:05.258445429Z"},{"file":"seeds/parquet/inapp_events.parquet","mtime":"2024-12-05T01:16:05.258445429Z"}]}},"incomplete_source_files":{"values":[]},"target_status":{"values":{"moms_flower_shop.raw.raw_inapp_events":{"purpose":"model","status":"succeeded","draft":false}}}}}
+{"_ts":"2024-12-05T01:16:07.300191633+00:00","_ll":"INFO","_fn":"crates/sdf-cli/src/subcommands/build.rs:3534","ev":"cmd.do.gen_makefile","st_dur_ms":0}
@@ -172,7 +172,7 @@ The following guide will go through the different logging options available in S 
 
 Working set 12 model files, 1 seed file, 19 .sdf files
-   Finished 1 model [1 succeeded] in 0.830 secs
+   Finished 1 model [1 succeeded] in 0.848 secs
 
 Schema moms_flower_shop.pub.compile_info
 ┌───────────────┬────────────────┬────────────┬─────────────┐
@@ -213,22 +213,22 @@ Schema moms_flower_shop.pub.compile_info
 
 Working set 13 model files, 1 seed file, 19 .sdf files
     Running moms_flower_shop.pub.query (./sdftarget/dbg/query.sql)
-   Finished 2 models [2 succeeded] in 0.832 secs
+   Finished 2 models [2 succeeded] in 0.840 secs
 
 Table moms_flower_shop.pub.query
 ┌─────────────────────────────────────┬──────────────────────────────┬────────────────┐
 │ timestamp                           ┆ event                        ┆ duration_in_ms │
 ╞═════════════════════════════════════╪══════════════════════════════╪════════════════╡
-│ 2024-12-04T23:42:02.585712418+00:00 ┆ sdf                          ┆                │
-│ 2024-12-04T23:42:02.585905330+00:00 ┆ sdf                          ┆                │
-│ 2024-12-04T23:42:02.586066432+00:00 ┆ sdf                          ┆                │
-│ 2024-12-04T23:42:02.587368592+00:00 ┆ initialize                   ┆                │
-│ 2024-12-04T23:42:02.587574367+00:00 ┆ initialize.load_workspace    ┆                │
-│ 2024-12-04T23:42:02.604758984+00:00 ┆ initialize.load_workspace    ┆ 17             │
-│ 2024-12-04T23:42:02.604870082+00:00 ┆ initialize.load_dependencies ┆                │
-│ 2024-12-04T23:42:02.604948819+00:00 ┆ initialize.load_dependencies ┆ 0              │
-│ 2024-12-04T23:42:02.605039409+00:00 ┆ initialize.load_prelude      ┆                │
-│ 2024-12-04T23:42:02.607689085+00:00 ┆ initialize.load_prelude      ┆ 2              │
+│ 2024-12-05T01:16:08.783577852+00:00 ┆ sdf                          ┆                │
+│ 2024-12-05T01:16:08.783746656+00:00 ┆ sdf                          ┆                │
+│ 2024-12-05T01:16:08.783910772+00:00 ┆ sdf                          ┆                │
+│ 2024-12-05T01:16:08.785207820+00:00 ┆ initialize                   ┆                │
+│ 2024-12-05T01:16:08.785415828+00:00 ┆ initialize.load_workspace    ┆                │
+│ 2024-12-05T01:16:08.802536268+00:00 ┆ initialize.load_workspace    ┆ 17             │
+│ 2024-12-05T01:16:08.802656862+00:00 ┆ initialize.load_dependencies ┆                │
+│ 2024-12-05T01:16:08.802736060+00:00 ┆ initialize.load_dependencies ┆ 0              │
+│ 2024-12-05T01:16:08.802827731+00:00 ┆ initialize.load_prelude      ┆                │
+│ 2024-12-05T01:16:08.805477351+00:00 ┆ initialize.load_prelude      ┆ 2              │
 └─────────────────────────────────────┴──────────────────────────────┴────────────────┘
 60 rows, showing only 10 rows.
   Run with --limit 0 to show all rows.
diff --git a/docs/guide/basics/build_and_deployment.mdx b/docs/guide/basics/build_and_deployment.mdx
new file mode 100644
index 00000000..ed075056
--- /dev/null
+++ b/docs/guide/basics/build_and_deployment.mdx
@@ -0,0 +1,259 @@
+---
+title: "Build & Deploy"
+description:
+  "Learn how to run pipelines in production while preserving data quality with SDF build"
+---
+
+# Overview
+
+This guide will walk you through the process of building and deploying your pipelines in production with SDF. Although this can be simple as running `sdf run` in your orchestrator, we strongly encourage users to leverage 
+the SDF build process to ensure that the data quality is preserved in production.
+
+Critically, build stages your data first, then tests it, and only if tests pass overwrites the production data. This is the best way to run models in production, since it ensures production data only gets updated if all data
+quality checks pass.
+
+
+  SDF build abstracts the write, audit, publish paradigm. You can think of this like a blue-green deployment in software engineering. Simply put, models are first _written_ to a temporary location, then _audited_ for data quality, and finally _published_ to the production location.
+  In an optimal scenario, the publish step should require no extra compute resources, since the data has already been processed and validated. For more on this, check out our [blog post on WAP](...).
+
+
+# Guide
+
+
+  
+    Let's start by creating a new sample workspace based on the Mom's Flower Shop example. This time, we'll use the completed version (not the one designed for the tutorial) so that the tests are already in place.
+
+    ```shell
+      sdf new --sample moms_flower_shop_completed
+    ```
+
+   
+  
+  
+    Great! Now that we have our workspace, let's start by running the tests. This will run the models, then test them against the assertions we've defined in the workspace.
+
+    ```shell
+      sdf test
+    ```
+
+

+
+
+Working set 12 model files, 1 test file, 27 .sdf files
+    Running moms_flower_shop.raw.raw_inapp_events (./models/raw/raw_inapp_events.sql)
+    Running moms_flower_shop.staging.inapp_events (./models/staging/inapp_events.sql)
+    Testing moms_flower_shop.staging.test_inapp_events (./sdftarget/dbg/tests/moms_flower_shop/staging/test_inapp_events.sql)
+   Finished 2 models [2 succeeded], 1 test [1 passed] in 0.971 secs
+[Pass] Test moms_flower_shop.staging.test_inapp_events
+
+
+

+

+    
+    Great! Our tests passed. However, for the sake of this demo let's introduce a bug that fails one of our data quality checks. 
+
+    The model `inapp_events` contains a column `event_value` with the following assertion tests applied to it:
+  
+    ```yml metadata/staging/inapp_events.sdf.yml
+      columns:
+        - name: event_value
+          tests:
+            - expect: valid_scalar("""event_value >= 0""")
+              severity: error
+            - expect: minimum(0)
+              severity: error
+    ```
+
+    So, let's say we intended to modify this column to subtract one, but instead subtracted the value _from_ one by mistake. In order to do this, update `inapp_events.sql` to:
+
+    ```sql models/staging/inapp_events.sql
+      SELECT 
+        event_id,
+        customer_id,
+        FROM_UNIXTIME(event_time/1000) AS event_time,  
+        event_name,
+        1 - event_value AS event_value,
+        additional_details,
+        platform,
+        campaign_id
+    FROM raw.raw_inapp_events
+    ```
+
+    {/* TODO (Elias): Add back once local compute warning is gone */}
+
+    {/* ```add sql
+      sdf_build_moms_flower/v1.inapp_events.sql -> moms_flower_shop_completed/models/staging/inapp_events.sql
+    ``` */}
+
+    Now, let's run the tests again, but this time turn off warnings so any errors are clear:
+
+    ```shell
+      sdf test
+    ```
+
+    {/* ```run quiet
+      cd tmp/moms_flower_shop_completed && $sdf test
+    ``` */}
+
+    Uh oh, both tests failed _and_ the model's data was updated with these faulty values. In an incremental scenario, this might result in some painful backfilling. Furthermore, in a production scenario, downstream 
+    data consumers like dashboards might now be displaying this faulty data. Let's move on and show how `sdf build` would prevent this scenario from happening.
+  
+
+  
+    So, we have a workspace with a failing model. Let's imagine this is a production scenario and we're about to deploy this pipeline. We'd want to make sure this test passes before we overwrite the production data.
+
+    This is where `sdf build` comes in. When we run build, SDF will stage the data by materializing it with `_draft` appended to its table name, then run tests on that table, and publish them to overwrite production only if the tests pass. Let's try it out.
+
+    ```shell
+      sdf build
+    ```
+
+    {/* TODO (Elias): Add back once local compute warning is gone */}
+    {/* ```run quiet
+      cd tmp/moms_flower_shop_completed && $sdf build
+    ``` */}
+
+    Nice! As you can see SDF prints out a clear `-- hard stop --` indicating:
+
+      1. None of the downstream models of `inapp_events` were run since this model failed the quality checks.
+      2. `inapp_events` was not overwritten itself, meaning any downstream dependencies like dashboards are only pulling slightly stale data, as opposed to bad data.
+    
+    Let's patch up this error by updating the `inapp_events.sql` file to our originally intended change:
+
+    ```sql models/staging/inapp_events.sql
+      SELECT 
+        event_id,
+        customer_id,
+        FROM_UNIXTIME(event_time/1000) AS event_time,  
+        event_name,
+        ABS(event_value - 1) as event_value,
+        additional_details,
+        platform,
+        campaign_id
+      FROM raw.raw_inapp_events
+    ```
+
+
+    Now, let's rerun build again, this time with a fully passing workspace:
+
+    ```shell
+      sdf build
+    ```
+
+

+
+
+Working set 12 model files, 1 test file, 27 .sdf files
+    Staging moms_flower_shop.raw.raw_addresses (./models/raw/raw_addresses.sql)
+    Staging moms_flower_shop.raw.raw_inapp_events (./models/raw/raw_inapp_events.sql)
+    Staging moms_flower_shop.raw.raw_marketing_campaign_events (./models/raw/raw_marketing_campaign_events.sql)
+    Staging moms_flower_shop.raw.raw_customers (./models/raw/raw_customers.sql)
+    Staging moms_flower_shop.staging.marketing_campaigns (./models/staging/marketing_campaigns.sql)
+    Staging moms_flower_shop.staging.inapp_events (./models/staging/inapp_events.sql)
+    Testing moms_flower_shop.staging.test_inapp_events (./sdftarget/dbg/tests/moms_flower_shop/staging/test_inapp_events.sql)
+    Staging moms_flower_shop.staging.app_installs (./models/staging/app_installs.sql)
+    Staging moms_flower_shop.staging.app_installs_v2 (./models/staging/app_installs_v2.sql)
+    Staging moms_flower_shop.analytics.agg_installs_and_campaigns (./models/analytics/agg_installs_and_campaigns.sql)
+    Staging moms_flower_shop.staging.stg_installs_per_campaign (./models/staging/stg_installs_per_campaign.sql)
+    Staging moms_flower_shop.staging.customers (./models/staging/customers.sql)
+    Staging moms_flower_shop.analytics.dim_marketing_campaigns (./models/analytics/dim_marketing_campaigns.sql)
+ Publishing 12 models, 1 tests
+   Finished 12 models [12 succeeded], 1 test [1 passed] in 1.056 secs
+[Pass] Test moms_flower_shop.staging.test_inapp_events
+
+
+

+

+
+    Great! Our pipeline built successfully, and we can be confident that our data is correct. 
+  
+
+  
+    Many people find it useful to build a specific model or a directory of models that might represent a DAG. In orchestration scenarios that use something like Airflow, this is a common pattern.
+    This can be accomplished simply by passing the model name or directory path to the `sdf build` command and combining it with the `--targets-only` flag. For more on target specification, check out our [IO Guide](/guide/setup/io#command-line-options).
+
+    For example, to build only the `inapp_events` model, you can run:
+
+    ```shell
+      sdf build --targets-only models/staging/inapp_events.sql
+    ```
+
+

+
+
+Working set 12 model files, 1 test file, 19 .sdf files
+ Publishing 2 models, 1 tests
+   Finished 2 models [2 reused], 1 test [1 passed (reused)] in 0.831 secs
+[Pass] Test moms_flower_shop.staging.test_inapp_events
+
+
+

+

+
+    This will build all models upstream of `inapp_events` (like `raw_inapp_events` as seen in the output) and then build `inapp_events` itself. Think of this like a build system, where we are building all the dependencies of a target before building the target itself.
+
+    In an Airflow orchestration scenario, you might manually specify that one DAG needs to run before another that depends on it. Then when running the second DAG, you'd be duplicating work by running the first DAG again. 
+    To avoid this and only run the DAG in a specific workspace, you can use the `--targets-only` flag with a directory path. For example, if we wanted to build models in a directory `models/my_dag`, we could run:
+
+    ```shell
+      sdf build --targets-only models/my_dag
+    ```
+
+    In this build scenario, SDF will stage all models in the `models/my_dag` directory first, then run tests against all of them, and only if all tests pass on all models in the DAG will it publish and overwrite the production data.
+
+    
+      The `--targets-only` flag is only functional in remote compute scenarios (i.e. while using SDF with Snowflake, BigQuery, Redshift, etc.). In local compute scenarios, the `--targets-only` flag builds models in the specified directory _and_ their upstream dependencies
+      since the local compute environment needs the results of upstream queries in memory in order to run the target query.
+    
+  
+
+
+# FAQ
+
+
+  
+    These three commands, though similar, have different purposes.
+
+    - `sdf run`: This command is used to run the pipeline. It will materialize the models specified and update the data in the production location. No tests are run with `sdf run`.
+    - `sdf test`: This command is used to run the tests. It will run the models specified, then test them against the assertions defined in the workspace. Notably the tests run _after_ the model has been materialized and data is updated. 
+    - `sdf build`: This command stages the data, runs the tests, then publishes the data to production only if the tests pass (i.e. WAP pattern). This is the best way to run models in production, since it ensures production data only gets updated if all data quality checks pass.
+
+    
+      In an ideal scenario, the `sdf build` command should be used in production scenarios to ensure data quality is preserved.
+    
+  
+  
+    The publish step leverages a transaction in the remote compute of choice that does two things:
+
+      1. Drops the staging table.
+      2. Renames the `_draft` table to the production table name using an `ALTER TABLE` statement.
+
+    This transaction ensures that the data is atomically updated, without rerunning the compute. If the publish step fails, the data is not updated and the transaction is rolled back.
+  
+  
+    Yes, `sdf build` will work with incremental models and snapshots, and will behave the same as `sdf run` in relation to the cache. 
+
+    Furthermore, in an incremental scenario, `sdf build` will only run the incremental computation, yet still test the full history of all increments. It works by:
+
+    1. Zero-copy cloning the incremental model to `_draft`
+    2. Running the incremental computation on top of the draft
+    3. Running tests against the draft, which contains all past increment
+    4. Publishing the draft to the production table if all tests pass
+  
+  
+    The simplest way to deploy SDF in Airflow while preserving data quality is to use the `sdf build` command on a cadence. If you have a more complex scenario where you need to run certain groups of models as DAGs in a specific order, you can use the `--targets-only` flag to build a specific model or directory of models. For example, you could write three Airflow DAGs with a simple BashOperator that runs:
+
+    1. `sdf build --targets-only models/dag1`
+    2. `sdf build --targets-only models/dag2`
+    3. `sdf build --targets-only models/dag3`
+
+    This does not need any other dependencies specific within the Airflow DAG, since SDF will handle the dependency resolution for you. Furthermore, this ensures dag1 runs before dag2, and dag2 runs before dag3.
+  
+  
+    The best way to deploy SDF in Dagster is to use the `sdf build` command in tandem with our Dagster integration. For more on how to set that up, check out our [Dagster integration guide](/integrations/dagster/getting-started).
+  
+
+
+# Conclusion
+
+In this guide, we learned how to build and deploy pipelines in production with SDF. We saw how the `sdf build` command can be used to stage data, run tests, and publish data to production only if the tests pass. This is a critical step in ensuring data quality in production and is a best practice for running models in production.
diff --git a/docs/guide/data-quality/checks.mdx b/docs/guide/data-quality/checks.mdx
index c67c1663..0617e3d9 100644
--- a/docs/guide/data-quality/checks.mdx
+++ b/docs/guide/data-quality/checks.mdx
@@ -36,7 +36,7 @@ Let's write and modify a check, starting with one of the sdf samples.
         
         📚 Read the Docs to Get Started -> https://docs.sdf.com/
         
-   Finished new in 0.279 secs
+   Finished new in 0.302 secs
@@ -78,7 +78,7 @@ Working set 1 model file, 1 .sdf file   Compiling hello.pub.main (./models/main.sql) Working set 1 check file, 1 .sdf file    Checking hello.pub.code_check (./checks/code_check.sql) -   Finished 1 model [1 succeeded], 1 check [1 passed] in 1.371 secs +   Finished 1 model [1 succeeded], 1 check [1 passed] in 1.386 secs [Pass] Check hello.pub.code_check @@ -122,7 +122,7 @@ Working set 1 model file, 1 .sdf file   Compiling hello.pub.main (./models/main.sql) Working set 1 check file, 1 .sdf file    Checking hello.pub.code_check (./checks/code_check.sql) -   Finished 1 model [1 succeeded], 1 check [1 failed] in 1.361 secs, for details see below. +   Finished 1 model [1 succeeded], 1 check [1 failed] in 1.413 secs, for details see below. [Fail] Check hello.pub.code_check ┌────────────┬─────────────┬─────────────┐ @@ -133,7 +133,7 @@ Working set 1 check file, 1 .sdf file 1 rows. ------- -Summary 1 model [1 succeeded], 1 check [1 failed] in 1.362 secs. +Summary 1 model [1 succeeded], 1 check [1 failed] in 1.413 secs. ------- @@ -195,7 +195,7 @@ Working set 2 model files, 1 .sdf file   Compiling hello.pub.sink (./models/sink.sql) Working set 1 check file, 1 .sdf file    Checking hello.pub.code_check (./checks/code_check.sql) -   Finished 2 models [2 succeeded], 1 check [1 passed] in 1.386 secs +   Finished 2 models [2 succeeded], 1 check [1 passed] in 1.451 secs [Pass] Check hello.pub.code_check diff --git a/docs/guide/data-quality/reports.mdx b/docs/guide/data-quality/reports.mdx index fa7fe5be..77f0dc72 100644 --- a/docs/guide/data-quality/reports.mdx +++ b/docs/guide/data-quality/reports.mdx @@ -50,7 +50,7 @@ Let's write and modify a report, starting with one of the sdf samples.                  📚 Read the Docs to Get Started -> https://docs.sdf.com/          -   Finished new in 0.265 secs +   Finished new in 0.300 secs @@ -115,17 +115,17 @@ Working set 12 model files, 3 .sdf files   Compiling payment.public.invoices (./ddls/payment/public/invoices.sql)   Compiling payment.public.organizations (./ddls/payment/public/organizations.sql)   Compiling transformations.internal.avg_invoice_amt (./models/internal/avg_invoice_amt.sql) -  Compiling transformations.external.org_invoice_stats (./models/external/org_invoice_stats.sql) -  Compiling transformations.internal.invoice_payment_delay (./models/internal/invoice_payment_delay.sql) -  Compiling transformations.internal.total_revenue_per_org (./models/internal/total_revenue_per_org.sql) -  Compiling transformations.internal.users_per_domain (./models/internal/users_per_domain.sql)   Compiling transformations.external.invoice_stats (./models/external/invoice_stats.sql) +  Compiling transformations.internal.invoice_payment_delay (./models/internal/invoice_payment_delay.sql)   Compiling transformations.internal.mau_per_org (./models/internal/mau_per_org.sql) +  Compiling transformations.internal.total_revenue_per_org (./models/internal/total_revenue_per_org.sql) +  Compiling transformations.external.org_invoice_stats (./models/external/org_invoice_stats.sql)   Compiling transformations.internal.most_frequent_payer (./models/internal/most_frequent_payer.sql) +  Compiling transformations.internal.users_per_domain (./models/internal/users_per_domain.sql)   Compiling transformations.internal.users_per_org (./models/internal/users_per_org.sql) Working set 1 report file, 1 .sdf file   Reporting sdf.reports.tables_with_pii (./reports/tables_with_pii.sql) -   Finished 12 models [12 succeeded], 1 report [1 succeeded] in 1.930 secs +   Finished 12 models [12 succeeded], 1 report [1 succeeded] in 1.952 secs Report sdf.reports.TABLES_WITH_PII ┌──────────────────────────────────────────────┬─────────────────────────┬───────────┐ @@ -133,8 +133,8 @@ Report sdf.reports.TABLES_WITH_PII ╞══════════════════════════════════════════════╪═════════════════════════╪═══════════╡ │ TRANSFORMATIONS.INTERNAL.MOST_FREQUENT_PAYER ┆                         ┆ snowflake │ │ payment.public.USERS                         ┆ DDL for the users table ┆ snowflake │ -│ TRANSFORMATIONS.INTERNAL.USERS_PER_DOMAIN    ┆                         ┆ snowflake │ │ TRANSFORMATIONS.EXTERNAL.INVOICE_STATS       ┆                         ┆ snowflake │ +│ TRANSFORMATIONS.INTERNAL.USERS_PER_DOMAIN    ┆                         ┆ snowflake │ └──────────────────────────────────────────────┴─────────────────────────┴───────────┘ 4 rows. @@ -167,19 +167,19 @@ Report sdf.reports.TABLES_WITH_PII Working set 12 model files, 3 .sdf files Working set 1 report file, 1 .sdf file   Reporting sdf.reports.tables_with_pii (./reports/tables_with_pii.sql) -   Finished 12 models [12 reused], 1 report [1 succeeded] in 1.638 secs +   Finished 12 models [12 reused], 1 report [1 succeeded] in 1.687 secs Report sdf.reports.TABLES_WITH_PII ┌──────────────────────────────────────────────┬──────────────┬─────────────────────────┬───────────┐ │ table_id                                     ┆ column_name  ┆ description             ┆ dialect   │ ╞══════════════════════════════════════════════╪══════════════╪═════════════════════════╪═══════════╡ +│ TRANSFORMATIONS.EXTERNAL.INVOICE_STATS       ┆ NAME         ┆                         ┆ snowflake │ +│ TRANSFORMATIONS.INTERNAL.USERS_PER_DOMAIN    ┆ EMAIL_DOMAIN ┆                         ┆ snowflake │ │ TRANSFORMATIONS.INTERNAL.MOST_FREQUENT_PAYER ┆ EMAIL        ┆                         ┆ snowflake │ │ payment.public.USERS                         ┆ NAME         ┆ DDL for the users table ┆ snowflake │ -│ TRANSFORMATIONS.INTERNAL.USERS_PER_DOMAIN    ┆ EMAIL_DOMAIN ┆                         ┆ snowflake │ -│ TRANSFORMATIONS.EXTERNAL.INVOICE_STATS       ┆ NAME         ┆                         ┆ snowflake │ -│ TRANSFORMATIONS.EXTERNAL.INVOICE_STATS       ┆ EMAIL        ┆                         ┆ snowflake │ │ payment.public.USERS                         ┆ EMAIL        ┆ DDL for the users table ┆ snowflake │ │ payment.public.USERS                         ┆ PHONE        ┆ DDL for the users table ┆ snowflake │ +│ TRANSFORMATIONS.EXTERNAL.INVOICE_STATS       ┆ EMAIL        ┆                         ┆ snowflake │ └──────────────────────────────────────────────┴──────────────┴─────────────────────────┴───────────┘ 7 rows. @@ -208,19 +208,19 @@ Report sdf.reports.TABLES_WITH_PII Working set 12 model files, 3 .sdf files Working set 2 report files, 1 .sdf file   Reporting sdf.reports.pii_datatypes (./reports/pii_datatypes.sql) -   Finished 12 models [12 reused], 2 reports [1 succeeded, 1 reused] in 1.679 secs +   Finished 12 models [12 reused], 2 reports [1 succeeded, 1 reused] in 1.707 secs Report sdf.reports.TABLES_WITH_PII ┌──────────────────────────────────────────────┬──────────────┬─────────────────────────┬───────────┐ │ table_id                                     ┆ column_name  ┆ description             ┆ dialect   │ ╞══════════════════════════════════════════════╪══════════════╪═════════════════════════╪═══════════╡ -│ TRANSFORMATIONS.EXTERNAL.INVOICE_STATS       ┆ EMAIL        ┆                         ┆ snowflake │ │ payment.public.USERS                         ┆ EMAIL        ┆ DDL for the users table ┆ snowflake │ │ payment.public.USERS                         ┆ PHONE        ┆ DDL for the users table ┆ snowflake │ +│ TRANSFORMATIONS.EXTERNAL.INVOICE_STATS       ┆ EMAIL        ┆                         ┆ snowflake │ +│ TRANSFORMATIONS.EXTERNAL.INVOICE_STATS       ┆ NAME         ┆                         ┆ snowflake │ +│ TRANSFORMATIONS.INTERNAL.USERS_PER_DOMAIN    ┆ EMAIL_DOMAIN ┆                         ┆ snowflake │ │ TRANSFORMATIONS.INTERNAL.MOST_FREQUENT_PAYER ┆ EMAIL        ┆                         ┆ snowflake │ │ payment.public.USERS                         ┆ NAME         ┆ DDL for the users table ┆ snowflake │ -│ TRANSFORMATIONS.INTERNAL.USERS_PER_DOMAIN    ┆ EMAIL_DOMAIN ┆                         ┆ snowflake │ -│ TRANSFORMATIONS.EXTERNAL.INVOICE_STATS       ┆ NAME         ┆                         ┆ snowflake │ └──────────────────────────────────────────────┴──────────────┴─────────────────────────┴───────────┘ 7 rows. diff --git a/docs/guide/data-quality/tests.mdx b/docs/guide/data-quality/tests.mdx index 4aa5ab61..f01030d5 100644 --- a/docs/guide/data-quality/tests.mdx +++ b/docs/guide/data-quality/tests.mdx @@ -26,7 +26,7 @@ When `sdf test` runs this query, it will check if any data is returned and fail Working set 1 model file, 1 test file, 1 .sdf file     Running hello.pub.main (./models/main.sql)     Testing hello.pub.test_main (./sdftarget/dbg/tests/hello/pub/test_main.sql) -   Finished 1 model [1 succeeded], 1 test [1 passed] in 0.854 secs +   Finished 1 model [1 succeeded], 1 test [1 passed] in 0.858 secs [Pass] Test hello.pub.test_main @@ -180,7 +180,7 @@ table: Working set 1 model file, 1 test file, 2 .sdf files     Running tests_workspace.pub.raw_inapp_events (./models/raw_inapp_events.sql)     Testing tests_workspace.pub.test_raw_inapp_events (./sdftarget/dbg/tests/tests_workspace/pub/test_raw_inapp_events.sql) -   Finished 1 model [1 succeeded], 1 test [1 passed] in 0.829 secs +   Finished 1 model [1 succeeded], 1 test [1 passed] in 0.920 secs [Pass] Test tests_workspace.pub.test_raw_inapp_events @@ -347,7 +347,7 @@ to run custom testing macros. 
 
     Cleaned sdftarget
-   Finished clean in 0.300 secs
+   Finished clean in 0.290 secs
 .
 ├── custom_tests
 ├── models
diff --git a/docs/guide/setup/io.mdx b/docs/guide/setup/io.mdx
index 40ebaec2..58e03d2d 100644
--- a/docs/guide/setup/io.mdx
+++ b/docs/guide/setup/io.mdx
@@ -93,7 +93,7 @@ Let's try these options. Start by creating a new SDF workspace with a sample pro
         
         📚 Read the Docs to Get Started -> https://docs.sdf.com/
         
-   Finished new in 0.315 secs
+   Finished new in 0.308 secs
@@ -122,9 +122,9 @@ sdf compile Working set 4 model files, 1 .sdf file   Compiling lineage.pub.source (./models/source.sql)   Compiling lineage.pub.middle (./models/middle.sql) -  Compiling lineage.pub.knis (./models/knis.sql)   Compiling lineage.pub.sink (./models/sink.sql) -   Finished 4 models [4 succeeded] in 0.845 secs +  Compiling lineage.pub.knis (./models/knis.sql) +   Finished 4 models [4 succeeded] in 0.858 secs diff --git a/docs/integrations/aws/S3/s3-example.mdx b/docs/integrations/aws/S3/s3-example.mdx index 5e6bf2cf..82794437 100644 --- a/docs/integrations/aws/S3/s3-example.mdx +++ b/docs/integrations/aws/S3/s3-example.mdx @@ -57,7 +57,7 @@ integration with SDF to analyze world population data. 
 
 Working set 1 .sdf file
-   Finished run in 0.823 secs
+   Finished run in 0.811 secs
@@ -116,7 +116,7 @@ Working set 2 model files, 1 .sdf file     Running hello_world.pub.un_pop_data (./remote/un_pop_data.sql) Downloading hello_world.pub.un_pop_data  (s3://sdfdatasets/hello-world/world_population_full.csv)     Running hello_world.pub.q1 (./remote/q1.sql) -   Finished 2 models [2 succeeded] in 1.431 secs +   Finished 2 models [2 succeeded] in 1.338 secs Table hello_world.pub.un_pop_data ┌───────┬───────────┬───────────────────┬───────────────┬─────────────────┬───────┬──────┬──────────────────────┬──────────────────────────────────┬────────────┐ diff --git a/docs/integrations/bigquery/basic-materialization.mdx b/docs/integrations/bigquery/basic-materialization.mdx index 6edb2161..e3ddcf5a 100644 --- a/docs/integrations/bigquery/basic-materialization.mdx +++ b/docs/integrations/bigquery/basic-materialization.mdx @@ -113,11 +113,11 @@ Before beginning, we'll need to make sure we have the following: Working set 2 model files, 1 .sdf file Downloading "integration-test-sdf".sdf_ecommerce_example.products (schema) -Downloading "integration-test-sdf".sdf_ecommerce_example.orders (schema) Downloading "integration-test-sdf".sdf_ecommerce_example.customers (schema) +Downloading "integration-test-sdf".sdf_ecommerce_example.orders (schema)   Compiling integration-test-sdf.sdf_ecommerce_example.aggregate_orders (./models/aggregate_orders.sql)   Compiling integration-test-sdf.sdf_ecommerce_example.customers_over_100 (./models/customers_over_100.sql) -   Finished 5 models [5 succeeded] in 1.686 secs +   Finished 5 models [5 succeeded] in 1.735 secs @@ -157,11 +157,11 @@ Downloading "integration-test-sdf".sdf_ecommerce_example. Working set 2 model files, 1 .sdf file Downloading "integration-test-sdf".sdf_ecommerce_example.customers (schema) -Downloading "integration-test-sdf".sdf_ecommerce_example.orders (schema) Downloading "integration-test-sdf".sdf_ecommerce_example.products (schema) +Downloading "integration-test-sdf".sdf_ecommerce_example.orders (schema)     Running integration-test-sdf.sdf_ecommerce_example.aggregate_orders (./models/aggregate_orders.sql)     Running integration-test-sdf.sdf_ecommerce_example.customers_over_100 (./models/customers_over_100.sql) -   Finished 5 models [5 succeeded] in 3.166 secs +   Finished 5 models [5 succeeded] in 3.415 secs @@ -178,22 +178,12 @@ Downloading "integration-test-sdf".sdf_ecommerce_example. 
 
 Working set 2 model files, 1 .sdf file
-Downloading "integration-test-sdf".sdf_ecommerce_example.customers (schema)
 Downloading "integration-test-sdf".sdf_ecommerce_example.products (schema)
 Downloading "integration-test-sdf".sdf_ecommerce_example.orders (schema)
+Downloading "integration-test-sdf".sdf_ecommerce_example.customers (schema)
     Running integration-test-sdf.sdf_ecommerce_example.aggregate_orders (./models/aggregate_orders.sql)
     Running integration-test-sdf.sdf_ecommerce_example.customers_over_100 (./models/customers_over_100.sql)
-   Finished 5 models [5 succeeded] in 5.900 secs
-
-Table "integration-test-sdf".sdf_ecommerce_example.products
-┌────────────┬──────────────┬────────┬─────────────┐
-│ product_id ┆ product_name ┆ price  ┆ category    │
-╞════════════╪══════════════╪════════╪═════════════╡
-│ 1          ┆ Laptop       ┆ 999.99 ┆ Electronics │
-│ 2          ┆ Headphones   ┆ 199.99 ┆ Electronics │
-│ 3          ┆ Coffee Maker ┆ 89.99  ┆ Appliances  │
-└────────────┴──────────────┴────────┴─────────────┘
-3 rows.
+   Finished 5 models [5 succeeded] in 6.694 secs
 
 Table "integration-test-sdf".sdf_ecommerce_example.customers
 ┌─────────────┬───────────────┬───────────────────────┬─────────────────────┐
@@ -205,6 +195,16 @@ Table "integration-test-sdf".sdf_ecommerce_example.custom
 └─────────────┴───────────────┴───────────────────────┴─────────────────────┘
 3 rows.
 
+Table "integration-test-sdf".sdf_ecommerce_example.products
+┌────────────┬──────────────┬────────┬─────────────┐
+│ product_id ┆ product_name ┆ price  ┆ category    │
+╞════════════╪══════════════╪════════╪═════════════╡
+│ 1          ┆ Laptop       ┆ 999.99 ┆ Electronics │
+│ 2          ┆ Headphones   ┆ 199.99 ┆ Electronics │
+│ 3          ┆ Coffee Maker ┆ 89.99  ┆ Appliances  │
+└────────────┴──────────────┴────────┴─────────────┘
+3 rows.
+
 Table "integration-test-sdf".sdf_ecommerce_example.orders
 ┌──────────┬─────────────┬────────────┬─────────────────────┬──────────┐
 │ order_id ┆ customer_id ┆ product_id ┆ order_date          ┆ quantity │
diff --git a/docs/integrations/bigquery/incremental-materialization.mdx b/docs/integrations/bigquery/incremental-materialization.mdx
index 938ce710..204c697a 100644
--- a/docs/integrations/bigquery/incremental-materialization.mdx
+++ b/docs/integrations/bigquery/incremental-materialization.mdx
@@ -52,7 +52,7 @@ You'll also need:
 Working set 2 model files, 1 .sdf file
 Downloading "bigquery-public-data".hacker_news."full" (schema)
   Compiling integration-test-sdf.pub.last_hn_timestamp (./models/last_hn_timestamp.sql)
-   Finished 2 models [2 succeeded] in 1.658 secs
+   Finished 2 models [2 succeeded] in 1.674 secs
 
 Schema "integration-test-sdf".pub.last_hn_timestamp
 ┌─────────────┬───────────┬────────────┬─────────────┐
@@ -129,7 +129,7 @@ Working set 2 model files, 1 .sdf file
 Downloading "integration-test-sdf".pub.popular_articles (exists_remotely)
 Downloading "bigquery-public-data".hacker_news."full" (schema)
   Compiling integration-test-sdf.pub.popular_articles (./models/popular_articles.sql)
-   Finished 2 models [2 succeeded] in 2.290 secs
+   Finished 2 models [2 succeeded] in 2.712 secs
 
 Schema "integration-test-sdf".pub.popular_articles
 ┌─────────────┬───────────┬────────────┬─────────────┐
@@ -196,7 +196,7 @@ Working set 2 model files, 1 .sdf file
 Downloading "integration-test-sdf".pub.popular_articles (exists_remotely)
 Downloading "bigquery-public-data".hacker_news."full" (schema)
     Running integration-test-sdf.pub.popular_articles (./models/popular_articles.sql)
-   Finished 2 models [2 succeeded] in 5.236 secs
+   Finished 2 models [2 succeeded] in 5.667 secs
 
 Table "integration-test-sdf".pub.popular_articles
 ┌────────────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────┬──────┬──────┬────────────────┬───────┬────────────┬─────────────────────┬───────┬──────────┬────────┬─────────────┬─────────┬─────────┐
@@ -290,7 +290,7 @@ Working set 2 model files, 1 .sdf file
 Downloading "integration-test-sdf".pub.popular_articles (exists_remotely)
 Downloading "bigquery-public-data".hacker_news."full" (schema)
   Compiling integration-test-sdf.pub.popular_articles (./models/popular_articles.sql)
-   Finished 2 models [2 succeeded] in 2.289 secs
+   Finished 2 models [2 succeeded] in 2.336 secs
 
 Schema "integration-test-sdf".pub.popular_articles
 ┌─────────────┬───────────┬────────────┬─────────────┐
diff --git a/docs/integrations/bigquery/seeds.mdx b/docs/integrations/bigquery/seeds.mdx
index 1b67b051..5cc340e0 100644
--- a/docs/integrations/bigquery/seeds.mdx
+++ b/docs/integrations/bigquery/seeds.mdx
@@ -130,7 +130,7 @@ You'll also need:
 Working set 1 model file, 1 seed file, 1 .sdf file
   Uploading '"integration-test-sdf".pub.raw_customers'
     Running integration-test-sdf.pub.french_customers (./models/french_customers.sql)
-   Finished 2 models [2 succeeded] in 5.130 secs
+   Finished 2 models [2 succeeded] in 5.372 secs
 
 Table "integration-test-sdf".pub.french_customers
 ┌────────────┬─────────────────┬────────────────┬────────────────────────────────────────┬─────────────────────────────────┬─────────────────────────────┬───────────┬─────────┬─────────────────────────┬─────────────────────────┐
@@ -167,7 +167,7 @@ Table "integration-test-sdf".pub.french_customers
 Working set 2 model files, 1 seed file, 1 .sdf file
   Uploading '"integration-test-sdf".pub.raw_customers'
     Running integration-test-sdf.pub.query (./sdftarget/test/query.sql)
-   Finished 2 models [2 succeeded] in 5.147 secs
+   Finished 2 models [2 succeeded] in 6.346 secs
 
 Table "integration-test-sdf".pub.query
 ┌─────┐
diff --git a/docs/integrations/dbt/migrating.mdx b/docs/integrations/dbt/migrating.mdx
index d183f1d7..d850b5e0 100644
--- a/docs/integrations/dbt/migrating.mdx
+++ b/docs/integrations/dbt/migrating.mdx
@@ -153,7 +153,7 @@ workspace:
 
 
 Working set 3 .sdf files
-   Finished 3 models [3 succeeded] in 0.833 secs
+   Finished 3 models [3 succeeded] in 0.860 secs
 
 Schema jaffle_shop.raw.raw_customers
 ┌─────────────┬───────────┬────────────┬─────────────┐
@@ -414,9 +414,9 @@ tmp/jaffle_shop/
 
 Working set 3 model files, 3 .sdf files
   Compiling jaffle_shop.staging.stg_payments (./models/staging/stg_payments.sql)
-  Compiling jaffle_shop.staging.stg_customers (./models/staging/stg_customers.sql)
   Compiling jaffle_shop.staging.stg_orders (./models/staging/stg_orders.sql)
-   Finished 6 models [6 succeeded] in 0.908 secs
+  Compiling jaffle_shop.staging.stg_customers (./models/staging/stg_customers.sql)
+   Finished 6 models [6 succeeded] in 0.876 secs
 
 
 

@@ -439,7 +439,7 @@ Working set 3 model files, 2 .sdf files
     Running jaffle_shop.staging.stg_customers (./models/staging/stg_customers.sql)
     Running jaffle_shop.staging.stg_payments (./models/staging/stg_payments.sql)
     Running jaffle_shop.staging.stg_orders (./models/staging/stg_orders.sql)
-   Finished 6 models [6 succeeded] in 0.870 secs
+   Finished 6 models [6 succeeded] in 0.902 secs
 
 Table jaffle_shop.staging.stg_customers
 ┌─────────────┬────────────┬───────────┐
@@ -644,7 +644,7 @@ Working set 4 model files, 3 .sdf files
   Compiling jaffle_shop.staging.stg_customers (./models/staging/stg_customers.sql)
   Compiling jaffle_shop.staging.stg_payments (./models/staging/stg_payments.sql)
   Compiling jaffle_shop.analytics.orders (./models/analytics/orders.sql)
-   Finished 7 models [7 succeeded] in 0.860 secs
+   Finished 7 models [7 succeeded] in 0.883 secs
@@ -660,10 +660,10 @@ Working set 4 model files, 3 .sdf files 
 
 Working set 4 model files, 2 .sdf files
-    Running jaffle_shop.staging.stg_orders (./models/staging/stg_orders.sql)
     Running jaffle_shop.staging.stg_payments (./models/staging/stg_payments.sql)
+    Running jaffle_shop.staging.stg_orders (./models/staging/stg_orders.sql)
     Running jaffle_shop.analytics.orders (./models/analytics/orders.sql)
-   Finished 5 models [5 succeeded] in 0.914 secs
+   Finished 5 models [5 succeeded] in 0.900 secs
 
 Table jaffle_shop.analytics.orders
 ┌──────────┬─────────────┬────────────┬───────────┬────────────────────┬───────────────┬──────────────────────┬──────────────────┬────────┐
@@ -721,11 +721,11 @@ Table jaffle_shop.analytics.orders
 
 
 Working set 4 model files, 4 .sdf files
-  Compiling jaffle_shop.staging.stg_orders (./models/staging/stg_orders.sql)
   Compiling jaffle_shop.staging.stg_payments (./models/staging/stg_payments.sql)
+  Compiling jaffle_shop.staging.stg_orders (./models/staging/stg_orders.sql)
   Compiling jaffle_shop.staging.stg_customers (./models/staging/stg_customers.sql)
   Compiling jaffle_shop.analytics.orders (./models/analytics/orders.sql)
-   Finished 7 models [7 succeeded] in 0.867 secs
+   Finished 7 models [7 succeeded] in 0.891 secs
 
 
 

@@ -858,15 +858,15 @@ tmp/jaffle_shop/
 
 Working set 4 model files, 3 test files, 7 .sdf files
     Running jaffle_shop.staging.stg_customers (./models/staging/stg_customers.sql)
-    Running jaffle_shop.staging.stg_payments (./models/staging/stg_payments.sql)
     Running jaffle_shop.staging.stg_orders (./models/staging/stg_orders.sql)
+    Running jaffle_shop.staging.stg_payments (./models/staging/stg_payments.sql)
+    Testing jaffle_shop.staging.test_stg_orders (./sdftarget/dbg/tests/jaffle_shop/staging/test_stg_orders.sql)
     Testing jaffle_shop.staging.test_stg_customers (./sdftarget/dbg/tests/jaffle_shop/staging/test_stg_customers.sql)
     Testing jaffle_shop.staging.test_stg_payments (./sdftarget/dbg/tests/jaffle_shop/staging/test_stg_payments.sql)
-    Testing jaffle_shop.staging.test_stg_orders (./sdftarget/dbg/tests/jaffle_shop/staging/test_stg_orders.sql)
    Finished 6 models [6 succeeded], 3 tests [3 passed] in 0.931 secs
 [Pass] Test jaffle_shop.staging.test_stg_customers
-[Pass] Test jaffle_shop.staging.test_stg_payments
 [Pass] Test jaffle_shop.staging.test_stg_orders
+[Pass] Test jaffle_shop.staging.test_stg_payments
@@ -918,12 +918,12 @@ tmp/jaffle_shop_complete/ 
 
 Working set 5 model files, 10 .sdf files
-  Compiling jaffle_shop.staging.stg_payments (./models/staging/stg_payments.sql)
   Compiling jaffle_shop.staging.stg_customers (./models/staging/stg_customers.sql)
+  Compiling jaffle_shop.staging.stg_payments (./models/staging/stg_payments.sql)
   Compiling jaffle_shop.staging.stg_orders (./models/staging/stg_orders.sql)
   Compiling jaffle_shop.analytics.customers (./models/analytics/customers.sql)
   Compiling jaffle_shop.analytics.orders (./models/analytics/orders.sql)
-   Finished 8 models [8 succeeded] in 0.889 secs
+   Finished 8 models [8 succeeded] in 0.890 secs
@@ -939,12 +939,12 @@ Working set 5 model files, 10 .sdf files 
 
 Working set 5 model files, 10 .sdf files
-    Running jaffle_shop.staging.stg_orders (./models/staging/stg_orders.sql)
     Running jaffle_shop.staging.stg_customers (./models/staging/stg_customers.sql)
     Running jaffle_shop.staging.stg_payments (./models/staging/stg_payments.sql)
+    Running jaffle_shop.staging.stg_orders (./models/staging/stg_orders.sql)
     Running jaffle_shop.analytics.customers (./models/analytics/customers.sql)
     Running jaffle_shop.analytics.orders (./models/analytics/orders.sql)
-   Finished 8 models [8 succeeded] in 0.928 secs
+   Finished 8 models [8 succeeded] in 0.938 secs
@@ -970,7 +970,7 @@ Working set 5 model files, 5 test files,      Running jaffle_shop.analytics.orders (./models/analytics/orders.sql)     Testing jaffle_shop.analytics.test_orders (./sdftarget/dbg/tests/jaffle_shop/analytics/test_orders.sql)     Testing jaffle_shop.analytics.test_customers (./sdftarget/dbg/tests/jaffle_shop/analytics/test_customers.sql) -   Finished 8 models [8 succeeded], 5 tests [5 passed] in 0.996 secs +   Finished 8 models [8 succeeded], 5 tests [5 passed] in 1.022 secs [Pass] Test jaffle_shop.staging.test_stg_orders [Pass] Test jaffle_shop.staging.test_stg_payments [Pass] Test jaffle_shop.staging.test_stg_customers diff --git a/docs/integrations/snowflake/basic-materialization.mdx b/docs/integrations/snowflake/basic-materialization.mdx index 10efdb55..080215ea 100644 --- a/docs/integrations/snowflake/basic-materialization.mdx +++ b/docs/integrations/snowflake/basic-materialization.mdx @@ -46,7 +46,7 @@ When installing the Cybersyn dataset, make sure to grant the Snowflake role you' Working set 3 model files, 1 .sdf file Downloading TECH__INNOVATION_ESSENTIALS.CYBERSYN.OPENALEX_FUNDERS_INDEX (schema)   Compiling sdf_snowflake.cybersyn_tech_innovation.funder_aggregates (./models/sdf_snowflake/cybersyn_tech_innovation/funder_aggregates.sql) -   Finished 2 models [2 succeeded] in 1.641 secs +   Finished 2 models [2 succeeded] in 1.685 secs Schema sdf_snowflake.cybersyn_tech_innovation.FUNDER_AGGREGATES ┌──────────────────────┬────────────────┬────────────┬──────────────────────────────────────────────────────────────────┐ @@ -172,7 +172,7 @@ Schema sdf_snowflake.cybersyn_tech_innovation.FUNDER_AGGREGATES Working set 4 model files, 1 .sdf file Downloading TECH__INNOVATION_ESSENTIALS.CYBERSYN.GITHUB_REPOS (schema)   Compiling sdf_snowflake.staging.latest_repo_name (./models/sdf_snowflake/staging/latest_repo_name.sql) -   Finished 2 models [2 succeeded] in 1.676 secs +   Finished 2 models [2 succeeded] in 1.733 secs Schema sdf_snowflake.staging.LATEST_REPO_NAME ┌─────────────┬────────────────┬────────────┬──────────────────────────────────────┐ @@ -216,7 +216,7 @@ Schema sdf_snowflake.staging.LATEST_REPO_NAME Working set 4 model files, 1 .sdf file Downloading TECH__INNOVATION_ESSENTIALS.CYBERSYN.GITHUB_REPOS (schema)     Running sdf_snowflake.staging.latest_repo_name (./models/sdf_snowflake/staging/latest_repo_name.sql) -   Finished 2 models [2 succeeded] in 5.181 secs +   Finished 2 models [2 succeeded] in 5.639 secs Table sdf_snowflake.staging.LATEST_REPO_NAME ┌──────────────────────────────────┬───────────┐ @@ -282,7 +282,7 @@ Table sdf_snowflake.staging.LATEST_REPO_NAME Working set 5 model files, 1 .sdf file Downloading TECH__INNOVATION_ESSENTIALS.CYBERSYN.GITHUB_EVENTS (schema)   Compiling sdf_snowflake.staging.push_events (./models/sdf_snowflake/staging/push_events.sql) -   Finished 2 models [2 succeeded] in 1.803 secs +   Finished 2 models [2 succeeded] in 1.726 secs Schema sdf_snowflake.staging.PUSH_EVENTS ┌─────────────┬────────────────┬────────────┬────────────────────────────────────────────────────────────────────────────────────────────────┐ @@ -348,20 +348,20 @@ Schema sdf_snowflake.staging.PUSH_EVENTS 
 
 Working set 6 model files, 1 .sdf file
-Downloading TECH__INNOVATION_ESSENTIALS.CYBERSYN.OPENALEX_FUNDERS_INDEX (schema)
-Downloading TECH__INNOVATION_ESSENTIALS.CYBERSYN.GITHUB_REPOS (schema)
 Downloading TECH__INNOVATION_ESSENTIALS.CYBERSYN.USPTO_CONTRIBUTOR_INDEX (schema)
+Downloading TECH__INNOVATION_ESSENTIALS.CYBERSYN.GITHUB_REPOS (schema)
 Downloading TECH__INNOVATION_ESSENTIALS.CYBERSYN.GITHUB_EVENTS (schema)
+Downloading TECH__INNOVATION_ESSENTIALS.CYBERSYN.OPENALEX_FUNDERS_INDEX (schema)
 Downloading TECH__INNOVATION_ESSENTIALS.CYBERSYN.USPTO_PATENT_INDEX (schema)
 Downloading TECH__INNOVATION_ESSENTIALS.CYBERSYN.GITHUB_STARS (schema)
 Downloading TECH__INNOVATION_ESSENTIALS.CYBERSYN.USPTO_PATENT_CONTRIBUTOR_RELATIONSHIPS (schema)
-  Compiling sdf_snowflake.staging.push_events (./models/sdf_snowflake/staging/push_events.sql)
   Compiling sdf_snowflake.cybersyn_tech_innovation.funder_aggregates (./models/sdf_snowflake/cybersyn_tech_innovation/funder_aggregates.sql)
   Compiling sdf_snowflake.cybersyn_tech_innovation.most_starred_repos (./models/sdf_snowflake/cybersyn_tech_innovation/most_starred_repos.sql)
   Compiling sdf_snowflake.staging.latest_repo_name (./models/sdf_snowflake/staging/latest_repo_name.sql)
+  Compiling sdf_snowflake.staging.push_events (./models/sdf_snowflake/staging/push_events.sql)
   Compiling sdf_snowflake.cybersyn_tech_innovation.all_nvidia_patents (./models/sdf_snowflake/cybersyn_tech_innovation/all_nvidia_patents.sql)
   Compiling sdf_snowflake.cybersyn_tech_innovation.events_by_repo (./models/sdf_snowflake/cybersyn_tech_innovation/events_by_repo.sql)
-   Finished 13 models [13 succeeded] in 1.854 secs
+   Finished 13 models [13 succeeded] in 1.889 secs
diff --git a/docs/integrations/snowflake/incremental-materialization.mdx b/docs/integrations/snowflake/incremental-materialization.mdx index 7d179275..73fef259 100644 --- a/docs/integrations/snowflake/incremental-materialization.mdx +++ b/docs/integrations/snowflake/incremental-materialization.mdx @@ -50,7 +50,7 @@ When installing the Cybersyn dataset, make sure to grant the Snowflake role you' Working set 3 model files, 1 .sdf file Downloading TECH__INNOVATION_ESSENTIALS.CYBERSYN.OPENALEX_FUNDERS_INDEX (schema)   Compiling sdf_snowflake.cybersyn_tech_innovation.funder_aggregates (./models/sdf_snowflake/cybersyn_tech_innovation/funder_aggregates.sql) -   Finished 2 models [2 succeeded] in 1.624 secs +   Finished 2 models [2 succeeded] in 1.771 secs Schema sdf_snowflake.cybersyn_tech_innovation.FUNDER_AGGREGATES ┌──────────────────────┬────────────────┬────────────┬──────────────────────────────────────────────────────────────────┐ @@ -137,7 +137,7 @@ Working set 4 model files, 1 .sdf file Downloading SDF_SNOWFLAKE.STAGING.DATAFUSION_PUSH_EVENTS (exists_remotely) Downloading TECH__INNOVATION_ESSENTIALS.CYBERSYN.GITHUB_EVENTS (schema)   Compiling sdf_snowflake.staging.datafusion_push_events (./models/sdf_snowflake/staging/datafusion_push_events.sql) -   Finished 2 models [2 succeeded] in 2.541 secs +   Finished 2 models [2 succeeded] in 2.448 secs Schema sdf_snowflake.staging.DATAFUSION_PUSH_EVENTS ┌────────────────────────────────┬────────────────┬────────────┬────────────────────────────────────────────────────────────────────────────────────────────────┐ @@ -350,7 +350,7 @@ Downloading SDF_SNOWFLAKE.CYBERSYN_TECH_INNOVATION.TOP_DATAFUSION_CONTRIBUT Downloading TECH__INNOVATION_ESSENTIALS.CYBERSYN.GITHUB_EVENTS (schema)   Compiling sdf_snowflake.staging.datafusion_push_events (./models/sdf_snowflake/staging/datafusion_push_events.sql)   Compiling sdf_snowflake.cybersyn_tech_innovation.top_datafusion_contributors (./models/sdf_snowflake/cybersyn_tech_innovation/top_datafusion_contributors.sql) -   Finished 3 models [3 succeeded] in 2.324 secs +   Finished 3 models [3 succeeded] in 2.399 secs Schema sdf_snowflake.cybersyn_tech_innovation.TOP_DATAFUSION_CONTRIBUTORS ┌─────────────────────┬────────────────┬────────────┬──────────────────────────────────────────────────────────────┐ diff --git a/docs/integrations/snowflake/seeds.mdx b/docs/integrations/snowflake/seeds.mdx index 41d20196..6b2ebe90 100644 --- a/docs/integrations/snowflake/seeds.mdx +++ b/docs/integrations/snowflake/seeds.mdx @@ -130,7 +130,7 @@ You'll also need: Working set 1 model file, 1 seed file, 1 .sdf file   Uploading 'INTEGRATION_TEST.PUB.RAW_CUSTOMERS'     Running integration_test.pub.french_customers (./models/french_customers.sql) -   Finished 2 models [2 succeeded] in 6.562 secs +   Finished 2 models [2 succeeded] in 6.923 secs Table INTEGRATION_TEST.PUB.FRENCH_CUSTOMERS ┌────────────┬─────────────────┬────────────────┬────────────────────────────────────────┬─────────────────────────────────┬─────────────────────────────┬───────────┬─────────┬─────────────────────────┬─────────────────────────┐ @@ -167,7 +167,7 @@ Table INTEGRATION_TEST.PUB.FRENCH_CUSTOMERS Working set 2 model files, 1 seed file, 1 .sdf file   Uploading 'INTEGRATION_TEST.PUB.RAW_CUSTOMERS'     Running integration_test.pub.query (./sdftarget/test/query.sql) -   Finished 2 models [2 succeeded] in 5.738 secs +   Finished 2 models [2 succeeded] in 6.042 secs Table INTEGRATION_TEST.PUB.QUERY ┌─────┐ diff --git a/docs/introduction/getting-started.mdx b/docs/introduction/getting-started.mdx index 83f56b89..3c077f95 100644 --- a/docs/introduction/getting-started.mdx +++ b/docs/introduction/getting-started.mdx @@ -81,7 +81,7 @@ After running the command, you will see the following output:                  📚 Read the Docs to Get Started -> https://docs.sdf.com/          -   Finished new in 0.268 secs +   Finished new in 0.302 secs @@ -169,7 +169,7 @@ Let's see just how easy it is to set up SDF and run your first query. Working set 1 model file, 1 .sdf file   Compiling hello.pub.main (./models/main.sql) -   Finished 1 model [1 succeeded] in 0.820 secs +   Finished 1 model [1 succeeded] in 0.844 secs Schema hello.pub.main ┌─────────────┬──────────────────┬────────────┬─────────────┐ @@ -206,7 +206,7 @@ Schema hello.pub.main Working set 2 model files, 1 .sdf file   Compiling hello.pub.main (./models/main.sql)   Compiling hello.pub.main2 (./models/main2.sql) -   Finished 2 models [2 succeeded] in 0.808 secs +   Finished 2 models [2 succeeded] in 0.848 secs hello.pub.main2.message │ │ copy @@ -231,7 +231,7 @@ hello.pub.main2.message Working set 2 model files, 1 .sdf file     Running hello.pub.main (./models/main.sql)     Running hello.pub.main2 (./models/main2.sql) -   Finished 2 models [2 succeeded] in 0.840 secs +   Finished 2 models [2 succeeded] in 0.843 secs Table hello.pub.main ┌──────────────┐ diff --git a/docs/mint.json b/docs/mint.json index 3472ead8..51f8c0ad 100644 --- a/docs/mint.json +++ b/docs/mint.json @@ -151,6 +151,7 @@ "group": "Using SDF", "icon": "play", "pages": [ + "guide/basics/build_and_deployment", "guide/basics/lineage_metadata", "guide/basics/classifiers", { diff --git a/docs/tutorials/creating-a-model.mdx b/docs/tutorials/creating-a-model.mdx index eac4d162..9743a5d8 100644 --- a/docs/tutorials/creating-a-model.mdx +++ b/docs/tutorials/creating-a-model.mdx @@ -102,18 +102,18 @@ Init for run commands: 
 
 Working set 11 model files, 22 .sdf files
+  Compiling moms_flower_shop.raw.raw_inapp_events (./models/raw/raw_inapp_events.sql)
   Compiling moms_flower_shop.raw.raw_marketing_campaign_events (./models/raw/raw_marketing_campaign_events.sql)
   Compiling moms_flower_shop.raw.raw_addresses (./models/raw/raw_addresses.sql)
   Compiling moms_flower_shop.raw.raw_customers (./models/raw/raw_customers.sql)
-  Compiling moms_flower_shop.raw.raw_inapp_events (./models/raw/raw_inapp_events.sql)
-  Compiling moms_flower_shop.staging.marketing_campaigns (./models/staging/marketing_campaigns.sql)
   Compiling moms_flower_shop.staging.inapp_events (./models/staging/inapp_events.sql)
+  Compiling moms_flower_shop.staging.marketing_campaigns (./models/staging/marketing_campaigns.sql)
   Compiling moms_flower_shop.staging.app_installs (./models/staging/app_installs.sql)
   Compiling moms_flower_shop.staging.app_installs_v2 (./models/staging/app_installs_v2.sql)
   Compiling moms_flower_shop.analytics.agg_installs_and_campaigns (./models/analytics/agg_installs_and_campaigns.sql)
   Compiling moms_flower_shop.staging.customers (./models/staging/customers.sql)
   Compiling moms_flower_shop.staging.stg_installs_per_campaign (./models/staging/stg_installs_per_campaign.sql)
-   Finished 11 models [11 succeeded] in 0.860 secs
+   Finished 11 models [11 succeeded] in 0.897 secs
@@ -145,7 +145,7 @@ Working set 12 model files, 15 .sdf files     Running moms_flower_shop.staging.app_installs (./models/staging/app_installs.sql)     Running moms_flower_shop.staging.stg_installs_per_campaign (./models/staging/stg_installs_per_campaign.sql)     Running moms_flower_shop.analytics.dim_marketing_campaigns (./models/analytics/dim_marketing_campaigns.sql) -   Finished 7 models [7 succeeded] in 0.973 secs +   Finished 7 models [7 succeeded] in 0.972 secs Table moms_flower_shop.analytics.dim_marketing_campaigns ┌─────────────┬─────────────────────────────────┬────────────────────┬───────────────────────────────┬────────────────────────┐ diff --git a/docs/tutorials/debugging.mdx b/docs/tutorials/debugging.mdx index eb402b6e..027f5d60 100644 --- a/docs/tutorials/debugging.mdx +++ b/docs/tutorials/debugging.mdx @@ -153,7 +153,7 @@ moms_flower_shop.staging.customers.campaign_id Working set 12 model files, 15 .sdf files   Compiling moms_flower_shop.staging.stg_installs_per_campaign (./models/staging/stg_installs_per_campaign.sql)   Compiling moms_flower_shop.analytics.dim_marketing_campaigns (./models/analytics/dim_marketing_campaigns.sql) -   Finished 11 models [2 succeeded, 9 reused] in 1.416 secs +   Finished 11 models [2 succeeded, 9 reused] in 1.380 secs @@ -211,7 +211,7 @@ Working set 12 model files, 15 .sdf files     Running moms_flower_shop.staging.app_installs_v2 (./models/staging/app_installs_v2.sql)     Running moms_flower_shop.staging.stg_installs_per_campaign (./models/staging/stg_installs_per_campaign.sql)     Running moms_flower_shop.analytics.dim_marketing_campaigns (./models/analytics/dim_marketing_campaigns.sql) -   Finished 7 models [7 succeeded] in 0.945 secs +   Finished 7 models [7 succeeded] in 0.983 secs Table moms_flower_shop.analytics.dim_marketing_campaigns ┌─────────────┬─────────────────────────────────┬────────────────────┬───────────────────────────────┬────────────────────────┐ diff --git a/docs/tutorials/script_test.mdx b/docs/tutorials/script_test.mdx index 07d977a9..12788761 100644 --- a/docs/tutorials/script_test.mdx +++ b/docs/tutorials/script_test.mdx @@ -10,10 +10,10 @@ Not an error: 
 
 Working set 12 model files, 24 .sdf files
+    Running moms_flower_shop.raw.raw_addresses (./models/raw/raw_addresses.sql)
     Running moms_flower_shop.raw.raw_customers (./models/raw/raw_customers.sql)
     Running moms_flower_shop.raw.raw_marketing_campaign_events (./models/raw/raw_marketing_campaign_events.sql)
     Running moms_flower_shop.raw.raw_inapp_events (./models/raw/raw_inapp_events.sql)
-    Running moms_flower_shop.raw.raw_addresses (./models/raw/raw_addresses.sql)
     Running moms_flower_shop.staging.marketing_campaigns (./models/staging/marketing_campaigns.sql)
     Running moms_flower_shop.staging.inapp_events (./models/staging/inapp_events.sql)
     Running moms_flower_shop.staging.app_installs (./models/staging/app_installs.sql)
@@ -22,7 +22,7 @@ Working set 12 model files, 24 .sdf files
     Running moms_flower_shop.staging.stg_installs_per_campaign (./models/staging/stg_installs_per_campaign.sql)
     Running moms_flower_shop.staging.customers (./models/staging/customers.sql)
     Running moms_flower_shop.analytics.dim_marketing_campaigns (./models/analytics/dim_marketing_campaigns.sql)
-   Finished 12 models [12 succeeded] in 1.027 secs
+   Finished 12 models [12 succeeded] in 1.077 secs
diff --git a/docs/tutorials/tutorials-intro.mdx b/docs/tutorials/tutorials-intro.mdx index 8d3b8c6a..369b8372 100644 --- a/docs/tutorials/tutorials-intro.mdx +++ b/docs/tutorials/tutorials-intro.mdx @@ -225,14 +225,14 @@ Table moms_flower_shop.raw.raw_customers 
 
 Working set 11 model files, 22 .sdf files
-  Compiling moms_flower_shop.staging.marketing_campaigns (./models/staging/marketing_campaigns.sql)
   Compiling moms_flower_shop.staging.inapp_events (./models/staging/inapp_events.sql)
-  Compiling moms_flower_shop.staging.app_installs (./models/staging/app_installs.sql)
+  Compiling moms_flower_shop.staging.marketing_campaigns (./models/staging/marketing_campaigns.sql)
   Compiling moms_flower_shop.staging.app_installs_v2 (./models/staging/app_installs_v2.sql)
-  Compiling moms_flower_shop.analytics.agg_installs_and_campaigns (./models/analytics/agg_installs_and_campaigns.sql)
+  Compiling moms_flower_shop.staging.app_installs (./models/staging/app_installs.sql)
   Compiling moms_flower_shop.staging.customers (./models/staging/customers.sql)
   Compiling moms_flower_shop.staging.stg_installs_per_campaign (./models/staging/stg_installs_per_campaign.sql)
-   Finished 11 models [7 succeeded, 4 reused] in 0.910 secs
+  Compiling moms_flower_shop.analytics.agg_installs_and_campaigns (./models/analytics/agg_installs_and_campaigns.sql)
+   Finished 11 models [7 succeeded, 4 reused] in 0.971 secs
@@ -258,18 +258,18 @@ Working set 11 model files, 22 .sdf files 
 
 Working set 11 model files, 22 .sdf files
-  Compiling moms_flower_shop.raw.raw_customers (./models/raw/raw_customers.sql)
   Compiling moms_flower_shop.raw.raw_marketing_campaign_events (./models/raw/raw_marketing_campaign_events.sql)
+  Compiling moms_flower_shop.raw.raw_customers (./models/raw/raw_customers.sql)
   Compiling moms_flower_shop.raw.raw_addresses (./models/raw/raw_addresses.sql)
   Compiling moms_flower_shop.raw.raw_inapp_events (./models/raw/raw_inapp_events.sql)
-  Compiling moms_flower_shop.staging.marketing_campaigns (./models/staging/marketing_campaigns.sql)
   Compiling moms_flower_shop.staging.inapp_events (./models/staging/inapp_events.sql)
+  Compiling moms_flower_shop.staging.marketing_campaigns (./models/staging/marketing_campaigns.sql)
   Compiling moms_flower_shop.staging.app_installs (./models/staging/app_installs.sql)
   Compiling moms_flower_shop.staging.app_installs_v2 (./models/staging/app_installs_v2.sql)
-  Compiling moms_flower_shop.staging.customers (./models/staging/customers.sql)
   Compiling moms_flower_shop.analytics.agg_installs_and_campaigns (./models/analytics/agg_installs_and_campaigns.sql)
+  Compiling moms_flower_shop.staging.customers (./models/staging/customers.sql)
   Compiling moms_flower_shop.staging.stg_installs_per_campaign (./models/staging/stg_installs_per_campaign.sql)
-   Finished 11 models [11 succeeded] in 0.911 secs
+   Finished 11 models [11 succeeded] in 0.921 secs