Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First draft of Reporting Source of Truth™ #496

Open
wants to merge 56 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
d780d76
First draft of sales script
wrridgeway Jun 6, 2024
00909fd
File renaming
wrridgeway Jun 6, 2024
2ac5982
Cleaner for loop
wrridgeway Jun 6, 2024
2107d2a
First draft taxes and exemptions table
wrridgeway Jun 11, 2024
c56aaaf
Wrap assessment_roll
wrridgeway Jun 12, 2024
6c81308
Correct size, count calculations
wrridgeway Jun 13, 2024
1bf9b9c
Wrap sales table
wrridgeway Jun 18, 2024
0a9e1f3
Correct stage grouping, counting
wrridgeway Jun 18, 2024
030a7c5
Fix assessment roll stage grouping
wrridgeway Jun 18, 2024
1c2adae
Clean output before writing
wrridgeway Jun 18, 2024
672bd1e
Begin dbt building
wrridgeway Jun 18, 2024
0c42e23
Merge branch 'master' into 387-reporting-sot
wrridgeway Jun 18, 2024
3f60a77
Attempt to build assessment_roll table
wrridgeway Jun 20, 2024
fdff457
Testing build on smaller input
wrridgeway Jun 20, 2024
6abd074
Trying to build on limited sample
wrridgeway Jun 20, 2024
fd342b6
Try to build sales table
wrridgeway Jun 20, 2024
cccf8e1
Try to build taxes and exemptions table
wrridgeway Jun 20, 2024
3656964
Try to build taxes and exemptions table
wrridgeway Jun 20, 2024
8b0f95f
Try to build taxes table
wrridgeway Jun 20, 2024
9383bdc
Try to build ratio stats table
wrridgeway Jun 20, 2024
08d3bd6
Add assesspy to ratio_stats table
wrridgeway Jun 20, 2024
d2cac22
ratio_stats builds in dbt, excluding assesspy funcs
wrridgeway Jun 24, 2024
f559753
sot_ratio_stats table building in dbt
wrridgeway Jun 26, 2024
1f8ad1f
Add res_other group
wrridgeway Jun 26, 2024
063591c
Add reassessment year indicator for assessment roll
wrridgeway Jun 27, 2024
a9ffc64
Retry assessment_year indicator
wrridgeway Jun 27, 2024
62dd68e
Assessment_roll should run with reassessment year indicator
wrridgeway Jun 28, 2024
c185e81
Add schema to assessment_roll table
wrridgeway Jun 28, 2024
d08bc3d
Correct output from sales and taxes tables
wrridgeway Jun 28, 2024
4808aa4
Add table schemas
wrridgeway Jun 28, 2024
08c8d53
Fix schemas
wrridgeway Jun 28, 2024
2f8dc3d
Resolve sales table column type issues
wrridgeway Jul 1, 2024
88ce049
Add exe_total to exemptions table
wrridgeway Jul 2, 2024
271576d
Add more ratio stats
wrridgeway Jul 2, 2024
c39a2d8
Clean sales table columns
wrridgeway Jul 2, 2024
20c9bd6
Clean taxes table columns
wrridgeway Jul 3, 2024
adc16ea
Clean assessment_roll columns
wrridgeway Jul 7, 2024
f8b87ab
Fix delta columns
wrridgeway Jul 7, 2024
54ebab8
Clean ratio table columns
wrridgeway Jul 7, 2024
d2dddab
Attempt to fix pin_n_tot type error that doesn't trigger locally
wrridgeway Jul 7, 2024
00e790c
Try again to fix pin_n_tot
wrridgeway Jul 7, 2024
408de56
Change ass roll sample to be able to compare across stages
wrridgeway Jul 7, 2024
fd95fcb
Add commenting for input tables, try to partion assessment_roll table
wrridgeway Jul 7, 2024
f296292
Comment python scripts
wrridgeway Jul 7, 2024
a23ff72
Clean up ratio_stats script
wrridgeway Jul 8, 2024
07f6dfe
Back to fixing pin_n_tot
wrridgeway Jul 8, 2024
b78a072
Replace nan with None
wrridgeway Jul 8, 2024
337954e
Partition input tables by year
wrridgeway Jul 8, 2024
1031144
Fix year partitioning
wrridgeway Jul 8, 2024
45ea305
Use double for nullable columns
wrridgeway Jul 8, 2024
ca139f3
Move data year specification to dbt seed
wrridgeway Jul 9, 2024
788f971
Formatting
wrridgeway Jul 9, 2024
4ea6718
Merge branch 'master' into 387-reporting-sot
wrridgeway Jul 9, 2024
5449d8c
Improve diff and pct_change syntax
wrridgeway Jul 9, 2024
c87713f
Simplify reassessment year syntax
wrridgeway Jul 9, 2024
d1079f0
More commenting
wrridgeway Jul 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,4 @@ venv/
# Ignore most CSVs, except those that are used as dbt seeds
*.csv
!dbt/seeds/**/*.csv

*.parquet.gzip
2 changes: 2 additions & 0 deletions dbt/dbt_project.yml
Original file line number Diff line number Diff line change
Expand Up @@ -73,5 +73,7 @@ seeds:
+schema: location
model:
+schema: model
reporting:
+schema: reporting
spatial:
+schema: spatial
69 changes: 69 additions & 0 deletions dbt/models/reporting/docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,75 @@ Materialized to speed up queries for Tableau.
`property_group`
{% enddocs %}

# sot_assessment_roll

{% docs table_sot_assessment_roll %}
Table to feed the Python dbt job that creates the
`reporting.sot_assessment_roll` table. Feeds public reporting assets.

**Primary Key**: `year`, `stage_name`, `geography_id`, `group_id`
Comment on lines +36 to +39
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (blocking): This is the same as the description for sot_assesssment_roll_input. Let's change the actual sot_ non-input table descriptions to include the table's purpose, structure, and a short description of how to use the table.

{% enddocs %}

# sot_assessment_roll_input

{% docs table_sot_assessment_roll_input %}
Table to feed the Python dbt job that creates the
`reporting.sot_assessment_roll` table. Feeds public reporting assets.

**Primary Key**: `year`, `stage_name`, `geography_id`, `group_id`
{% enddocs %}

# sot_ratio_stats
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (blocking): No other dbt models have plural names, let's stick to that convention. So:

  • sot_ratio_stats --> sot_ratio_stat
  • sot_sales --> sot_sale

etc.


{% docs table_sot_ratio_stats %}
Feeds public reporting assets.

**Primary Key**: `year`, `stage_name`, `geography_id`, `group_id`
{% enddocs %}

# sot_ratio_stats_input

{% docs table_sot_ratio_stats_input %}
Table to feed the Python dbt job that creates the
`reporting.sot_ratio_stats` table. Feeds public reporting assets.

**Primary Key**: `year`, `stage_name`, `geography_id`, `group_id`
{% enddocs %}

# sot_sales

{% docs table_sot_sales %}
Feeds public reporting assets.

**Primary Key**: `year`, `geography_id`, `group_id`
{% enddocs %}

# sot_sales_input

{% docs table_sot_sales_input %}
Table to feed the Python dbt job that creates the
`reporting.sot_sales` table. Feeds public reporting assets.

**Primary Key**: `year`, `geography_id`, `group_id`
{% enddocs %}

# sot_taxes_exemptions

{% docs table_sot_taxes_exemptions %}
Feeds public reporting assets.

**Primary Key**: `year`, `geography_id`, `group_id`
{% enddocs %}

# sot_taxes_exemptions_input

{% docs table_sot_taxes_exemptions_input %}
Table to feed the Python dbt job that creates the
`reporting.sot_taxes_exemptions` table. Feeds public reporting assets.

**Primary Key**: `year`, `geography_id`, `group_id`
{% enddocs %}

# vw_assessment_roll

{% docs view_vw_assessment_roll %}
Expand Down
Loading
Loading