Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create addressbasepremium_load_files.py #1470

Merged
merged 11 commits into from
Nov 2, 2023
Merged

Create addressbasepremium_load_files.py #1470

merged 11 commits into from
Nov 2, 2023

Conversation

LBHSBALLEY
Copy link
Contributor

@LBHSBALLEY LBHSBALLEY commented Oct 27, 2023 •
edited
Loading

2 jobs and 2 look up files to load a full supply of AddressBase Premium GB into the DP. This is a manual process done every 6 weeks:

  • Manually drop a full supply into the raw zone/unrestricted/osaddressbasepremium (about 300 csv files), in a new folder indicating the epoch number
  • Run the job addressbasepremium_load_files after setting the path of the the raw and the preprocessed files in the parameters
  • Run the job addressbasepremium_create_address_table after setting the path of the preprocessed files in the parameters. The look-up files need to be refreshed every year or so.

@LBHSBALLEY
Create addressbasepremium_load_files.py
11cf9ea
@LBHSBALLEY
Update 24-aws-glue-spatial.tf
604a7f3 
Add module for addressbase file loader and 2 lookup files for the future national_address module
@LBHSBALLEY
Add files via upload
ba0d560 
2 utility files for addressbase premium loader
@LBHSBALLEY
Update 24-aws-glue-spatial.tf
d3f39b8 
recommit after uploading uyility files
@LBHSBALLEY
Update 24-aws-glue-spatial.tf
b4ed4e9 
correct util file name
@LBHSBALLEY
Create addressbasepremium_create_national_address_table
18ca95d
@LBHSBALLEY
Update 24-aws-glue-spatial.tf
5679c4c 
fix tf syntax
@LBHSBALLEY
Update 24-aws-glue-spatial.tf
bdb95be 
Module for the job creating the address table
@LBHSBALLEY
Rename addressbasepremium_create_national_address_table to addressbas…
29552dc 
…epremium_create_address_table.py

Rename script to match terraform
@LBHSBALLEY
Update addressbasepremium_create_address_table.py
133ea66 
add comments
@LBHSBALLEY
Update 24-aws-glue-spatial.tf
d030873 
4 workers for file loader job
@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 14 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@LBHSBALLEY LBHSBALLEY marked this pull request as ready for review October 27, 2023 21:19
LBH-wgreeff
LBH-wgreeff approved these changes Oct 27, 2023
View reviewed changes
Copy link
Collaborator

@LBH-wgreeff LBH-wgreeff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@LBHSBALLEY LBHSBALLEY merged commit 00b9f09 into main Nov 2, 2023
6 checks passed
@LBHSBALLEY LBHSBALLEY deleted the LBHSBALLEY-abp-loader branch November 2, 2023 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@LBH-wgreeff LBH-wgreeff LBH-wgreeff approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@LBHSBALLEY @LBH-wgreeff