Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flatten artifacts on load #84

Merged
merged 38 commits into from
Feb 22, 2022
Merged

Flatten artifacts on load #84

merged 38 commits into from
Feb 22, 2022

Conversation

alanmcruickshank
Copy link
Contributor

@alanmcruickshank alanmcruickshank commented Feb 12, 2022 •
edited
Loading

This is a draft solution to #62, #45, #20 & #78.

Results are flattened on upload into three tables:

  • one for run result files (same granularity as artifacts)
  • one for the flattened results together
  • one for the flattened manifest elements (with sources, exposures compressed in with other nodes)

This upload is facilitated by a new V2 upload macro, and then the results of both this approach and the old V1 upload method are unioned together to give teams flexibility on which they use.

This should bypass field size limits (because we use insert into), and preloads the compute load of flattening large JSON blobs so allow larger projects to work well.

@alanmcruickshank alanmcruickshank changed the title Ac/materialise on load Flatten artifacts on load Feb 12, 2022
@alanmcruickshank alanmcruickshank marked this pull request as draft February 12, 2022 16:55
@alanmcruickshank
Add run_results query and schema
c848c73 
Manifest nodes and create table queries

Handle stages better

adjust insert into

Swap all from COPY TO to INSERT

Swtich names for artifact tables.

update sources file

Fix manifest node reference

correct the manifest time

Actually log the queries

Add sources and exposures

typo on insert queries

correct the schema for sources and exposures

fix wrong reference
@alanmcruickshank
Add instructions in README
722c1b5
@alanmcruickshank
Union in the V2 manifest elements
b15a730
@alanmcruickshank alanmcruickshank temporarily deployed to Approve Integration Tests February 12, 2022 16:56 Inactive
@alanmcruickshank alanmcruickshank mentioned this pull request Feb 13, 2022
Merged
@alanmcruickshank alanmcruickshank temporarily deployed to Approve Integration Tests February 15, 2022 14:05 Inactive
NiallRees
NiallRees reviewed Feb 16, 2022
View reviewed changes
README.md Outdated Show resolved Hide resolved
@alanmcruickshank alanmcruickshank temporarily deployed to Approve Integration Tests February 21, 2022 16:32 Inactive
@alanmcruickshank alanmcruickshank temporarily deployed to Approve Integration Tests February 21, 2022 17:25 Inactive
@alanmcruickshank alanmcruickshank temporarily deployed to Approve Integration Tests February 21, 2022 21:42 Inactive
@alanmcruickshank alanmcruickshank temporarily deployed to Approve Integration Tests February 21, 2022 21:42 Inactive
@NiallRees NiallRees had a problem deploying to Approve Integration Tests February 21, 2022 22:35 Failure
@NiallRees NiallRees had a problem deploying to Approve Integration Tests February 21, 2022 22:35 Failure
@NiallRees NiallRees temporarily deployed to Approve Integration Tests February 21, 2022 22:35 Inactive
@NiallRees NiallRees temporarily deployed to Approve Integration Tests February 21, 2022 22:35 Inactive
NiallRees
NiallRees reviewed Feb 21, 2022
View reviewed changes
README.md Outdated Show resolved Hide resolved
NiallRees
NiallRees reviewed Feb 21, 2022
View reviewed changes
README.md Outdated Show resolved Hide resolved
NiallRees
NiallRees reviewed Feb 21, 2022
View reviewed changes
README.md Outdated Show resolved Hide resolved
@NiallRees NiallRees had a problem deploying to Approve Integration Tests February 21, 2022 22:44 Failure
@NiallRees NiallRees had a problem deploying to Approve Integration Tests February 21, 2022 22:44 Failure
@NiallRees NiallRees temporarily deployed to Approve Integration Tests February 21, 2022 22:44 Inactive
@NiallRees NiallRees temporarily deployed to Approve Integration Tests February 21, 2022 22:44 Inactive
NiallRees
NiallRees approved these changes Feb 21, 2022
View reviewed changes
Copy link
Contributor

@NiallRees NiallRees left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job. Tests too!

@NiallRees NiallRees merged commit f435114 into brooklyn-data:main Feb 22, 2022
@alanmcruickshank alanmcruickshank mentioned this pull request Feb 23, 2022
Merged
@alanmcruickshank alanmcruickshank deleted the ac/materialise_on_load branch March 13, 2022 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@NiallRees NiallRees NiallRees approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@alanmcruickshank @NiallRees