Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit the reading of CSVs in seed directory to dbt seed #867

Closed
chrismeier opened this issue Jul 18, 2018 · 2 comments
Closed

Limit the reading of CSVs in seed directory to dbt seed #867

chrismeier opened this issue Jul 18, 2018 · 2 comments
Assignees
Milestone

Comments

@chrismeier
Copy link

When using dbt commands beyond dbt seed (i.e. dbt run), the parse_seed_file function in parser.py loads the CSV file unnecessarily. In the event that there are larger CSVs in the seed directory, this load time can take a while.

To limit the loading of the CSVs, the following section of parse_seed_file should only run during dbt seed.

 try:
        table = dbt.clients.agate_helper.from_csv(abspath)
    except ValueError as e:
        dbt.exceptions.raise_compiler_error(str(e), node)
    table.original_abspath = abspath
    node['agate_table'] = table

The original Slack conversation that this issue came up in is here.

@beckjake beckjake self-assigned this Sep 28, 2018
beckjake added a commit that referenced this issue Oct 9, 2018
…eds-only

Only parse CSV files during "dbt seed" (#867)
@beckjake
Copy link
Contributor

Fixed in #1046

@chrismeier
Copy link
Author

This is super exciting @beckjake! Thanks for putting a together a fix -- looking forward to testing it out :)

@drewbanin drewbanin added this to the Guion Bluford milestone Oct 25, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants