Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Lake][DuckDB] Raw data update fails if there is CSV data for a table but no rows in production table #1038

Closed
KatunaNorbert opened this issue May 14, 2024 · 1 comment
Assignees
Labels
Type: Bug Something isn't working

Comments

@KatunaNorbert
Copy link
Member

Ho to reproduce:

  • go trough etl raw update flow
  • use the drop raw etl command to remove data from the a raw table
  • try to run etl raw update again and you should get the following error:

Screenshot 2024-05-14 at 12.29.20.png

When checking if there is a need for data to be loaded from CSVs, it tries to get timestamp value from table, but because the table is empty the mentioned error is thrown

@idiom-bytes
Copy link
Member

I can't see the error you posted, but I can confirm that I'm able to repeat the problem of:

  1. Create the lake tables
  2. Drop the rows
  3. Run into problems because it won't build correctly again

I have verified that I can

  1. Try to run an update on with empty tables that have been dropped
  2. Have the tables rebuild correctly
  3. Run describe, and see it working again

I had to push a fix, such that I could properly drop the the tables in raw + etl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants