Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry of producing a stage output #167

Open
windiana42 opened this issue Mar 21, 2024 · 0 comments
Open

Retry of producing a stage output #167

windiana42 opened this issue Mar 21, 2024 · 0 comments
Labels

Comments

@windiana42
Copy link
Member

We currently can only retrieve results from cache for tables that managed to get through a stage commit (i.e. schema swapping procedure). However, in case of really large tables, it might be nice to just continue getting a stage output produced where it stopped in a previous crash. We do cache invalidation on table level. So it would be possible to track that. The tricky aspect would be that we basically need to validate that everything in the transaction schema is cache valid up to one point and then continue from there.

There is a workaround for this problem. One can wrap every table with its own nested Stage. This actually achieves pretty much the desired result as described above. However, in the end, it would be nice if all tables were still located within one schema.
The other downside of the workaround is that in fact the schema swapping of the surrounding Stage is made useless since each table has a separate stage commit and overwrites its cache one by one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant