You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We currently can only retrieve results from cache for tables that managed to get through a stage commit (i.e. schema swapping procedure). However, in case of really large tables, it might be nice to just continue getting a stage output produced where it stopped in a previous crash. We do cache invalidation on table level. So it would be possible to track that. The tricky aspect would be that we basically need to validate that everything in the transaction schema is cache valid up to one point and then continue from there.
There is a workaround for this problem. One can wrap every table with its own nested Stage. This actually achieves pretty much the desired result as described above. However, in the end, it would be nice if all tables were still located within one schema.
The other downside of the workaround is that in fact the schema swapping of the surrounding Stage is made useless since each table has a separate stage commit and overwrites its cache one by one.
The text was updated successfully, but these errors were encountered:
We currently can only retrieve results from cache for tables that managed to get through a stage commit (i.e. schema swapping procedure). However, in case of really large tables, it might be nice to just continue getting a stage output produced where it stopped in a previous crash. We do cache invalidation on table level. So it would be possible to track that. The tricky aspect would be that we basically need to validate that everything in the transaction schema is cache valid up to one point and then continue from there.
There is a workaround for this problem. One can wrap every table with its own nested Stage. This actually achieves pretty much the desired result as described above. However, in the end, it would be nice if all tables were still located within one schema.
The other downside of the workaround is that in fact the schema swapping of the surrounding Stage is made useless since each table has a separate stage commit and overwrites its cache one by one.
The text was updated successfully, but these errors were encountered: