-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
importer: add ImportStartWallTime table descriptor field #85852
importer: add ImportStartWallTime table descriptor field #85852
Conversation
324d337
to
e2108ea
Compare
66edc1b
to
740d8ef
Compare
@dt I added an ImportType field to the table descriptor. I need this ensure that IMPORT PGDUMP descriptors remain offline once the RESTORE completes. |
af632e5
to
1368086
Compare
9e4a62c
to
f5a79db
Compare
@msbutler Can't we just only set the import wall time field to a non-zero number for non-new imports? And then all other imports would not be restored? |
My motivation for adding the ImportType is that during the restore job, i need to distinguish an IMPORT PGDUMP/MYSQL from any kind of IMPORT INTO, because for the former, I need to keep its table descriptor offline. For the IMPORT PGDUMP case, instead of adding the |
You know what, I'm going to scrap the |
The special value is just... zero, right? And then we only restore offline descriptors if they have a non-zero value, and we elide all rows at/above that value? |
We just don't restore the desc at all here, right? we don't want to keep it offline since there is no job that is going to come along and remove it later. Basically, it seems like this should be a) if importing into an existing table -- empty or not --, record the time that all import writes will be at/above in the desc b) when restoring, if a table is offline we do not restore the desc or data unless it has a import-writes-are-at-or-above time, in which case we do restore it but elide all rows at/above that time, and the table ends up public at the end of restore just like all the others. Does that sound right? |
Ah yes, for an in-progress IMPORT PGDUMP, we really should drop all of its associated descriptors on RESTORE. One piece you're missing: I only backup offline tables iff its I wonder if it's worth only focusing on the IMPORT INTO case for now, and revisit incrementally backing up IMPORT PGDUMP in stability/23.1. |
f5a79db
to
8722d84
Compare
removed the import type. Currently nothing gets set for IMPORT PGDUMP. Would like to merge this ASAP to make it easier to work on downstream stuff. |
8722d84
to
c1a3b60
Compare
Release note: none
This patch introduces the new ImportStartWallTime table descriptor field which provides the time at which an in-progress import began writing data to disk. This field is nonzero if the table is offline during an import. In future PRs, this field will be used to: - incrementally back up in progress imports, preventing a large BACKUP workload after an Import finishes. - elide importing keys in RESTORE, ensuring a table -- with an in-progress import in the back up-- contains no in-progress importing keys and is available on the restored cluster. Informs cockroachdb#85138 Release note: none
c1a3b60
to
5c16ad4
Compare
TFTR! bors r=dt |
Build failed (retrying...): |
Build failed (retrying...): |
Build failed (retrying...): |
Build succeeded: |
This patch introduces the new ImportStartWallTime table descriptor field which
provides the time at which an in-progress import began writing data to disk.
In future PRs, this field will be used to:
after an Import finishes.
in the back up-- contains no in-progress importing keys and is available on
the restored cluster.
Informs #85138
Release note: none