Fix extract transcript #7434
Fix extract transcript #7434
Conversation
|
FYI @warner I plan on figuring something out for handling bundleIDs in the workerOptions. |
| @@ -131,6 +140,14 @@ export function makeTranscriptStore( | |||
| return transcripts; | |||
| } | |||
|
|
|||
| function* readVatTranscript(vatID) { | |||
There was a problem hiding this comment.
We might call this readFullVatTranscript to make it clear that it's fetching multiple spans, which is not something we'd normally do for operational purposes. readVatTranscript isn't wrong.. it fits with our hierarchy of names:
- "transcript": everything from vat creation to vat termination (or latest delivery)
- "incarnation": everything from one upgrade to the next upgrade (or latest delivery, or termination)
- "span": everything from heap snapshot save/load to next upgrade, snapshot write, termination, or latest delivery
- "item" or "entry": one delivery
but "transcript" to mean "everything" is so rarely used, that it doesn't hurt to give it a longer name.
| function* readVatTranscript(vatID) { | ||
| for (const { startPos, endPos } of sqlDumpVatSpansQuery.iterate(vatID)) { | ||
| for (const row of sqlDumpItemsQuery.iterate(vatID, startPos, endPos)) { | ||
| yield row; |
There was a problem hiding this comment.
be thinking about what happens if we've pruned the span and/or the items, we probably want this to throw an error rather than yield broken items.. maybe?
There was a problem hiding this comment.
Good call! This method will simply be missing the historical span entries that were pruned. While this is a "full transcript" I think it's acceptable for it to yield a partial history given what we yield is both the transcript entry and its position.
I've updated the consumer (extract-transcript) to not write anything until it sees a startVat delivery. There was already logic to throw if the entries were not continuous (gap in positions). Before the change it would have thrown on pruned historical transcript spans. Now it will only return a full transcript of the last incarnation, but handle partially pruned transcripts of previous incarnations.
| transcriptNum += 1; | ||
| const entry = { transcriptNum, d: delivery, syscalls }; |
There was a problem hiding this comment.
BTW I'd love to find a clean way to keep the transcriptNum in the DB, to avoid potential drift or fencepost errors with this external incrementing counter.
There was a problem hiding this comment.
It is in the DB already. Here it's from the slog, which doesn't include the transcript pos, so it has to be regenerated. It would indeed be nice to include this info in the slogfile.
| return kvStore.get(key); | ||
| const value = kvStore.get(key); | ||
| if (value === undefined) { | ||
| throw Error(`Inexistent kvStore entry for ${key}`); |
There was a problem hiding this comment.
I'd say "non-existent" or "missing", but "inexistent" sounds like a cool word too :)
There was a problem hiding this comment.
You caught my Latin roots :)
| } | ||
| } else if (/(supervisor|lockdown)(B|-b)undle/.test(vatName)) { |
There was a problem hiding this comment.
is this abusing the CLI argument so that when you say "please replay the vat named supervisor-bundle", it will actually extract the given bundle and write it to disk?
weird, but I'm ok with it
There was a problem hiding this comment.
I thought I had removed this, since it's been included in the extract-bundle tool instead...
| for (const { position, item } of transcript) { | ||
| const entry = JSON.parse(item); | ||
| if (entry.d[0] === 'startVat') { | ||
| fs.ftruncateSync(fd); |
There was a problem hiding this comment.
This truncate is new, right? Were you having problems with overwriting pre-existing files that needed it? If so, it seems like it'd be better to open the file in the mode that overwrites/truncates the old one, or at least do the truncation immediately after opening, instead of inside this loop.
Or.. oh, are you writing every entry, but then you discover a startVat and retroactively decide that the transcript ought to begin here?
Ah, ok, so what you really want is a transcriptStore.readIncarnationItems(vatID, incarnationID), or a readCurrentIncarnation(vatID), so you don't have to deduce where the current incarnation begins.
As we improve the transcript to track upgrades and snapshot reads/writes, we should consider exposing incarnation numbers to the DB (as I think we discussed already, you and me and @FUDCo, and I think we agreed in principle on doing it). Probably sooner rather than later.
There was a problem hiding this comment.
Yup that was the only way I found to only extract the latest incarnation given our DB model, and the motivation for my request to include the incarnation number.
mhofman
force-pushed
the
mhofman/fix-extract-transcript-from-kernel-db
branch
from
c0c0997 to
4cbfc93
Compare
Does not include bundle info
mhofman
force-pushed
the
mhofman/fix-extract-transcript-from-kernel-db
branch
from
d3a7e00 to
d67bec4
Compare
refs: #7432
Description
Update the transcript extract tools (and swingstore) to support the latest changes to transcripts.
create-vatin the slog extract tool to be more similar to the kernel DB extract tool (minus the bundle ID changes)The swingstore is extended as needed to support the extract tooling.
Some of these commits are cherry-picks from #6723 that were omitted from #7432
Security Considerations
None, tooling only
Scaling Considerations
None
Documentation Considerations
Internal tooling
Testing Considerations
Manually verified with from the state and slog created by a modified
test-vaults-integration.jsbootstrap test. Verified in a merge with #7432 that the transcripts could be replayed successfully.