feat(en): Unify snapshot recovery and recovery from L1

[slowli]

What ❔

• Replaces storage_logs.{address, key} fields in the snapshot with storage_logs.hashed_key.
• Makes storage_logs.{address, key} fields in Postgres optional and adapts the codebase for it.

Why ❔

• This will make snapshot data equivalent to data obtainable from L1, and thus would allow to unify snapshot recovery and recovery from L1 data.
• Decreases the snapshot size somewhat.

Checklist

• PR title corresponds to the body of PR (we generate changelog entries from PRs).
• Tests for the changes have been added / updated.
• Documentation comments have been added / updated.
• Code has been formatted via zk fmt and zk lint.
• Spellcheck has been run via zk spellcheck.

[slowli]

I feel that backward compatibility is clunky, but I'm not sure how to make it better.

• The snapshot creator now supports creating both v0 and v1 snapshots (configured via an env variable). If we start creating only v1 snapshots, the old EN versions will fail to recover from a snapshot.
• Snapshot creator logic is largely unchanged, meaning that it's impossible to create v0 and v1 snapshots for the same L1 batch, or to start creating a v1 snapshot while v0 is in progress (or vice versa). The latter looks like an arbitrary restriction, but it looks moderately easy to fix. Having an L1 batch number as the snapshot ID is more deep-rooted in the codebase / API, so there could be issues changing it.
• New EN versions support recovery from both v0 and v1 snapshots. However, there's currently no way to command the EN to recover from a snapshot with a specific version (which could be useful for E2E testing), other than manually calling RPC methods and finding an L1 batch number with the appropriate version, and then specifying recovery from this specific snapshot.

[tomg10]

Well... We knew you as soon as we release the Node, this unification would be painful. I don't think we should add support for having multiple versions for the same l1 batch, I'm also against changing API just for executing this migration. Can you write down migration plan to the new format as you envision it now?

[slowli]

@tomg10 I don't think the migration would be that painful if we design it the right way, but it certainly would be non-trivial; I don't think not having backward compatibility is an option. As far as I see it, it goes as follows:

• We still have a single snapshots table with the l1_batch_number primary key.
• OTOH, we can create v0 and v1 snapshots using differing snapshot creator jobs. The simplest way to do this is make the snapshot creator always start a new snapshot unless explicitly asked to continue a previous one (e.g., via a command-line arg). Naturally, the creator would not create a snapshot if one already exists for the target L1 batch (even if it has a different version). If this is the case, we can run v0 and v1 snapshot creation with some overlap, provided that the DB replica has sufficient I/O capacity.
• On the API side, I think we can change API in a backward-compatible way. Namely, we can add an optional param like version (or maxVersion?) to snapshots_getAllSnapshots set to 0 by default; the method would filter snapshots by version correspondingly. New ENs would set this param to 1. This ensures that old ENs would continue working as expected; they would only get v0 snapshots from snapshots_getAllSnapshots.

We can gradually make v0 snapshots less frequent and eventually stop creating them at all. Effectively, this will mean deprecation of the v0 format. Naturally, it needs to be communicated as "please use a new EN version if you want to recover from a snapshot".

One open question I haven't mentioned above is whether we want to have control over how EN recovers from a v0 snapshot; theoretically, it could still put key preimages in storage_logs (in the current version of this PR, it doesn't). So if we think there's a risk that we've missed some uses of preimages, we can make v1 format support opt-in on the EN side initially, and only make it the default after we're sufficiently sure that it works (e.g., by running the E2E workflow). It doesn't look too complicated, but still requires some work. WDYT? cc @RomanBrodetski @popzxc

[slowli]

I think I've figured out how migration to v1 snapshots can be made less painful, at least initially, and implemented it in the recent commits. Basically, allow the EN to drop storage key preimages if recovering from snapshot v0; in this way, we can test soundness of not having key preimages without ever creating snapshots v1. We can set this flag for E2E recovery test and for some of our ENs. I can theoretically now remove v1 snapshots altogether from this PR; but I'd prefer not to because they are implemented already 🙃

[RomanBrodetski]

Great work!

I don't have capacity to do a detailed review, but I had a quick look and it looks good. I also think there is a lot of value in delivering this - even though it won't be used that much, it's important to have this functionality - it's important for the chain liveness in case of a catastrophic event

[popzxc]

I can't say I did a very thorough review (not enough time, unfortunately), but I've read through the whole diff and didn't find any significant issues.
The change makes sense, and the migration process seems to be adequate (though we probably will need to involve DevRell to communicate it).

[popzxc]

The third time's the charm

[popzxc]

OK, here goes the fourth