Implement storage versioning

[serathius]

Implementing a backward compatibility for etcd data directory is a prerequisite to implement downgrades (#11716)

Proposal https://docs.google.com/document/d/1yD0GDkxqWBPAax6jLZ97clwAz2Gp0Gux6xaTrtJ6wHE/edit?usp=sharing

Milestone 1

Storage version is available for snapshots.

• Etcd saves storage version during first raft snapshot Etcd version should be easier to access when performing snapshot/restore #13070
• During bootstrap etcd should panic when encountering higher version Implement schema migration and validate storage version during boostrap #13200
• Add etcdutl migrate --force as an escape hatch etcdutl: Implement migrate --force command #13220

Milestone 2

Etcd code is version aware making downgrade implementation possible

• Implement version gating for all features
• Implement version annotations for golang structs and proto Annotate proto messages with version and detect etcd version generated the wal #13216
• Create a DB file schema
  • Abstract modification of DB fields
    • etcdserver: Move Read/Update methods on Meta bucket to one place #13155
    • etcdserver: Extract functions for setting and reading compaction information in backend #13160
    • etcdserver: Membership uses MembershipStorage interface instead of directly accessing Backend #13161
    • etcdserver: Move read/update methods on Auth bucket to one place #13162
    • etcdserver: Move all get/put/delete on AuthUsers and AuthRoles to buckets module #13163
    • etcdserver: Move put/read/delete on Alarm bucket to bucket package #13164
    • etcdserver: Move get/put/delete on Lease bucket to bucket package #13165
• Add static analysis to validate annotations Implement WAL static analysis to ensure that new entries will be properly annotated #13490
• Implement simple downgrade upgrade tests that verify storage versioning (Implement single node downgrades #13405)

Milestone 3

Downgrades can be implemented based on storage versioning

• Etcd should validate storage version during bootstrap
• Etcdutl migrate can upgrade and downgrade data dir files
  • Fix ClusterVersionSet blocking downgrades to v3.5

Followups

• Fix setting clusterversion after downgrade&upgrade timeout
• Automate snapshotting on downgrade

[lilic]

Thanks for the detailed document and this issue! 🎉 Looks good overall to me.

What is the diff between this in milestone 1:

Add etcdutl migrate --force as an escape hatch

and this task in milestone 2?

Implement etcdutl migrate

Also, could we have some issues open for contributions? For example if someone new or existing contributors want to contribute one or two of these tasks maybe?

[serathius]

What is the diff between this in milestone 1:

Add etcdutl migrate --force as an escape hatch

and this task in milestone 2?

Implement etcdutl migrate

First milestone introduces storage version that should be used to prevent using outdated tools/etcd to load storage data. For example etcd v3.6 should not be able to start from data left by v3.7 without doing explicit downgrade of this data.

This is technically a breaking change as currently you could technically load etcd v3.4 data by etcd v3.0 and vice versa. Official we only support only one by one upgrade (v3.0 -> v3.1 -> v3.2 -> v3.3 -> v3.4) and no downgrades (use backup), but there is no mechanism that would prevent user from shutting themselves in a foot.

Milestone 1 introduces mechanism to explicitly prevent this, so we need to introduce an escape hatch for users that still want to do an downgrade, but they should do it explicitly by running migrate --force.

Difference between Milestone 1 and 3 etcdutl migrate is that the first one just implements the escape hatch, and the second whole schema upgrades and downgrades.

[serathius]

Also, could we have some issues open for contributions? For example if someone new or existing contributors want to contribute one or two of these tasks maybe?

As implementing versioning of storage touches basically whole server codebase I was worried about potential code conflicts before we update the codebase structure. To minimize future conflicts I have started from large scale refactors to restructure code to accommodate versioning, but got bottle-necked on reviews.

Currently I have local going couple of PR in future as my published PRs are waiting for review from a week. I don't think adding new contributors will help the effort or be a good experience as they would have the same problem.

I'm happy to start creating issues when the ground work will be done.

[serathius]

@YoyinZyc @hexfusion @wenjiaswe @ptabor
Could you do a review the design and give an approval on the design, so we have a community consensus and can move forward with implementing the breaking changes.