etcd Go maintenance plan

[hexfusion]

In the past, it has been the policy to maintain (as long as possible) the same Go version per minor release. So giving the example of release-3.4 we have and continue to build new releases with Go 1.12.x. I think there are some advantages to this including shielding etcd from possibly complicated performance regressions. In the past Go version iteration was on a much slower cadence than it is today. But now Go is iterating is quite fast and managing things like CVE on EOL Go versions can become unfortunate.

The Go team maintains 2[1] versions of Go which unfortunately puts our latest versions of etcd in an undesirable condition. While all users are free to build and compile etcd with any version of go they wish they do so without the upstream test cycles and validation/blessing of maintainers.

I think it would be worthwhile to consider adopting a policy that states within the supported versions of etcd efforts will be made to align the Go version we build and ship upstream etcd with supported versions of Go (within reason). This policy would come as a burden to maintainers as a group and should require explicit responsibilities from maintainers tasked with these maintenance tasks.

One thought is having a maintainer(s) responsible for bumping Go on the supported versions of etcd. The task would be to ensure CI is aligned with expectations and as a group look at integrating performance regression checks into CI. There should be no rush to bump etcd to a new Go release version but all attempts should be made to align with upstream Kubernetes where possible. I think moving forward 3.5+ this should be more straightforward but fairly large gaps still exist in 3.3 and 3.4 releases.

Let's talk about this further at the community meeting tomorrow?

cc @ptabor @gyuho @wenjiaswe @marun @lilic @retroflexer

[1] https://golang.org/doc/devel/release.html#policy
[2] #12330
[3] #12732

[ptabor]

I think we should update the CI or build binaries using the newer version, but at the same time I think we should cease using new features/libraries to would force our clients to upgrade go version.

I.e. if etcd-3.4.0 was compilable&testable using go-1.12, 3.4.99 should remain go-1.12 compatible.

The problem would also be less significant if we have minor release more frequently (e.g. every 6m).

[hexfusion]

great input, lets try to harden this.

[lilic]

The problem would also be less significant if we have minor release more frequently (e.g. every 6m).

+1 from me on this. What are the current issues around not doing those more frequently? Or is there just no need?

I.e. if etcd-3.4.0 was compilable&testable using go-1.12, 3.4.99 should remain go-1.12 compatible.

But we could bump to the latest go 1.12.x version right? go 1.12.17 seems to be the latest.

[lilic]

I opened #12847 as per the above comment by @ptabor and per discussion in yesterdays meeting. I thought it would be good to have one place to sync what needs to be done to get to more frequent releases. Let me know what you think, thank you!

[lilic]

As for this issue around go maintenance plan, yesterday in the meeting it was discussed:

• Add testing in etcd release-3.4 in CI for go v1.15 (I am happy to take this one if everyone is okay with it?)
• etcd release v3.5 should be tested against go v1.16
• Right now there won't be extra binaries for the multiple supported go versions

Let me know if I missed any other items?

[ptabor]

Thank you.

1. SGTM
2. I think v3.5 will just fully switch to 1.16 (so not only testing, but binaries build & go.mod as well)
3. SGTM

[lavacat]

@ptabor @ahrtr @ptabor @hexfusion @lilic Currently tests are broken in go 1.15. We need to update bolt to newer version to make it work

Let's decide if we want to migrate 3.4 to latest go version. 1.15 is already EOL. Go 1.18 is released, so we should use 1.17 or 1.18. If we don't want to migrate, I can remove 1.15 references and keep everything on 1.12.

My suggestion is to try moving everything to 1.17 if there are too many changes or failures, we can decide to keep 1.12. Personally I'd prefer to have same go version for tests and build.

@ptabor Based on your comment my understanding is that we should keep go 1.12 reference in go.mod. That's fine, but FYI, Kube is changing those for older supported releases.

[ahrtr]

Golang 1.17 is definitely not an option for stable releases, including 3.4 and 3.5, because there is a breaking change in Golang 1.17. Please see my explanation in issuecomment-115301872.

Proposed actions:

1. Bump go version to 1.16 for release 3.4. I already submitted a PR 14136 for this.
2. We release a minor etcd version (such as 3.6, 3.7, etc.) every 6 months, and bump the go version for each minor etcd release.
   2.1) Starting from 3.6 (main), let's bump go 1.18, and bump go 1.19 for 3.7 and so on. If there is no any objection, I will submit a PR to bump go 1.18 for 3.6(main) soon.

[jmhbnz]

Closing this issue as:

• We now have a baseline documented go version policy here: https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/dependency_management.md#golang-versions
• Adopt Kubernetes Go version update process #16592 Is open to track our discussion for the next iteration of our golang version policy.