considerations about enhancement tracking

[neolit123]

introduction

each release, before feature freeze, the release team enhancement sub-team has the task to verify the status of pending features to be merged in k/kubernetes.

the team uses a google docs spread sheet that includes a lot of details like pending PRs, SIG owners, feature state and even some statistics.

each feature has a tracking issue located in the kuberenetes/enhancements (k/e) repository and after a recent change all features must also have a KEP, also located in the same repository.

problem statements

while attempting to obtain information about the current state-of-implementation of a certain feature, the release team has to post messages in k/e tracking issues, asking owners and maintainers:

what pending external to k/e tracking issues are there?

this topic is noisy because there is no standardization. every SIG uses a different schema for tracking their work. some use separate repositories, some use multiple issues for each item, some use a single tracking issue for each item.

proposal

the issue template has to be updated to include a list of tracking issues (or a single umbrella tracking issue). a list of tracking issues should look like this:

state: Alpha
target: 1.14
description of item1
- [ ] kubernetes/kubernetes#1234567
description of item2
- [ ] TODO

description of itemX must match with an item in the KEP.
kubernetes/kubernetes#1234567 is a tracking issue of an item.
- [ ] is a checkbox that indicates completion.

what PRs are merged and what are yet to be created?

often times this question is hard to answer. multiple PR can accumulate over the course of the cycle for the exact same feature or work items. this can be very noisy and difficult to track by the release team.

proposal

PRs should not be an indication if a feature is "ready" for this release. tracking issues should be. the maintainer of the k/e main tracking issues should add checkmarks and links to the features that are completed. an external tracking issue and its PRs should cross reference. once all PRs are in the external tracking issue should be closed.

state: Alpha
target: 1.14
description of item1
- [x] kubernetes/kubernetes#1234567
description of item2
- [x] kubernetes/kubernetes#7654321

do you have documentation and test coverage?

arguable the hardest items to complete for a feature...

proposal

the proposal here is fairly simple. just extend the list to add tracking items for tests and documentation:

state: Beta
target: 1.15
description of item1
- [x] kubernetes/kubernetes#1234567
description of item2
- [x] kubernetes/kubernetes#7654321
documentation:
- [ ] kubernetes/website#12345
tests:
- [ ] kubernetes/test-infra#12345

note that an Alpha state might not require a testing plan or documentation.
also that website and test-infra are just example locations. repos for tracking issues are matter of preference.

is this feature going to "make it" this release?

a vague question for both sides, as those who are asking might not know what the feature does, while those who are questioned might not know if they can complete in time.

the feature lifecycle is undefined (or probably defined in a hard to find document).
if one googles "kubernetes feature lifecycle alpha beta ga" it gives them this link:
https://kubernetes.io/docs/reference/using-api/deprecation-policy/

proposal

the issue template from k/e should link to such a lifecycle document.

if such a document does not provide the following information, it should:

• Alpha means experimental feature, that does not require documentation, while tests are nice to have.
• Beta means that a feature should have tests and documentation in place and be ready for consumption.
• GA means that all important bugs from beta are fixed, tests and documentation are updated and the feature is ready for general availability.

state: GA
target: 1.16
description of item1
- [x] kubernetes/kubernetes#1234567
description of item2
- [x] kubernetes/kubernetes#7654321
documentation:
- [x] kubernetes/website#12345
tests:
- [x] kubernetes/test-infra#12345

at this point, if a release team member sees the above list, verifies that all items are indeed closed and completed, there would be no need to ask questions even

examples of implementation lifecycles of a feature

state: Alpha
target: 1.14
description of item1
- [x] kubernetes/kubernetes#1234567

state: Beta
target: 1.15
description of item1
- [x] kubernetes/kubernetes#1234567
description of item2
- [x] kubernetes/kubernetes#1234567
documentation:
- [x] kubernetes/website#12345
tests:
- [x] kubernetes/test-infra#12345

state: GA
target: 1.16
description of item1
- [x] kubernetes/kubernetes#1234567
description of item2
- [x] kubernetes/kubernetes#7654321
documentation:
- [x] kubernetes/website#12345
tests:
- [x] kubernetes/test-infra#12345

state: Alpha
target: 1.14
description of item1
- [x] kubernetes/kubernetes#1234567
description of item2
- [x] kubernetes/kubernetes#1234567

state: Beta
target: TODO

state: GA
target: TODO

other considerations:

handling subprojects

subprojects should not be treated in a special way (e.g. see "kubeadm" here and here).

all features that land in k/k should follow a standardized way of organization of the tracked issues.
links from k/e tracking issues could link to any GitHub issue.

project boards (alternative approach)

while SIGs such as Windows and API-Machinery are already using project boards for tracking their own work, they can be tricky with regards to:

• permissions (who can edit?).
• organization of tabs, searching for labels, grouping of issues.
• they collide with milestones.
• AFAIK they do not have edit history.

while i definitely see the potential of using them, i went with the alternative approach of linking individual tracking issues. this approach is more linear and easy to follow by the release team, but requires more from the SIGs.

if SIG Release prefers project boards, we can go with this idea and link k/e tickets to project boards, but this requires some extra planning.

a problem with permissions

for both approaches, there is a problem with permissions. currently, only milestone maintainers can edit the k/e tickets (some of these rot with outdated information). while project boards also require a specific team to be able to edit. project boards win here in terms of flexibility, but again lack the linearity of the implementation timeline and edit history.

ideally the list of k/e ticket maintainers should be extended outside of milestone maintainers. milestone maintainers have the power to apply milestone in k/k while here we can have a separate group e.g. feature maintainers, which is a limited scope.

auto-pings vs human contact

often times developers are busy and/or they forget. so while we can continue to ping them in k/e tickets and slack, the general reminders can be automated in the lines of "hey, feature freeze is soon. please edit the OP of this document to include relevant items and/or update the status". if there is no follow up update after the automated message, human contact can follow on slack.

[neolit123]

/assign @tpepper @justaugustus
cc @timothysc @spiffxp
cc @kubernetes/sig-release
cc @kubernetes/sig-cluster-lifecycle

/kind design
/priority important-longterm

[nikopen]

+1000, all are really good proposals that will ease the burden of the release team as a whole, especially the roles of bug triage and enhancements.

All of bug triage pre-freeze work can be clarified and automated with the above suggestions, and ease the ticket filtering during freeze and burndown with more organization by the feature owners and related SIGs - whether it is project boards, tracking issues, feature lifecycle document links etc.

If the underlying organization is solid and the proper linking is done by feature owners themselves, there is no point for manual work by other people. This will give all the visibility that is needed.

I suggest they're all implemented for v1.15 as they are and communicated in a clear way to all contributors to be aware.

[rosti]

The format looks suspiciously like YAML, which brings into mind a whole plethora of release team aid scripts and automation to query and verify the release readiness.

[justaugustus]

@neolit123 / @nikopen / @rosti -- you're all stumbling onto what we (SIG PM) had in mind for moving Enhancements Tracking forward. I'll write something detailed up when I'm back at a computer.
/remove-priority important-longterm
/priority critical-urgent
/sig pm release
/milestone v1.15

[neolit123]

@justaugustus

you're all stumbling onto what we (SIG PM) had in mind for moving Enhancements Tracking forward. I'll write something detailed up when I'm back at a computer.

interesting, could you please share VOD links (optionally timestamp) of when this was discussed?

[timothysc]

I'm torn, and I need to ruminate on this, b/c the purposes of keps was to get aggregate feature signal for a release, and it's clear we need to refine the process and automate the heck out of it.

Right now we have a hybrid approach that lives in an "in-between" world. Ideally I would like KEPs to be the source of truth and contain enough meta-data so release owners can just parse.

How SIGS do their work is up to them, but final read out imo should be a KEP PR with the punchlist done.

I'm going to start tackling these sets of problems in the next quarter.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[justaugustus]

/remove-lifecycle stale

[tpepper]

Since a release team hasn't implemented something, could tooling be build from somebody in Release Engineering?

Or are alternatives such that we can close this?

[tpepper]

@justaugustus it seems like this is not priority critical urgent, can we demote it since there's been no work?

[neolit123]

please, note that i'm fine with closing this ticket especially after the discussion that alternative approaches can be taken. sadly, i forgot most of the details on this topic.

[mrbobbytables]

/assign

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[justaugustus]

/remove-lifecycle stale

[lasomethingsomething]

Hi, with this issue partially acted upon via #17473, what might be the remaining unresolved questions/to-do's on this issue?

[lasomethingsomething]

Hiya, circling back on my question ^^

[neolit123]

hello, i saw the previous question too, but was hoping that one of the sig-release maintainers will reply.

with this issue partially acted upon via #17473

it doesn't seem directly related.

the idea here was to allow the release team to monitor the tracking issue progress via automation from a machine readable format.
the problem with this idea is that someone has to update the status of a tracking issue and maintainers may forget to do that, thus the nagging (human interaction) from the release team on tracking issue status would happen regardless.

in any case i'm fine if this ticket is closed and if there are better ideas.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[palnabarun]

/remove-lifecycle rotten

[saschagrunert]

in any case i'm fine if this ticket is closed and if there are better ideas.

I think we can solve some parts of this issue with the Receipts KEP: kubernetes/enhancements#2167 / kubernetes/enhancements#2140

Let's close this for now and reopen if we feel that this should be addressed.
/close

[k8s-ci-robot]

@saschagrunert: Closing this issue.

In response to this:

in any case i'm fine if this ticket is closed and if there are better ideas.

I think we can solve some parts of this issue with the Receipts KEP: kubernetes/enhancements#2167

Let's close this for now and reopen if we feel that this should be addressed.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.