Implement release blocking job criteria

[spiffxp]

This is an umbrella issue for followup work to #346

That PR describes aspirational release blocking job criteria. This issue is intended to track the followup work, including:

• assignment of owners to all jobs
• descriptions added to all release-master-blocking jobs
• propose the creation of [email protected] or reuse of sig-foo-test-failures for all sigs that need to be responsive to test failures
• the creation of the release-informing dashboard, and moving jobs out of release-master-blocking to that dashboard
• a bigquery run that generates metrics for jobs currently on the release-master-blocking dashboard

/sig release
this is sig-release policy
/sig testing
this will be assisted by sig-testing tooling

EDIT 2019-07-23: AFAIK metrics is the only thing that remains to close this out

[jberkus]

Are we consolidating all non-blocking dashboards into -informing?

I'd be in favor of that.

[BenTheElder]

/cc

[spiffxp]

/milestone v1.14

I would like for us to implement this for the v1.14 release

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[justaugustus]

/remove-lifecycle stale

[justaugustus]

/help
/milestone v1.15

[spiffxp]

• assignment of owners to all jobs
• descriptions added to all release-master-blocking jobs
• propose the creation of [email protected] or reuse of sig-foo-test-failures for all sigs that need to be responsive to test failures

/assign
I'm handling this in #441

• the creation of the release-informing dashboard, and moving jobs out of release-master-blocking to that dashboard

this is done

• a bigquery run that generates metrics for jobs currently on the release-master-blocking dashboard

I would recommend someone look at https://github.com/kubernetes/test-infra/tree/master/metrics for this

[jeefy]

/assign @jberkus

[wojtek-t]

This is sort of the mechanism by which people review PRs ... I really hope you are not missing 75% or more of these confused

I actually bet I do if people don't ping me directly. I just opened a list of those where I was reviewer and I didn't see majority of those. If something is critical and really requires my attention, I should be at the very least assigned as approver (those I'm trying to follow, but I'm also missing some percentage of those).

That's not really reasonable, policies should be expected to evolve, not be set in stone, and tomorrow I may introduce some unforeseen form of testing that we need to account for.
I also don't think we should be trying to handle soak testing at all right now. There's no useful soak testing on which to inform policy.

I disagree with that. We shouldn't create a policy that we know want work as soon as we have soak tests. Those are important enough that I really believe we will prioritize them soon.

I'm not seeing an actual resolution in this thread,

There was a 1-hour-long discussion that happened during the zoom meeting. Where I think we roughly converged. @jberkus is actually documenting the outcome of it in his PR.

[BenTheElder]

I actually bet I do if people don't ping me directly. I just opened a list of those where I was reviewer and I didn't see majority of those. If something is critical and really requires my attention, I should be at the very least assigned as approver (those I'm trying to follow, but I'm also missing some percentage of those).

This is a separate discussion but ... this kinda defeats the purpose of having assigned reviewers. Please consider dropping a note to contribex about why / how this doesn't work so we can fix this. :/

I disagree with that. We shouldn't create a policy that we know want work as soon as we have soak tests. Those are important enough that I really believe we will prioritize them soon.

There has been no indication that we will have soak tests.

What we call soak tests now are easily among the longest failing tests we have and likely to be removed in the near future... These have been relatively unmaintained for on the order of year(s). Who is prioritizing them? I've heard zero discussion of this in SIG-Testing or SIG-Release ...

There was a 1-hour-long discussion that happened during the zoom meeting. Where I think we roughly converged. @jberkus is actually documenting the outcome of it in his PR.

Excellent. 👍

[spiffxp]

I've got an initial attempt at a dashboard that displays metrics relevant to release-blocking criteria: kubernetes/test-infra#13879 (comment)

http://velodrome.k8s.io/dashboard/db/job-health-release-blocking

[spiffxp]

I caught that the bazel jobs were postsubmits that didn't meet the "scheduled at least 3 hours" criteria so swapped them with periodics that did kubernetes/test-infra#13907

[spiffxp]

The serial job takes way too long, and is failing due to timeout. I think we should kick out egregious offenders, and encourage a pattern of adding a feature-specific job if the feature is truly necessary to be release-blocking.

Opened issues to start this for

• kubelet resource tracking tests: The serial regular resource usage tracking e2e tests are too slow kubernetes#81490
• HPA tests: The serial HPA e2e tests are too slow kubernetes#81491

[tpepper]

/assign @tpepper

[guineveresaenger]

/assign @guineveresaenger

[spiffxp]

@msau42 is looking to split out serial storage tests into another release-blocking job as well kubernetes/test-infra#13936

I feel like splitting tests into more parallel blocking jobs is a sound approach for now. But, it's only going to get us but so far before we run into new limits:

• we pick up additional overhead of standup/teardown of yet another cluster
• more jobs = more opportunity for one of them to flake, so overall reliability of master-blocking may go down
• what other jobs should be running these tests? eg: skew, upgrade/downgrade, scale, other envs, etc

At some point it's worth questioning why these tests need to be release-blocking, and if there is some sort of bar they should be held to. We presumably do this for Conformance tests, though IMO it's not as rigorously measured as it could be, and relies on extensive human review.

[jberkus]

Not completed despite the merger of #752. Mostly because there's still some issues unanswered:

• Revise Blocking criteria around flakiness - Revise Blocking criteria around flakiness #773
• Decide on procedure for jobs entering (and leaving) Blocking - Decide on procedure for jobs entering (and leaving) Blocking #774
• Document process/criteria by which we decide that Informing failures are tolerable - Document process/criteria by which we decide that Informing failures are tolerable #775

[alejandrox1]

will start with #775
/assign

[alejandrox1]

/remove-help
/milestone v1.17

[guineveresaenger]

Closing in favor of #773, #774, #775.

Thanks everyone!