Flake deploymentconfigs with test deployments [Conformance] [It] should run a deployment to completion and then scale to zero

[abutcher]

• Failure [26.555 seconds]
deploymentconfigs
/go/src/github.com/openshift/origin/test/extended/deployments/deployments.go:908
  with test deployments [Conformance]
  /go/src/github.com/openshift/origin/test/extended/deployments/deployments.go:309
    should run a deployment to completion and then scale to zero [It]
    /go/src/github.com/openshift/origin/test/extended/deployments/deployments.go:308

    Expected
        <string>: --> pre: Running hook pod ...
        test pre hook executed
        --> pre: Success
        --> Scaling deployment-test-1 to 2
        --> Waiting up to 10m0s for pods in rc deployment-test-1 to become ready
    to contain substring
        <string>: --> Success

    /go/src/github.com/openshift/origin/test/extended/deployments/deployments.go:256

https://ci.openshift.redhat.com/jenkins/job/test_pull_request_openshift_ansible/87/consoleFull#-210271339656cbb9a5e4b02b88ae8c2f77

[enj]

Seen in https://ci.openshift.redhat.com/jenkins/job/merge_pull_request_origin/657/

[bparees]

https://ci.openshift.redhat.com/jenkins/job/merge_pull_request_origin/706/

[smarterclayton]

This is now the #1 break in the queue and testing on master

[mfojtik]

@smarterclayton ok, this code does oc logs -f pods/deployer to collect the logs. It stores the output of that command into out variable when the -f drops (which happen when the deployer pod is gone/complete). It seems like:

1. We miss buff.sync() somewhere to flush all logs into out
2. Something changed in oc logs -f and we won't get the complete logs from the pod or we miss the last line of the log output (did something changed in upstream re: logging?)

[mfojtik]

I ran this test for some time in a loop and no luck reproducing this :/

From the log it also seems that the rollout completed successfully, there is no error in the third RC and the rollout took reasonable time as well.

[mfojtik]

@Kargakis @smarterclayton ok, the only reasonable way to fix it and unblock the queue is to use the deployment config condition to check the success (i can check the NewRcAvailableReason).

I will add FIXME there for now and will continue investigating.

PR: #14395

[mfojtik]

This is journald rate-limiter issue, as a band-aid we are going to disable the --> Success check in logs (we already check the conditions/reason to make sure the rollout completed successfully).

Moving this to P2 just to track this once the rate-limiter is fixed.

[smarterclayton]

Still happening

[bparees]

https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_conformance_install_update/5157/

[bparees]

Moving this to P2 just to track this once the rate-limiter is fixed.

@mfojtik was the rate-limiter ever fixed?

[mfojtik]

@bparees it was tweaked for some tests but not for deployment tests... so no, it was not fixed (the original issues is still opened).

[jim-minter]

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17248/test_pull_request_origin_extended_conformance_install_update/9178/

[tnozicka]

@mfojtik close in favor of #17747 ?

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close