flake Available origin-docker-excluder version 3.6.0 is higher than the upgrade target version 1.5

[tbielawa]

TASK [Get available excluder version] ******************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/pre/validate_excluder.yml:7
ok: [localhost] => {
    "changed": false, 
    "cmd": [
        "repoquery", 
        "--plugins", 
        "--qf", 
        "%{version}", 
        "origin-docker-excluder"
    ], 
    "delta": "0:00:25.714260", 
    "end": "2017-04-20 13:05:26.989698", 
    "failed": false, 
    "failed_when_result": false, 
    "generated_timestamp": "2017-04-20 13:05:27.009577", 
    "rc": 0, 
    "start": "2017-04-20 13:05:01.275438", 
    "stderr": [], 
    "stdout": [
        "3.6.0"
    ], 
    "warnings": []
}

TASK [Docker excluder version detected] ****************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/pre/validate_excluder.yml:14
ok: [localhost] => {
    "generated_timestamp": "2017-04-20 13:05:27.062843", 
    "msg": "origin-docker-excluder: 3.6.0"
}

TASK [Check the available origin-docker-excluder version is at most of the upgrade target version] ***
task path: /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/pre/validate_excluder.yml:18
fatal: [localhost]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "generated_timestamp": "2017-04-20 13:05:27.123663", 
    "msg": "Available origin-docker-excluder version 3.6.0 is higher than the upgrade target version 1.5"
}

PLAY RECAP *********************************************************************
 [WARNING]: Could not create retry file '/usr/share/ansible/openshift-
ansible/playbooks/byo/openshift-cluster/upgrades/v3_5/upgrade.retry'.
[Errno 13] Permission denied: u'/usr/share/ansible/openshift-
ansible/playbooks/byo/openshift-cluster/upgrades/v3_5/upgrade.retry'
localhost                  : ok=48   changed=2    unreachable=0    failed=1   

++ export status=FAILURE
++ status=FAILURE
+ set +o xtrace
########## FINISHED STAGE: FAILURE: INSTALL ORIGIN GA AND UPDATE ##########
Build step 'Execute shell' marked build as failure

https://ci.openshift.redhat.com/jenkins/job/test_pull_request_openshift_ansible_extended_conformance_install_update/140/consoleFull#77014445158b6e51eb7608a5981914356

Encountered in openshift/openshift-ansible#3944 (comment)

[dcbw]

@ewolinetz not a test flake? i see you removed the label...

[ewolinetz]

it may be due to a change to the tests, i think its happening too frequently to be a flake. waiting to confirm first

[sdodson]

@dcbw Do we use that label when it's 100% failure rate?

What's happening is that we changed the test jobs earlier today to disable the excluders (see openshift-eng/aos-cd-jobs#204) because the excluders that are in the 1.5.0-0.rc1 builds are broken and without writing additional automation we cannot upgrade out of that broken situation.

However now that they're disabled this has exposed a problem in openshift-ansible that happens only when you have multiple repos enabled. The test jobs, as far as I can tell, enable all the repos, 1.4, 1.5, 3.6, etc which is not a situation we generally see in OCP, whenever the 3.6 repo is enabled you intend to upgrade to 3.6.

Looking at options to fix this in ansible.

[ingvagabund]

This is actually a bug, not a flake: https://bugzilla.redhat.com/show_bug.cgi?id=1434679

[ingvagabund]

I had a PR fixing that that got lost in the process :(. Sorry about that

[dmage]

We need tag kind/test-flake to be able to run test/merge once again.

[0xmichalis]

Increasing to P0 since this is blocking the merge queue

[sdodson]

@dmage sure, but until this is fixed there's no point in re-submitting, it's just going to fail again.

[sdodson]

Changes merged, working to get the new builds into the centos paas sig repos at which point this should stop blocking the queue.