openshift: "503 Service Unavailable" flake

[jim-minter]

$ oc get route/nginx-example -n default -o jsonpath={.spec.host}
2018/05/14 17:44:00 got status "503 Service Unavailable" while trying to access nginx-example-default.40.114.111.175.nip.io

[jim-minter]

https://circleci.com/gh/Azure/acs-engine/26335

[jim-minter]

@Kargakis can you take a look?

[0xmichalis]

Hrm, likely nip.io downtime? This will not be an issue if we ever switch to use azure dns in the router.

[jim-minter]

Seems improbable that it was DNS if it got as far as receiving a 503 response?

[0xmichalis]

/kind flake

[0xmichalis]

default                             nginx-example-1-build            0/1       Completed   0          3m        10.128.0.7   ocp-master-17366486-0
default                             nginx-example-2-k7nm5            1/1       Running     0          23s       10.130.0.8   ocp-compute-17366486-0

The pod is running OK

Nothing in the router log:

I0605 08:56:25.913295       1 template.go:260] Starting template router (v3.9.25)
I0605 08:56:25.915317       1 metrics.go:157] Router health and metrics port listening at 0.0.0.0:1936
I0605 08:56:25.967966       1 router.go:441] Router reloaded:
 - Checking http://localhost:80 ...
 - Health check ok : 0 retry attempt(s).
I0605 08:56:25.968041       1 router.go:228] Router is including routes in all namespaces
I0605 08:56:26.295429       1 router.go:441] Router reloaded:
 - Checking http://localhost:80 ...
 - Health check ok : 0 retry attempt(s).
I0605 08:56:31.302262       1 router.go:441] Router reloaded:
 - Checking http://localhost:80 ...
 - Health check ok : 0 retry attempt(s).
I0605 08:56:36.298116       1 router.go:441] Router reloaded:
 - Checking http://localhost:80 ...
 - Health check ok : 0 retry attempt(s).
I0605 08:56:43.306637       1 router.go:441] Router reloaded:
 - Checking http://localhost:80 ...
 - Health check ok : 0 retry attempt(s).
I0605 08:56:51.221342       1 router.go:441] Router reloaded:
 - Checking http://localhost:80 ...
 - Health check ok : 0 retry attempt(s).
I0605 08:57:41.852712       1 router.go:441] Router reloaded:
 - Checking http://localhost:80 ...
 - Health check ok : 0 retry attempt(s).
I0605 08:59:40.815927       1 router.go:441] Router reloaded:
 - Checking http://localhost:80 ...
 - Health check ok : 0 retry attempt(s).
I0605 08:59:46.655505       1 router.go:441] Router reloaded:
 - Checking http://localhost:80 ...
 - Health check ok : 0 retry attempt(s).
I0605 08:59:51.647294       1 router.go:441] Router reloaded:
 - Checking http://localhost:80 ...
 - Health check ok : 0 retry attempt(s).
I0605 09:01:30.250751       1 router.go:441] Router reloaded:
 - Checking http://localhost:80 ...
 - Health check ok : 0 retry attempt(s).
I0605 09:01:35.218554       1 router.go:441] Router reloaded:
 - Checking http://localhost:80 ...
 - Health check ok : 0 retry attempt(s).
I0605 09:02:38.462433       1 router.go:441] Router reloaded:
 - Checking http://localhost:80 ...
 - Health check ok : 0 retry attempt(s).
I0605 09:02:43.448130       1 router.go:441] Router reloaded:
 - Checking http://localhost:80 ...
 - Health check ok : 0 retry attempt(s).

[0xmichalis]

Need to check whether the pod uses a readiness check and also whether the deployment config waits for it to turn ready.

[0xmichalis]

Need to check whether the pod uses a readiness check and also whether the deployment config waits for it to turn ready.

Both readiness probe is used and the DC has sane rolling parameter defaults.

[0xmichalis]

ugh, retries were using microseconds for intervals instead of the intended milliseconds - opened #3202