[chectl] default timeouts don't work in 'chectl server:deploy' command on minikube

[dmytro-ndp]

Describe the bug

Default timeouts 600000 milisecs don't work in 'chectl server:deploy' command on minikube https://github.com/che-incubator/chectl :

--k8spodwaittimeout [default: 600000]

https://codeready-workspaces-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/Multiuser-Che-PR-check-E2E-Happy-path-tests-against-k8s/4068/console

+ /mnt/hudson_workspace/workspace/Multiuser-Che-PR-check-E2E-Happy-path-tests-against-k8s/chectl server:deploy --k8spodreadytimeout=180000 --listr-renderer=verbose --platform=minikube --che-operator-cr-patch-yaml=/mnt/hudson_workspace/workspace/Multiuser-Che-PR-check-E2E-Happy-path-tests-against-k8s/custom-resource-patch.yaml --chenamespace=eclipse-che
› Current Kubernetes context: 'minikube'
› Installer type is set to: 'operator'
[17:21:26] Verify Kubernetes API [started]
[17:21:26] Verify Kubernetes API...OK [title changed]
[17:21:26] Verify Kubernetes API...OK [completed]
[17:21:26] 👀  Looking for an already existing Eclipse Che instance [started]
[17:21:26] Verify if Eclipse Che is deployed into namespace "eclipse-che" [started]
[17:21:26] Verify if Eclipse Che is deployed into namespace "eclipse-che"...it is not [title changed]
[17:21:26] Verify if Eclipse Che is deployed into namespace "eclipse-che"...it is not [completed]
[17:21:26] 👀  Looking for an already existing Eclipse Che instance [completed]
[17:21:26] ✈️  Minikube preflight checklist [started]
[17:21:26] Verify if kubectl is installed [started]
[17:21:26] Verify if kubectl is installed [completed]
[17:21:26] Verify if minikube is installed [started]
[17:21:26] Verify if minikube is installed [completed]
[17:21:26] Verify if minikube is running [started]
[17:21:26] Verify if minikube is running [completed]
[17:21:26] Start minikube [started]
[17:21:26] Start minikube [skipped]
[17:21:26] → Minikube is already running.
[17:21:26] Check Kubernetes version [started]
[17:21:26] Check Kubernetes version: Found v1.14.3. [title changed]
[17:21:26] Check Kubernetes version: Found v1.14.3. [completed]
[17:21:26] Verify if minikube ingress addon is enabled [started]
[17:21:26] Verify if minikube ingress addon is enabled [completed]
[17:21:26] Enable minikube ingress addon [started]
[17:21:26] Enable minikube ingress addon [skipped]
[17:21:26] → Ingress addon is already enabled.
[17:21:26] Retrieving minikube IP and domain for ingress URLs [started]
[17:21:26] Retrieving minikube IP and domain for ingress URLs...10.0.105.125.nip.io. [title changed]
[17:21:26] Retrieving minikube IP and domain for ingress URLs...10.0.105.125.nip.io. [completed]
[17:21:26] Checking minikube version [started]
[17:21:26] Checking minikube version... 1.1.1 [title changed]
[17:21:26] Checking minikube version... 1.1.1 [completed]
[17:21:26] Check if cluster accessible [started]
[17:21:26] Check if cluster accessible [skipped]
[17:21:26] ✈️  Minikube preflight checklist [completed]
[17:21:26] Following Eclipse Che logs [started]
[17:21:26] Start following Operator logs [started]
[17:21:26] Start following Operator logs...done [title changed]
[17:21:26] Start following Operator logs...done [completed]
[17:21:26] Start following Eclipse Che server logs [started]
[17:21:26] Start following Eclipse Che server logs...done [title changed]
[17:21:26] Start following Eclipse Che server logs...done [completed]
[17:21:26] Start following Postgres logs [started]
[17:21:26] Start following Postgres logs...done [title changed]
[17:21:26] Start following Postgres logs...done [completed]
[17:21:26] Start following Keycloak logs [started]
[17:21:26] Start following Keycloak logs...done [title changed]
[17:21:26] Start following Keycloak logs...done [completed]
[17:21:26] Start following Plugin registry logs [started]
[17:21:26] Start following Plugin registry logs...done [title changed]
[17:21:26] Start following Plugin registry logs...done [completed]
[17:21:26] Start following Devfile registry logs [started]
[17:21:26] Start following Devfile registry logs...done [title changed]
[17:21:26] Start following Devfile registry logs...done [completed]
[17:21:26] Start following namespace events [started]
[17:21:26] Start following namespace events...done [title changed]
[17:21:26] Start following namespace events...done [completed]
[17:21:26] Following Eclipse Che logs [completed]
[17:21:26] 🏃‍  Running the Eclipse Che operator [started]
[17:21:26] Copying operator resources [started]
[17:21:26] Copying operator resources...done. [title changed]
[17:21:26] Copying operator resources...done. [completed]
[17:21:26] Create Namespace (eclipse-che) [started]
[17:21:26] Create Namespace (eclipse-che)...Done. [title changed]
[17:21:26] Create Namespace (eclipse-che)...Done. [completed]
[17:21:26] Create ServiceAccount che-operator in namespace eclipse-che [started]
[17:21:26] Create ServiceAccount che-operator in namespace eclipse-che...done. [title changed]
[17:21:26] Create ServiceAccount che-operator in namespace eclipse-che...done. [completed]
[17:21:26] Create Role che-operator in namespace eclipse-che [started]
[17:21:26] Create Role che-operator in namespace eclipse-che...done. [title changed]
[17:21:26] Create Role che-operator in namespace eclipse-che...done. [completed]
[17:21:26] Create ClusterRole eclipse-che-che-operator [started]
[17:21:26] Create ClusterRole eclipse-che-che-operator...done. [title changed]
[17:21:26] Create ClusterRole eclipse-che-che-operator...done. [completed]
[17:21:26] Create ClusterRole eclipse-che-che-namespace-editor [started]
[17:21:26] Create ClusterRole eclipse-che-che-namespace-editor...done. [title changed]
[17:21:26] Create ClusterRole eclipse-che-che-namespace-editor...done. [completed]
[17:21:26] Create ClusterRoleBinding eclipse-che-che-operator-namespace-editor [started]
[17:21:26] Create ClusterRoleBinding eclipse-che-che-operator-namespace-editor...done. [title changed]
[17:21:26] Create ClusterRoleBinding eclipse-che-che-operator-namespace-editor...done. [completed]
[17:21:26] Create RoleBinding che-operator in namespace eclipse-che [started]
[17:21:26] Create RoleBinding che-operator in namespace eclipse-che...done. [title changed]
[17:21:26] Create RoleBinding che-operator in namespace eclipse-che...done. [completed]
[17:21:26] Create ClusterRoleBinding eclipse-che-che-operator [started]
[17:21:26] Create ClusterRoleBinding eclipse-che-che-operator...done. [title changed]
[17:21:26] Create ClusterRoleBinding eclipse-che-che-operator...done. [completed]
[17:21:26] Create CRD checlusters.org.eclipse.che [started]
[17:21:26] Create CRD checlusters.org.eclipse.che...done. [title changed]
[17:21:26] Create CRD checlusters.org.eclipse.che...done. [completed]
[17:21:26] Waiting 5 seconds for the new Kubernetes resources to get flushed [started]
[17:21:31] Waiting 5 seconds for the new Kubernetes resources to get flushed...done. [title changed]
[17:21:31] Waiting 5 seconds for the new Kubernetes resources to get flushed...done. [completed]
[17:21:31] Create deployment che-operator in namespace eclipse-che [started]
[17:21:32] Create deployment che-operator in namespace eclipse-che...done. [title changed]
[17:21:32] Create deployment che-operator in namespace eclipse-che...done. [completed]
[17:21:32] Operator pod bootstrap [started]
[17:21:32] Scheduling [started]
[17:21:32] Scheduling [failed]
[17:21:32] → Failed to schedule a pod: Timeout:
Pod: che-operator-6cc7b544f4-jnnp5
[17:21:32] Operator pod bootstrap [failed]
[17:21:32] → Failed to schedule a pod: Timeout:
Pod: che-operator-6cc7b544f4-jnnp5
[17:21:32] 🏃‍  Running the Eclipse Che operator [failed]
[17:21:32] → Failed to schedule a pod: Timeout:
Pod: che-operator-6cc7b544f4-jnnp5
 ›   Error: Error: Failed to schedule a pod: Timeout:
 ›   Pod: che-operator-6cc7b544f4-jnnp5
 ›   Command server:deploy failed. Error log: 
 ›   /home/hudson/.cache/chectl/error.log

--k8spoddownloadimagetimeout [default: 600000]

https://codeready-workspaces-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/Multiuser-Che-PR-check-E2E-Happy-path-tests-against-k8s/4070/console

+ /mnt/hudson_workspace/workspace/Multiuser-Che-PR-check-E2E-Happy-path-tests-against-k8s/chectl server:deploy --k8spodwaittimeout=600000 --listr-renderer=verbose --platform=minikube --che-operator-cr-patch-yaml=/mnt/hudson_workspace/workspace/Multiuser-Che-PR-check-E2E-Happy-path-tests-against-k8s/custom-resource-patch.yaml --chenamespace=eclipse-che
› Current Kubernetes context: 'minikube'
› Installer type is set to: 'operator'
[17:57:59] Verify Kubernetes API [started]
[17:57:59] Verify Kubernetes API...OK [title changed]
[17:57:59] Verify Kubernetes API...OK [completed]
[17:57:59] 👀  Looking for an already existing Eclipse Che instance [started]
[17:57:59] Verify if Eclipse Che is deployed into namespace "eclipse-che" [started]
[17:57:59] Verify if Eclipse Che is deployed into namespace "eclipse-che"...it is not [title changed]
[17:57:59] Verify if Eclipse Che is deployed into namespace "eclipse-che"...it is not [completed]
[17:57:59] 👀  Looking for an already existing Eclipse Che instance [completed]
[17:57:59] ✈️  Minikube preflight checklist [started]
[17:57:59] Verify if kubectl is installed [started]
[17:57:59] Verify if kubectl is installed [completed]
[17:57:59] Verify if minikube is installed [started]
[17:57:59] Verify if minikube is installed [completed]
[17:57:59] Verify if minikube is running [started]
[17:57:59] Verify if minikube is running [completed]
[17:57:59] Start minikube [started]
[17:57:59] Start minikube [skipped]
[17:57:59] → Minikube is already running.
[17:57:59] Check Kubernetes version [started]
[17:58:00] Check Kubernetes version: Found v1.14.3. [title changed]
[17:58:00] Check Kubernetes version: Found v1.14.3. [completed]
[17:58:00] Verify if minikube ingress addon is enabled [started]
[17:58:00] Verify if minikube ingress addon is enabled [completed]
[17:58:00] Enable minikube ingress addon [started]
[17:58:00] Enable minikube ingress addon [skipped]
[17:58:00] → Ingress addon is already enabled.
[17:58:00] Retrieving minikube IP and domain for ingress URLs [started]
[17:58:00] Retrieving minikube IP and domain for ingress URLs...10.0.105.182.nip.io. [title changed]
[17:58:00] Retrieving minikube IP and domain for ingress URLs...10.0.105.182.nip.io. [completed]
[17:58:00] Checking minikube version [started]
[17:58:00] Checking minikube version... 1.1.1 [title changed]
[17:58:00] Checking minikube version... 1.1.1 [completed]
[17:58:00] Check if cluster accessible [started]
[17:58:00] Check if cluster accessible [skipped]
[17:58:00] ✈️  Minikube preflight checklist [completed]
[17:58:00] Following Eclipse Che logs [started]
[17:58:00] Start following Operator logs [started]
[17:58:00] Start following Operator logs...done [title changed]
[17:58:00] Start following Operator logs...done [completed]
[17:58:00] Start following Eclipse Che server logs [started]
[17:58:00] Start following Eclipse Che server logs...done [title changed]
[17:58:00] Start following Eclipse Che server logs...done [completed]
[17:58:00] Start following Postgres logs [started]
[17:58:00] Start following Postgres logs...done [title changed]
[17:58:00] Start following Postgres logs...done [completed]
[17:58:00] Start following Keycloak logs [started]
[17:58:00] Start following Keycloak logs...done [title changed]
[17:58:00] Start following Keycloak logs...done [completed]
[17:58:00] Start following Plugin registry logs [started]
[17:58:00] Start following Plugin registry logs...done [title changed]
[17:58:00] Start following Plugin registry logs...done [completed]
[17:58:00] Start following Devfile registry logs [started]
[17:58:00] Start following Devfile registry logs...done [title changed]
[17:58:00] Start following Devfile registry logs...done [completed]
[17:58:00] Start following namespace events [started]
[17:58:00] Start following namespace events...done [title changed]
[17:58:00] Start following namespace events...done [completed]
[17:58:00] Following Eclipse Che logs [completed]
[17:58:00] 🏃‍  Running the Eclipse Che operator [started]
[17:58:00] Copying operator resources [started]
[17:58:00] Copying operator resources...done. [title changed]
[17:58:00] Copying operator resources...done. [completed]
[17:58:00] Create Namespace (eclipse-che) [started]
[17:58:00] Create Namespace (eclipse-che)...Done. [title changed]
[17:58:00] Create Namespace (eclipse-che)...Done. [completed]
[17:58:00] Create ServiceAccount che-operator in namespace eclipse-che [started]
[17:58:00] Create ServiceAccount che-operator in namespace eclipse-che...done. [title changed]
[17:58:00] Create ServiceAccount che-operator in namespace eclipse-che...done. [completed]
[17:58:00] Create Role che-operator in namespace eclipse-che [started]
[17:58:00] Create Role che-operator in namespace eclipse-che...done. [title changed]
[17:58:00] Create Role che-operator in namespace eclipse-che...done. [completed]
[17:58:00] Create ClusterRole eclipse-che-che-operator [started]
[17:58:00] Create ClusterRole eclipse-che-che-operator...done. [title changed]
[17:58:00] Create ClusterRole eclipse-che-che-operator...done. [completed]
[17:58:00] Create ClusterRole eclipse-che-che-namespace-editor [started]
[17:58:00] Create ClusterRole eclipse-che-che-namespace-editor...done. [title changed]
[17:58:00] Create ClusterRole eclipse-che-che-namespace-editor...done. [completed]
[17:58:00] Create ClusterRoleBinding eclipse-che-che-operator-namespace-editor [started]
[17:58:00] Create ClusterRoleBinding eclipse-che-che-operator-namespace-editor...done. [title changed]
[17:58:00] Create ClusterRoleBinding eclipse-che-che-operator-namespace-editor...done. [completed]
[17:58:00] Create RoleBinding che-operator in namespace eclipse-che [started]
[17:58:00] Create RoleBinding che-operator in namespace eclipse-che...done. [title changed]
[17:58:00] Create RoleBinding che-operator in namespace eclipse-che...done. [completed]
[17:58:00] Create ClusterRoleBinding eclipse-che-che-operator [started]
[17:58:00] Create ClusterRoleBinding eclipse-che-che-operator...done. [title changed]
[17:58:00] Create ClusterRoleBinding eclipse-che-che-operator...done. [completed]
[17:58:00] Create CRD checlusters.org.eclipse.che [started]
[17:58:01] Create CRD checlusters.org.eclipse.che...done. [title changed]
[17:58:01] Create CRD checlusters.org.eclipse.che...done. [completed]
[17:58:01] Waiting 5 seconds for the new Kubernetes resources to get flushed [started]
[17:58:06] Waiting 5 seconds for the new Kubernetes resources to get flushed...done. [title changed]
[17:58:06] Waiting 5 seconds for the new Kubernetes resources to get flushed...done. [completed]
[17:58:06] Create deployment che-operator in namespace eclipse-che [started]
[17:58:06] Create deployment che-operator in namespace eclipse-che...done. [title changed]
[17:58:06] Create deployment che-operator in namespace eclipse-che...done. [completed]
[17:58:06] Operator pod bootstrap [started]
[17:58:06] Scheduling [started]
[17:58:06] Scheduling...done [title changed]
[17:58:06] Scheduling...done [completed]
[17:58:06] Downloading images [started]
[17:58:06] Downloading images [failed]
[17:58:06] → Failed to download image: Timeout:
Pod: che-operator-6cc7b544f4-r5xhj
		status: [
  {
    "image": "quay.io/eclipse/che-operator:nightly",
    "imageID": "",
    "lastState": {},
    "name": "che-operator",
    "ready": false,
    "restartCount": 0,
    "state": {
      "waiting": {
        "reason": "ContainerCreating"
      }
    }
  }
]
		conditions: [
  {
    "lastProbeTime": null,
    "lastTransitionTime": "2020-11-25T22:58:06.000Z",
    "status": "True",
    "type": "Initialized"
  },
  {
    "lastProbeTime": null,
    "lastTransitionTime": "2020-11-25T22:58:06.000Z",
    "message": "containers with unready status: [che-operator]",
    "reason": "ContainersNotReady",
    "status": "False",
    "type": "Ready"
  },
  {
    "lastProbeTime": null,
    "lastTransitionTime": "2020-11-25T22:58:06.000Z",
    "message": "containers with unready status: [che-operator]",
    "reason": "ContainersNotReady",
    "status": "False",
    "type": "ContainersReady"
  },
  {
    "lastProbeTime": null,
    "lastTransitionTime": "2020-11-25T22:58:06.000Z",
    "status": "True",
    "type": "PodScheduled"
  }
]
[17:58:06] Operator pod bootstrap [failed]
[17:58:06] → Failed to download image: Timeout:
Pod: che-operator-6cc7b544f4-r5xhj
		status: [
  {
    "image": "quay.io/eclipse/che-operator:nightly",
    "imageID": "",
    "lastState": {},
    "name": "che-operator",
    "ready": false,
    "restartCount": 0,
    "state": {
      "waiting": {
        "reason": "ContainerCreating"
      }
    }
  }
]
		conditions: [
  {
    "lastProbeTime": null,
    "lastTransitionTime": "2020-11-25T22:58:06.000Z",
    "status": "True",
    "type": "Initialized"
  },
  {
    "lastProbeTime": null,
    "lastTransitionTime": "2020-11-25T22:58:06.000Z",
    "message": "containers with unready status: [che-operator]",
    "reason": "ContainersNotReady",
    "status": "False",
    "type": "Ready"
  },
  {
    "lastProbeTime": null,
    "lastTransitionTime": "2020-11-25T22:58:06.000Z",
    "message": "containers with unready status: [che-operator]",
    "reason": "ContainersNotReady",
    "status": "False",
    "type": "ContainersReady"
  },
  {
    "lastProbeTime": null,
    "lastTransitionTime": "2020-11-25T22:58:06.000Z",
    "status": "True",
    "type": "PodScheduled"
  }
]
[17:58:06] 🏃‍  Running the Eclipse Che operator [failed]
[17:58:06] → Failed to download image: Timeout:
Pod: che-operator-6cc7b544f4-r5xhj
		status: [
  {
    "image": "quay.io/eclipse/che-operator:nightly",
    "imageID": "",
    "lastState": {},
    "name": "che-operator",
    "ready": false,
    "restartCount": 0,
    "state": {
      "waiting": {
        "reason": "ContainerCreating"
      }
    }
  }
]
		conditions: [
  {
    "lastProbeTime": null,
    "lastTransitionTime": "2020-11-25T22:58:06.000Z",
    "status": "True",
    "type": "Initialized"
  },
  {
    "lastProbeTime": null,
    "lastTransitionTime": "2020-11-25T22:58:06.000Z",
    "message": "containers with unready status: [che-operator]",
    "reason": "ContainersNotReady",
    "status": "False",
    "type": "Ready"
  },
  {
    "lastProbeTime": null,
    "lastTransitionTime": "2020-11-25T22:58:06.000Z",
    "message": "containers with unready status: [che-operator]",
    "reason": "ContainersNotReady",
    "status": "False",
    "type": "ContainersReady"
  },
  {
    "lastProbeTime": null,
    "lastTransitionTime": "2020-11-25T22:58:06.000Z",
    "status": "True",
    "type": "PodScheduled"
  }
]
 ›   Error: Error: Failed to download image: Timeout:
 ›   Pod: che-operator-6cc7b544f4-r5xhj
 ›   		status: [
 ›     {
 ›       "image": "quay.io/eclipse/che-operator:nightly",
 ›       "imageID": "",
 ›       "lastState": {},
 ›       "name": "che-operator",
 ›       "ready": false,
 ›       "restartCount": 0,
 ›       "state": {
 ›         "waiting": {
 ›           "reason": "ContainerCreating"
 ›         }
 ›       }
 ›     }
 ›   ]
 ›   		conditions: [
 ›     {
 ›       "lastProbeTime": null,
 ›       "lastTransitionTime": "2020-11-25T22:58:06.000Z",
 ›       "status": "True",
 ›       "type": "Initialized"
 ›     },
 ›     {
 ›       "lastProbeTime": null,
 ›       "lastTransitionTime": "2020-11-25T22:58:06.000Z",
 ›       "message": "containers with unready status: [che-operator]",
 ›       "reason": "ContainersNotReady",
 ›       "status": "False",
 ›       "type": "Ready"
 ›     },
 ›     {
 ›       "lastProbeTime": null,
 ›       "lastTransitionTime": "2020-11-25T22:58:06.000Z",
 ›       "message": "containers with unready status: [che-operator]",
 ›       "reason": "ContainersNotReady",
 ›       "status": "False",
 ›       "type": "ContainersReady"
 ›     },
 ›     {
 ›       "lastProbeTime": null,
 ›       "lastTransitionTime": "2020-11-25T22:58:06.000Z",
 ›       "status": "True",
 ›       "type": "PodScheduled"
 ›     }
 ›   ]
 ›   Command server:deploy failed. Error log: 
 ›   /home/hudson/.cache/chectl/error.log

Che version

• latest
• nightly
• other: please specify

Steps to reproduce

Expected behavior

Runtime

• kubernetes (include output of kubectl version)
• Openshift (include output of oc version)
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: (please specify)

Screenshots

Installation method

• chectl
  • provide a full command that was used to deploy Eclipse Che (including the output)
  • provide an output of chectl version command
• OperatorHub
• I don't know

Environment

• my computer
  • Windows
  • Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• other: CRW CCI

Eclipse Che Logs

Additional context

Related commit: che-incubator/chectl@a6d48b1