[bug] Kubernetes Ingress check fails for AWS

[iameskild]

0.4.0 test tracker: #976
During the 0.4.0 testing, the AWS deployment failed at the end of the 04-kubernetes-ingress step while trying to _attempt_tcp_connect even though the "hostname" has been created and is available:

[terraform]: load_balancer_address = {
[terraform]:   "hostname" = "a179cf11016ba4403b6f914bb1336a19-672589103.us-west-1.elb.amazonaws.com"
[terraform]:   "ip" = ""
[terraform]: }
INFO:qhub.provider.terraform:terraform apply took 38.957 [s]
INFO:qhub.provider.terraform:terraform=/var/folders/ny/_kl83bcn59q8dngjrh_4qhjr0000gn/T/terraform/1.0.5/terraform output directory=stages/04-kubernetes-ingress
INFO:qhub.provider.terraform:terraform output took 3.160 [s]
Traceback (most recent call last):
  File "/Users/eskild/miniconda3/envs/qhub-dev/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/eskild/miniconda3/envs/qhub-dev/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/eskild/miniconda3/envs/qhub-dev/lib/python3.8/site-packages/qhub/__main__.py", line 11, in <module>
    main()
  File "/Users/eskild/miniconda3/envs/qhub-dev/lib/python3.8/site-packages/qhub/__main__.py", line 7, in main
    cli(sys.argv[1:])
  File "/Users/eskild/miniconda3/envs/qhub-dev/lib/python3.8/site-packages/qhub/cli/__init__.py", line 52, in cli
    args.func(args)
  File "/Users/eskild/miniconda3/envs/qhub-dev/lib/python3.8/site-packages/qhub/cli/deploy.py", line 65, in handle_deploy
    deploy_configuration(
  File "/Users/eskild/miniconda3/envs/qhub-dev/lib/python3.8/site-packages/qhub/deploy.py", line 36, in deploy_configuration
    guided_install(
  File "/Users/eskild/miniconda3/envs/qhub-dev/lib/python3.8/site-packages/qhub/deploy.py", line 814, in guided_install
    provision_04_kubernetes_ingress(stage_outputs, config)
  File "/Users/eskild/miniconda3/envs/qhub-dev/lib/python3.8/site-packages/qhub/deploy.py", line 409, in provision_04_kubernetes_ingress
    check_04_kubernetes_ingress(stage_outputs, config)
  File "/Users/eskild/miniconda3/envs/qhub-dev/lib/python3.8/site-packages/qhub/deploy.py", line 443, in check_04_kubernetes_ingress
    if not _attempt_tcp_connect(host, port):
  File "/Users/eskild/miniconda3/envs/qhub-dev/lib/python3.8/site-packages/qhub/deploy.py", line 417, in _attempt_tcp_connect
    host = socket.gethostbyname(host)
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

We ran the deployment again and this time the TCP connection tests were successful, leading us to believe the output json is not available at the time of the check.

A possible solution might be to move the check_04_kubernetes_ingress out of the provision_04_kubernetes_ingress step and call it separately before step 05.

[iameskild]

Fix can be found here:
https://github.com/Quansight/qhub/blob/b319bd356bf703b47a272fcba49380113c0185f6/qhub/deploy.py#L416-L433

[iameskild]

However, during the same stage, checking that the DNS record was created also results in a failure:

INFO:qhub.provider.dns.cloudflare:record name=eaettt type=CNAME address=a74f54b37fe2a4726b01a3e46e78f20d-936064546.us-west-1.elb.amazonaws.com does not exists creating
Attempt 1 polling DNS domain=eaettt.qhub.dev does not match ip=54.153.71.23 instead got 23.202.231.169
Attempt 2 polling DNS domain=eaettt.qhub.dev does not match ip=54.153.71.23 instead got 23.202.231.169
Attempt 3 polling DNS domain=eaettt.qhub.dev does not match ip=54.153.71.23 instead got 23.202.231.169
Attempt 4 polling DNS domain=eaettt.qhub.dev does not match ip=54.153.71.23 instead got 23.202.231.169
Attempt 5 polling DNS domain=eaettt.qhub.dev does not match ip=54.153.71.23 instead got 23.202.231.169
Attempt 6 polling DNS domain=eaettt.qhub.dev does not match ip=54.153.71.23 instead got 23.202.231.169
Attempt 7 polling DNS domain=eaettt.qhub.dev does not match ip=54.153.71.23 instead got 23.202.231.169
Attempt 8 polling DNS domain=eaettt.qhub.dev does not match ip=54.153.71.23 instead got 23.202.231.169
Attempt 9 polling DNS domain=eaettt.qhub.dev does not match ip=54.153.71.23 instead got 23.202.231.169
Attempt 10 polling DNS domain=eaettt.qhub.dev does not match ip=54.153.71.23 instead got 23.202.231.169
Attempt 11 polling DNS domain=eaettt.qhub.dev does not match ip=54.153.71.23 instead got 23.202.231.169
Attempt 12 polling DNS domain=eaettt.qhub.dev does not match ip=54.153.71.23 instead got 23.202.231.169
ERROR: After stage directory=stages/04-kubernetes-ingress DNS domain=eaettt.qhub.dev does not point to ip=54.153.71.23

[costrouc]

@iameskild is this still an error? This does look like an issue with the dns A record changing.