Istio MTLS certs not working in RC2

[n1koo]

Describe the bug
After upgrading to -rc2 from previous -ea6 istio upstream tls certs are causing exits

2018-12-31 12:08:49 kubewatch [21 TMainThread] 0.50.0 INFO: resolved: {'cert_chain_file': '/etc/istiocerts/cert-chain.pem', 'private_key_file': '/etc/istiocerts/key.pem'}
2018-12-31 12:08:49 kubewatch [21 TMainThread] 0.50.0 WARNING: kubewatch failed!
Traceback (most recent call last):
  File "/ambassador/kubewatch.py", line 520, in <module>
    main()
  File "/usr/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/ambassador/kubewatch.py", line 513, in main
    watcher.run(sync_only=True)
  File "/ambassador/kubewatch.py", line 356, in run
    self.restarter.restart()
  File "/ambassador/kubewatch.py", line 170, in restart
    bootstrap_config, ads_config = self.generate_config(changes, output)
  File "/ambassador/kubewatch.py", line 224, in generate_config
    ir = IR(aconf, tls_secret_resolver=kube_tls_secret_resolver)
  File "/usr/lib/python3.6/site-packages/ambassador-0.0.0.dev0-py3.6.egg/ambassador/ir/ir.py", line 174, in __init__
  File "/usr/lib/python3.6/site-packages/ambassador-0.0.0.dev0-py3.6.egg/ambassador/ir/irmapping.py", line 631, in load_all
  File "/usr/lib/python3.6/site-packages/ambassador-0.0.0.dev0-py3.6.egg/ambassador/ir/irmapping.py", line 120, in __init__
  File "/usr/lib/python3.6/site-packages/ambassador-0.0.0.dev0-py3.6.egg/ambassador/ir/irmapping.py", line 243, in match_tls_context

Seems to come from https://github.com/datawire/ambassador/blob/release/0.50.0/ambassador/ambassador/ir/irmapping.py#L241-L248 checking the hosts.

This changed in #1066 but haven't spotted yet what actually caused it.

To Reproduce
Steps to reproduce the behavior:

1. Configure Istio MTLS as described in https://www.getambassador.io/user-guide/with-istio/
2. Pods crash on the exception above

Specifically:

        upstream:
          cert_chain_file: /etc/istiocerts/cert-chain.pem
          private_key_file: /etc/istiocerts/key.pem

on the servicie

Expected behavior
A clear and concise description of what you expected to happen.

Versions (please complete the following information):

• Ambassador: 0.50.0-rc2
• AWS
• Kubernetes 1.12.3

[christianhuening]

Is this specific to istio certs? I am intending to use Linkerd2 with Ambassador

[n1koo]

FWIW still relevant in RC4 cc @richarddli

2019-01-10 07:20:44 kubewatch [22 TMainThread] 0.50.0 DEBUG: 10-second watch loop delay
Traceback (most recent call last):
  File "/ambassador/kubewatch.py", line 521, in <module>
    main()
  File "/usr/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/ambassador/kubewatch.py", line 514, in main
    watcher.run(sync_only=True)
  File "/ambassador/kubewatch.py", line 353, in run
    self.restarter.restart()
  File "/ambassador/kubewatch.py", line 170, in restart
    bootstrap_config, ads_config = self.generate_config(changes, output)
  File "/ambassador/kubewatch.py", line 224, in generate_config
    ir = IR(aconf)
  File "/usr/lib/python3.6/site-packages/ambassador-0.0.0.dev0-py3.6.egg/ambassador/ir/ir.py", line 168, in __init__
  File "/usr/lib/python3.6/site-packages/ambassador-0.0.0.dev0-py3.6.egg/ambassador/ir/irmapping.py", line 647, in load_all
  File "/usr/lib/python3.6/site-packages/ambassador-0.0.0.dev0-py3.6.egg/ambassador/ir/irmapping.py", line 121, in __init__
  File "/usr/lib/python3.6/site-packages/ambassador-0.0.0.dev0-py3.6.egg/ambassador/ir/irmapping.py", line 244, in match_tls_context

[johnb-mty]

I'm also seeing this in ambassador:0.50.0-rc3 running on Kubernetes 1.11.5-eks with the same istio configurations