Replace sidecar nginx proxy with envoy / Istio

[vaikas]

We currently inject an nginx proxy. In order to get more comformance with ingress routing that we use Istio for as well as dealing with only one proxy, we should replace the sidecar proxy with Istio/Envoy.
https://github.com/IBM/microservices-traffic-management-using-istio/blob/master/README.md#2-inject-istio-envoys-on-the-application

We should do this in our revision/controller.go and replace this with envoy injection. But above link should give enough context to figure out what the injection should parameters look:
https://github.com/google/elafros/blob/master/pkg/controller/revision/controller.go#L591

[josephburnett]

This is what Istio Envoy injection looks like: https://gist.github.com/josephburnett/a3c7eddb84ce9554c472abdbd50db6e5
That's a LOT of stuff to hardcode in our pod spec.

Alternatively we can setup a webhook to inject Envoy into all pods in the default-ela namespace: https://istio.io/docs/setup/kubernetes/sidecar-injection.html#installing-the-webhook

@vaikas-google which direction should we go?

[josephburnett]

I would prefer to use the Istio code to inject the Envoy sidecar in order to stay in sync with their development. That mean either using their webhook, or doing it by API (feature request: istio/old_issues_repo#148).

[grantr]

Another advantage of replacing nginx with Envoy is that Envoy can route to HTTP/2 and gRPC backends.

[kelseyhightower]

The new istio 0.5.0 release make it easy to leverage the sidecar injection using a mutating webhook. I'm currently doing this in the riff tutorial where the isito sidecar is being injected into every function created by the riff function controller.

[kelseyhightower]

I have tested the istio sidecar injection with elafros and there was only one major hiccup:

• The Fluentd container in the elafros pod was not able to obtain the default credentials from the instance metadata service because the Istio (envoy) sidecar does not permit access to IPs outside the service mesh by default.

Ideally we should revisit the need to install Fluentd at all since it's already installed in most Kubernetes clusters, including GKE. If elafros does need its own copy of Fluentd, then we should consider using an explicit IAM service account backed by a Kubernetes secret that is attached to the pod at runtime.

[mattmoor]

No access to http://metadata seems like a problem without a baked simple alternative for application identity.

[evankanderson]

re @mattmoor -- we should publish a service account certificate to the container at a well-known address (i.e. as part of the container contract) in the short term, and coordinate with the per-pod identity efforts in k8s in the medium term.

Some of them sit nearby the Googlers in Seattle; I can arrange an introduction if there's someone ready to tackle the identity work.

[mattmoor]

I believe sidecar injection is done now, so can we close this?

[mdemirhan]

Yes, this is now complete. Closing the issue.