Feature Request: Exchange arbitrary metadata between sidecars at connection time

[mandarjog]

Title: Exchange arbitrary metadata during TLS handshake or some other process

Description:
In order for Envoy to produce rich telemetry about details of source and destination workloads, we need a mechanism to exchange this information.

As an example, Istio telemetry produces highly dimensioned metrics that identify source and destination of the traffic. The metrics are dimensioned using source and destination metadata such as NodeID and labels associated with the pods or VMs.

For HTTP this can be achieved easily by forwarding the node metadata to destination in a well-known header, and the destination responding with its own header.
Even though the above solution works, it needlessly sends the same information over and over with every request.

For TCP this does not work, and we have to rely on ip address to metadata mapping at the destination and source.

Options:

1. When mTLS is used, it gives us a way exchange this metadata up front.
   If there is a way to add metadata during TLS handshake, it works.
2. Use HaProxy protocol to exchange this information. Not sure if it works both ways.
3. https://www.akamai.com/us/en/multimedia/documents/technical-publication/tls-aux-associating-auxiliary-data-tls-connections.pdf

[mandarjog]

@PiotrSikora

[yangminzhu]

/cc @yangminzhu

[incfly]

/cc @incfly

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.

[gargnupur]

Exploring if TCP over HTTP/2 metadata frames can be used to implement this...

We would need both these issues fixed for that afaik:

1. Proxy TCP connections over HTTP2 connection pools #1630
2. Support HTTP/2 METADATA frames #2394

[gargnupur]

/cc @alyssawilk

To use TCP over HTTP/2 metadata frames, looks like support for TCP over HTTP(#1630) is a blocker right now.

We are going to explore TLS extension path for which #173 is currently a blocker..

Ref: @kyessenov has a great doc on metadata exchange https://docs.google.com/document/d/1bWQAsrBZguk5HCmBVDEgEMVGS91r9uh3SIr7D7ELZBk/edit.

[mandarjog]

A required PR to enable "network filters for upstream" #7503

[gargnupur]

@mandarjog : I think this can be closed now...