Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PROXY protocol does not work with 1.0.0 and above when Ambassador is terminating tls #2348

Closed
iNoahNothing opened this issue Feb 24, 2020 · 0 comments
Closed

PROXY protocol does not work with 1.0.0 and above when Ambassador is terminating tls #2348

iNoahNothing opened this issue Feb 24, 2020 · 0 comments
Assignees
@kflynn
Labels
p:high Issue is of high importance t:bug Something isn't working
Milestone
2020 Cycle 1 Cool...

Comments

@iNoahNothing
Copy link
Contributor

Ambassador 1.0.0+ does not appear to properly handle the proxy protocol when terminating tls.

To reproduce:

  1. Deploy aes

  2. set use_proxy_proto: true in a Module

  3. Exec into the ambassador container and send a curl to an endpoint with --haproxy-protocol flag (this is easier than needing to configure a load balancer that enables the proxy protocol)

  4. Get an SSL error:

$ curl -vk https://localhost:8443/ambassador/v0/diag/
*   Trying 127.0.0.1:8443...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:8443 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:8443

A look at the envoy.json seems correct:

{
    "@type": "/envoy.config.bootstrap.v2.Bootstrap",
    "layered_runtime": {
        "layers": [
            {
                "name": "static_layer",
                "static_layer": {
                    "envoy.deprecated_features:envoy.api.v2.route.HeaderMatcher.regex_match": true,
                    "envoy.deprecated_features:envoy.api.v2.route.RouteMatch.regex": true,
                    "envoy.deprecated_features:envoy.config.filter.http.ext_authz.v2.ExtAuthz.use_alpha": true,
                    "envoy.deprecated_features:envoy.config.trace.v2.ZipkinConfig.HTTP_JSON_V1": true
                }
            }
        ]
    },
    "static_resources": {
        "clusters": [
            {
                "connect_timeout": "3.000s",
                "dns_lookup_family": "V4_ONLY",
                "http2_protocol_options": {},
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_127_0_0_1_8500_ambassador",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "127.0.0.1",
                                                "port_value": 8500,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_127_0_0_1_8500_ambassador",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3.000s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_127_0_0_1_8877_ambassador",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "127.0.0.1",
                                                "port_value": 8877,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_127_0_0_1_8877_ambassador",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3.000s",
                "dns_lookup_family": "V4_ONLY",
                "http2_protocol_options": {},
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_extauth_127_0_0_1_8500_ambassador",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "127.0.0.1",
                                                "port_value": 8500,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_extauth_127_0_0_1_8500_ambassador",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3.000s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_quote_default",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "quote.default",
                                                "port_value": 80,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_quote_default",
                "type": "STRICT_DNS"
            }
        ],
        "listeners": [
            {
                "address": {
                    "socket_address": {
                        "address": "0.0.0.0",
                        "port_value": 8443,
                        "protocol": "TCP"
                    }
                },
                "filter_chains": [
                    {
                        "filter_chain_match": {
                            "transport_protocol": "tls"
                        },
                        "filters": [
                            {
                                "name": "envoy.http_connection_manager",
                                "typed_config": {
                                    "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
                                    "access_log": [
                                        {
                                            "name": "envoy.file_access_log",
                                            "typed_config": {
                                                "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
                                                "format": "ACCESS [%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\"\n",
                                                "path": "/dev/fd/1"
                                            }
                                        }
                                    ],
                                    "http_filters": [
                                        {
                                            "config": {
                                                "grpc_service": {
                                                    "envoy_grpc": {
                                                        "cluster_name": "cluster_extauth_127_0_0_1_8500_ambassador"
                                                    },
                                                    "timeout": "5.000s"
                                                },
                                                "status_on_error": {
                                                    "code": 504
                                                },
                                                "use_alpha": true
                                            },
                                            "name": "envoy.ext_authz"
                                        },
                                        {
                                            "name": "envoy.cors"
                                        },
                                        {
                                            "config": {
                                                "domain": "ambassador",
                                                "rate_limit_service": {
                                                    "grpc_service": {
                                                        "envoy_grpc": {
                                                            "cluster_name": "cluster_127_0_0_1_8500_ambassador"
                                                        }
                                                    }
                                                },
                                                "request_type": "both",
                                                "timeout": "0.020s"
                                            },
                                            "name": "envoy.rate_limit"
                                        },
                                        {
                                            "name": "envoy.router"
                                        }
                                    ],
                                    "http_protocol_options": {
                                        "accept_http_10": false
                                    },
                                    "normalize_path": true,
                                    "route_config": {
                                        "virtual_hosts": [
                                            {
                                                "domains": [
                                                    "*"
                                                ],
                                                "name": "ambassador-listener-8443-external.nkrause.k736.net-forced-star",
                                                "routes": [
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/.well-known/acme-challenge/"
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_extauth_127_0_0_1_8500_ambassador",
                                                            "prefix_rewrite": "/.well-known/acme-challenge/",
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "headers": [
                                                                {
                                                                    "exact_match": "https",
                                                                    "name": "x-forwarded-proto"
                                                                }
                                                            ],
                                                            "prefix": "/.ambassador/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500_ambassador",
                                                            "priority": null,
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/.ambassador/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500_ambassador",
                                                            "priority": null,
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "headers": [
                                                                {
                                                                    "exact_match": "https",
                                                                    "name": "x-forwarded-proto"
                                                                }
                                                            ],
                                                            "prefix": "/edge_stack/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500_ambassador",
                                                            "prefix_rewrite": "/edge_stack_ui/edge_stack/",
                                                            "priority": null,
                                                            "timeout": "60.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/edge_stack/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500_ambassador",
                                                            "prefix_rewrite": "/edge_stack_ui/edge_stack/",
                                                            "priority": null,
                                                            "timeout": "60.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "headers": [
                                                                {
                                                                    "exact_match": "https",
                                                                    "name": "x-forwarded-proto"
                                                                }
                                                            ],
                                                            "prefix": "/ambassador/v0/check_ready",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8877_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8877_ambassador",
                                                            "prefix_rewrite": "/ambassador/v0/check_ready",
                                                            "priority": null,
                                                            "timeout": "10.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/ambassador/v0/check_ready",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8877_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8877_ambassador",
                                                            "prefix_rewrite": "/ambassador/v0/check_ready",
                                                            "priority": null,
                                                            "timeout": "10.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "headers": [
                                                                {
                                                                    "exact_match": "https",
                                                                    "name": "x-forwarded-proto"
                                                                }
                                                            ],
                                                            "prefix": "/ambassador/v0/check_alive",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8877_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8877_ambassador",
                                                            "prefix_rewrite": "/ambassador/v0/check_alive",
                                                            "priority": null,
                                                            "timeout": "10.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/ambassador/v0/check_alive",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8877_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8877_ambassador",
                                                            "prefix_rewrite": "/ambassador/v0/check_alive",
                                                            "priority": null,
                                                            "timeout": "10.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "headers": [
                                                                {
                                                                    "exact_match": "https",
                                                                    "name": "x-forwarded-proto"
                                                                }
                                                            ],
                                                            "prefix": "/documentation/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500_ambassador",
                                                            "prefix_rewrite": "/docs/",
                                                            "priority": null,
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/documentation/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500_ambassador",
                                                            "prefix_rewrite": "/docs/",
                                                            "priority": null,
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "headers": [
                                                                {
                                                                    "exact_match": "https",
                                                                    "name": "x-forwarded-proto"
                                                                }
                                                            ],
                                                            "prefix": "/ambassador/v0/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8877_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8877_ambassador",
                                                            "prefix_rewrite": "/ambassador/v0/",
                                                            "priority": null,
                                                            "timeout": "10.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/ambassador/v0/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8877_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8877_ambassador",
                                                            "prefix_rewrite": "/ambassador/v0/",
                                                            "priority": null,
                                                            "timeout": "10.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "headers": [
                                                                {
                                                                    "exact_match": "https",
                                                                    "name": "x-forwarded-proto"
                                                                }
                                                            ],
                                                            "prefix": "/openapi/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500_ambassador",
                                                            "priority": null,
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/openapi/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500_ambassador",
                                                            "priority": null,
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "headers": [
                                                                {
                                                                    "exact_match": "https",
                                                                    "name": "x-forwarded-proto"
                                                                }
                                                            ],
                                                            "prefix": "/backend/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_quote_default"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_quote_default",
                                                            "prefix_rewrite": "/",
                                                            "priority": null,
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/backend/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_quote_default"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_quote_default",
                                                            "prefix_rewrite": "/",
                                                            "priority": null,
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "headers": [
                                                                {
                                                                    "exact_match": "https",
                                                                    "name": "x-forwarded-proto"
                                                                }
                                                            ],
                                                            "prefix": "/docs/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500_ambassador",
                                                            "prefix_rewrite": "/docs/",
                                                            "priority": null,
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/docs/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500_ambassador",
                                                            "prefix_rewrite": "/docs/",
                                                            "priority": null,
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500_ambassador",
                                                            "prefix_rewrite": "/edge_stack_ui/",
                                                            "priority": null,
                                                            "timeout": "60.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500_ambassador",
                                                            "prefix_rewrite": "/edge_stack_ui/",
                                                            "priority": null,
                                                            "timeout": "60.000s"
                                                        }
                                                    }
                                                ]
                                            }
                                        ]
                                    },
                                    "server_name": "envoy",
                                    "stat_prefix": "ingress_http",
                                    "use_remote_address": true,
                                    "xff_num_trusted_hops": 0
                                }
                            }
                        ],
                        "tls_context": {
                            "common_tls_context": {
                                "tls_certificates": [
                                    {
                                        "certificate_chain": {
                                            "filename": "/ambassador/snapshots/ambassador/secrets-decoded/external.nkrause.k736.net/765818A6764A5FE672D1A2B340952A354478D3BF.crt"
                                        },
                                        "private_key": {
                                            "filename": "/ambassador/snapshots/ambassador/secrets-decoded/external.nkrause.k736.net/765818A6764A5FE672D1A2B340952A354478D3BF.key"
                                        }
                                    }
                                ]
                            }
                        },
                        "use_proxy_proto": true
                    }
                ],
                "listener_filters": [
                    {
                        "config": {},
                        "name": "envoy.listener.tls_inspector"
                    }
                ],
                "name": "ambassador-listener-8443",
                "traffic_direction": "UNSPECIFIED"
            },
            {
                "address": {
                    "socket_address": {
                        "address": "0.0.0.0",
                        "port_value": 8080,
                        "protocol": "TCP"
                    }
                },
                "filter_chains": [
                    {
                        "filter_chain_match": {},
                        "filters": [
                            {
                                "name": "envoy.http_connection_manager",
                                "typed_config": {
                                    "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
                                    "access_log": [
                                        {
                                            "name": "envoy.file_access_log",
                                            "typed_config": {
                                                "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
                                                "format": "ACCESS [%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\"\n",
                                                "path": "/dev/fd/1"
                                            }
                                        }
                                    ],
                                    "http_filters": [
                                        {
                                            "config": {
                                                "grpc_service": {
                                                    "envoy_grpc": {
                                                        "cluster_name": "cluster_extauth_127_0_0_1_8500_ambassador"
                                                    },
                                                    "timeout": "5.000s"
                                                },
                                                "status_on_error": {
                                                    "code": 504
                                                },
                                                "use_alpha": true
                                            },
                                            "name": "envoy.ext_authz"
                                        },
                                        {
                                            "name": "envoy.cors"
                                        },
                                        {
                                            "config": {
                                                "domain": "ambassador",
                                                "rate_limit_service": {
                                                    "grpc_service": {
                                                        "envoy_grpc": {
                                                            "cluster_name": "cluster_127_0_0_1_8500_ambassador"
                                                        }
                                                    }
                                                },
                                                "request_type": "both",
                                                "timeout": "0.020s"
                                            },
                                            "name": "envoy.rate_limit"
                                        },
                                        {
                                            "name": "envoy.router"
                                        }
                                    ],
                                    "http_protocol_options": {
                                        "accept_http_10": false
                                    },
                                    "normalize_path": true,
                                    "route_config": {
                                        "virtual_hosts": [
                                            {
                                                "domains": [
                                                    "*"
                                                ],
                                                "name": "ambassador-listener-8080-external.nkrause.k736.net-forced-star",
                                                "routes": [
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/.well-known/acme-challenge/"
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_extauth_127_0_0_1_8500_ambassador",
                                                            "prefix_rewrite": "/.well-known/acme-challenge/",
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/.ambassador/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500_ambassador",
                                                            "priority": null,
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/edge_stack/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500_ambassador",
                                                            "prefix_rewrite": "/edge_stack_ui/edge_stack/",
                                                            "priority": null,
                                                            "timeout": "60.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/ambassador/v0/check_ready",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8877_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8877_ambassador",
                                                            "prefix_rewrite": "/ambassador/v0/check_ready",
                                                            "priority": null,
                                                            "timeout": "10.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/ambassador/v0/check_alive",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8877_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8877_ambassador",
                                                            "prefix_rewrite": "/ambassador/v0/check_alive",
                                                            "priority": null,
                                                            "timeout": "10.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/documentation/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500_ambassador",
                                                            "prefix_rewrite": "/docs/",
                                                            "priority": null,
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/ambassador/v0/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8877_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8877_ambassador",
                                                            "prefix_rewrite": "/ambassador/v0/",
                                                            "priority": null,
                                                            "timeout": "10.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/openapi/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500_ambassador",
                                                            "priority": null,
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/backend/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_quote_default"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_quote_default",
                                                            "prefix_rewrite": "/",
                                                            "priority": null,
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/docs/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500_ambassador",
                                                            "prefix_rewrite": "/docs/",
                                                            "priority": null,
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500_ambassador"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500_ambassador",
                                                            "prefix_rewrite": "/edge_stack_ui/",
                                                            "priority": null,
                                                            "timeout": "60.000s"
                                                        }
                                                    }
                                                ]
                                            }
                                        ]
                                    },
                                    "server_name": "envoy",
                                    "stat_prefix": "ingress_http",
                                    "use_remote_address": true,
                                    "xff_num_trusted_hops": 0
                                }
                            }
                        ],
                        "use_proxy_proto": true
                    }
                ],
                "listener_filters": [],
                "name": "ambassador-listener-8080",
                "traffic_direction": "UNSPECIFIED"
            }
        ]
    }
}

This behavior is not in 86.1 and lower:

envoy.json

{
    "@type": "/envoy.config.bootstrap.v2.Bootstrap",
    "static_resources": {
        "clusters": [
            {
                "connect_timeout": "3.000s",
                "dns_lookup_family": "V4_ONLY",
                "http2_protocol_options": {},
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_127_0_0_1_8500",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "127.0.0.1",
                                                "port_value": 8500,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_127_0_0_1_8500",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3.000s",
                "dns_lookup_family": "V4_ONLY",
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_127_0_0_1_8877",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "127.0.0.1",
                                                "port_value": 8877,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_127_0_0_1_8877",
                "type": "STRICT_DNS"
            },
            {
                "connect_timeout": "3.000s",
                "dns_lookup_family": "V4_ONLY",
                "http2_protocol_options": {},
                "lb_policy": "ROUND_ROBIN",
                "load_assignment": {
                    "cluster_name": "cluster_extauth_127_0_0_1_8500",
                    "endpoints": [
                        {
                            "lb_endpoints": [
                                {
                                    "endpoint": {
                                        "address": {
                                            "socket_address": {
                                                "address": "127.0.0.1",
                                                "port_value": 8500,
                                                "protocol": "TCP"
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "name": "cluster_extauth_127_0_0_1_8500",
                "type": "STRICT_DNS"
            }
        ],
        "listeners": [
            {
                "address": {
                    "socket_address": {
                        "address": "0.0.0.0",
                        "port_value": 8443,
                        "protocol": "TCP"
                    }
                },
                "filter_chains": [
                    {
                        "filters": [
                            {
                                "config": {
                                    "access_log": [
                                        {
                                            "config": {
                                                "format": "ACCESS [%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\"\n\n",
                                                "path": "/dev/fd/1"
                                            },
                                            "name": "envoy.file_access_log"
                                        }
                                    ],
                                    "http_filters": [
                                        {
                                            "config": {
                                                "grpc_service": {
                                                    "envoy_grpc": {
                                                        "cluster_name": "cluster_extauth_127_0_0_1_8500"
                                                    },
                                                    "timeout": "5.000s"
                                                },
                                                "status_on_error": {
                                                    "code": 504
                                                },
                                                "use_alpha": true
                                            },
                                            "name": "envoy.ext_authz"
                                        },
                                        {
                                            "config": {
                                                "domain": "ambassador",
                                                "rate_limit_service": {
                                                    "grpc_service": {
                                                        "envoy_grpc": {
                                                            "cluster_name": "cluster_127_0_0_1_8500"
                                                        }
                                                    }
                                                },
                                                "request_type": "both",
                                                "timeout": "0.020s"
                                            },
                                            "name": "envoy.rate_limit"
                                        },
                                        {
                                            "name": "envoy.cors"
                                        },
                                        {
                                            "name": "envoy.router"
                                        }
                                    ],
                                    "http_protocol_options": {
                                        "accept_http_10": false
                                    },
                                    "normalize_path": true,
                                    "route_config": {
                                        "virtual_hosts": [
                                            {
                                                "domains": [
                                                    "*"
                                                ],
                                                "name": "backend",
                                                "routes": [
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/ambassador/v0/check_ready",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8877"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8877",
                                                            "prefix_rewrite": "/ambassador/v0/check_ready",
                                                            "priority": null,
                                                            "timeout": "10.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/ambassador/v0/check_alive",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8877"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8877",
                                                            "prefix_rewrite": "/ambassador/v0/check_alive",
                                                            "priority": null,
                                                            "timeout": "10.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/documentation/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500",
                                                            "prefix_rewrite": "/docs/",
                                                            "priority": null,
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/ambassador/v0/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8877"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8877",
                                                            "prefix_rewrite": "/ambassador/v0/",
                                                            "priority": null,
                                                            "timeout": "10.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/openapi/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500",
                                                            "priority": null,
                                                            "timeout": "3.000s"
                                                        }
                                                    },
                                                    {
                                                        "match": {
                                                            "case_sensitive": true,
                                                            "prefix": "/docs/",
                                                            "runtime_fraction": {
                                                                "default_value": {
                                                                    "denominator": "HUNDRED",
                                                                    "numerator": 100
                                                                },
                                                                "runtime_key": "routing.traffic_shift.cluster_127_0_0_1_8500"
                                                            }
                                                        },
                                                        "route": {
                                                            "cluster": "cluster_127_0_0_1_8500",
                                                            "prefix_rewrite": "/docs/",
                                                            "priority": null,
                                                            "timeout": "3.000s"
                                                        }
                                                    }
                                                ]
                                            }
                                        ]
                                    },
                                    "server_name": "envoy",
                                    "stat_prefix": "ingress_http",
                                    "use_remote_address": true,
                                    "xff_num_trusted_hops": 0
                                },
                                "name": "envoy.http_connection_manager"
                            }
                        ],
                        "tls_context": {
                            "common_tls_context": {
                                "tls_certificates": [
                                    {
                                        "certificate_chain": {
                                            "filename": "/ambassador/snapshots/ambassador/secrets-decoded/self-signed/7C702B0E3ED30C96F41303B6EDC11B7F3F7B19BA.crt"
                                        },
                                        "private_key": {
                                            "filename": "/ambassador/snapshots/ambassador/secrets-decoded/self-signed/7C702B0E3ED30C96F41303B6EDC11B7F3F7B19BA.key"
                                        }
                                    }
                                ]
                            }
                        },
                        "use_proxy_proto": true
                    }
                ],
                "name": "ambassador-listener-8443"
            }
        ]
    }
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kflynn kflynn

Labels
p:high Issue is of high importance t:bug Something isn't working
Projects
None yet
Milestone
2020 Cycle 1 Cool-down
Development

Successfully merging a pull request may close this issue.

Switch to Envoy proxy_protocol filter instead of using use_proxy_proto. emissary-ingress/emissary
3 participants
@khussey @kflynn @iNoahNothing