Allow /healthcheck/fail from not admin webpage

[stevesloka]

Title: Allow the result of calling /healthcheck/fail but not from the Admin Webpage

Description:
I'd like a way to tell Envoy to begin draining connections and start shutting down, but not require the admin page to do this. Currently we have a related CVE in Contour (GHSA-5ph6-qq5x-7jwc) due to this since you can create a Kubernetes Service type=ExternalName pointing to localhost which allow the admin webpage to be exposed.

Contour has a way to enable a clean shutdown sequence (https://projectcontour.io/docs/v1.17.1/redeploy-envoy/) by first calling the /healthcheck/fail endpoint in Envoy and then polling for open connections.

We also expose metrics on a new listener, so I can hide the rest of the admin functionality via that new listener. We're struggling with how to approach this. Our current solution is to disable externalName service but users want to use them for various different reasons.

The ultimate goal would be if we could drop the admin webpage from running, then we'd be able to better manage this issue.

[mattklein123]

I'd like a way to tell Envoy to begin draining connections and start shutting down, but not require the admin page to do this.

What do you suggest? I could see different somewhat hacky ways of doing this such as custom signals, watching some file, listening on a unix domain socket, etc., but it seems like the system using Envoy should be able to protect from such external attacks?

[sunjayBhatia]

Could be something in the runtime config? Might not be the right place since its not only configurable using the local filesystem, but using a file watch was what i thought of as well

[sunjayBhatia]

I think changing the Admin server to listen on a socket file shared between containers would work for us, havent tried it yet with the admin server listener

I see discussion in #2763 about making the admin server use a fully fledged listener, which would also be quite nice to give us the flexibility to fix it

[mattklein123]

You should be able to change the admin config to use a unix domain socket today.

[sunjayBhatia]

I get a malformed IP address exception when trying to use a Pipe address type

envoy/source/server/admin/admin.cc

Lines 120 to 144 in 9bd86b8

	void AdminImpl::startHttpListener(const std::list<AccessLog::InstanceSharedPtr>& access_logs,
	const std::string& address_out_path,
	Network::Address::InstanceConstSharedPtr address,
	const Network::Socket::OptionsSharedPtr& socket_options,
	Stats::ScopePtr&& listener_scope) {
	for (const auto& access_log : access_logs) {
	access_logs_.emplace_back(access_log);
	}
	null_overload_manager_.start();
	socket_ = std::make_shared<Network::TcpListenSocket>(address, socket_options, true);
	// TODO(mattklein123): We lost error handling along the way for the listen() call. Add it back.
	socket_->ioHandle().listen(ENVOY_TCP_BACKLOG_SIZE);
	socket_factory_ = std::make_unique<AdminListenSocketFactory>(socket_);
	listener_ = std::make_unique<AdminListener>(*this, std::move(listener_scope));
	ENVOY_LOG(info, "admin address: {}", socket().addressProvider().localAddress()->asString());
	if (!address_out_path.empty()) {
	std::ofstream address_out_file(address_out_path);
	if (!address_out_file) {
	ENVOY_LOG(critical, "cannot open admin address output file {} for writing.",
	address_out_path);
	} else {
	address_out_file << socket_->addressProvider().localAddress()->asString();
	}
	}
	}

doesnt seem to allow it

if youre amenable to it, we can probably make the change to support it, seems simple as checking the requested address type and creating the correct listen socket type?

[mattklein123]

I'm pretty positive people run admin over UDS, so not sure exactly what problem you are facing. I would debug it and if there is an actual bug, sure please fix it.

[sunjayBhatia]

yep youre right, had a typo somewhere else! yeah that should hopefully work for us, if we can also have some documentation how to still get to the admin page cc @stevesloka

[dprotaso]

if we can also have some documentation how to still get to the admin pag

@sunjayBhatia you could suggest kubectl exec into the container and have folks curl the socket address

ie.

admin:
  address:
    pipe: {"path":"/envoyproxy.sock"}

curl --unix-socket /envoyproxy.sock  http://localhost:

Abstract sockets work too

admin:
  address:
    pipe: {"path":"@/envoyproxy/admin"}

package main

import (
	"net/http"
	"net"
	"fmt"
	"context"
	"io/ioutil"
)

func main() {
	transport := http.DefaultTransport.(*http.Transport).Clone()
	transport.DialContext = func(_ context.Context, _, _ string) (net.Conn, error) {
		return net.Dial("unix", "@/envoyproxy/admin")
	}

	client := &http.Client{Transport: transport}
	resp, err := client.Get("http://localhost:")
	if err != nil {
		fmt.Println(err)
		return
	}

	defer resp.Body.Close()
	fmt.Println(resp.StatusCode)

	bytes, err := ioutil.ReadAll(resp.Body)
	fmt.Println(string(bytes), err)
}

[sunjayBhatia]

Only worried if we use abstract sockets if we ever enable envoy running on windows we will have to make another change but yeah we should def do this

[stevesloka]

I've got this working on a PR for Contour. I'll see if I can update docs in Envoy and also blog about it at least to explain the process if it helps anyone else out. =)

Thanks everyone for the help and input!