Handling ephemeral port exhaustion when using UDP protocol

[mzdeb]

Title: Handling ephemeral port exhaustion when using UDP protocol

Description:
Hi! I'm using envoy to load balance UDP traffic for service similar to DNS (1 packet with query, 1 packet with response). Everything works fine until traffic burst causes ephemeral port exhaustion and no new UDP sockets can be created. I'm able to delay the problem with idle_timeout setting for UdpProxyConfig however it's not a long term solution.

Questions:

1. Why envoy keeps outgoing UDP sockets in UNCONN state? This causes that SO_REUSEADDR cannot by used and local ports cannot be reused for different upstream IP/PORT tuples.

udp   UNCONN      0       0               0.0.0.0:2057           0.0.0.0:*
udp   UNCONN      0       0               0.0.0.0:2062           0.0.0.0:*
udp   UNCONN      0       0               0.0.0.0:2067           0.0.0.0:*
udp   UNCONN      0       0               0.0.0.0:2068           0.0.0.0:*
udp   UNCONN      0       0               0.0.0.0:2070           0.0.0.0:*
udp   UNCONN      0       0               0.0.0.0:2073           0.0.0.0:*
udp   UNCONN      0       0               0.0.0.0:2080           0.0.0.0:*
udp   UNCONN      0       0               0.0.0.0:2081           0.0.0.0:*
[...]

2. Is it possible to implement a list of source addresses in BindConfig to use multiple source ip? This is probably a proper long term solution recomended by HAProxy, Nginx and others.

Thanks for any hints! :)

[RyanTheOptimist]

@mattklein123 @michalmaka looks like you're both quite familiar with the UdpProxy filter so perhaps you have thoughts here.

[mattklein123]

Why envoy keeps outgoing UDP sockets in UNCONN state? This causes that SO_REUSEADDR cannot by used and local ports cannot be reused for different upstream IP/PORT tuples.

Sorry I don't know the details of this and how Envoy could effect it. If there are kernel/socket options we could be applying here feel free to propose them.

Is it possible to implement a list of source addresses in BindConfig to use multiple source ip? This is probably a proper long term solution recomended by HAProxy, Nginx and others.

Yes it's possible and is a partial duplicate of #15590

[mzdeb]

@mattklein123 usually when UDP socket is created for outgoing connection it should be created in following way:

s = socket()
s.connect('127.0.0.1', 12345)
s.send(payload)
data = s.recv()

these are so called connected UDP sockets (when creating socket we define the remote ip and port).

Envoy is doing it like this:

s = socket()
s.bind('0.0.0.0', 0)
s.send(data, '127.0.0.1', 12345)

and this one is unconnected UDP socket. The remote IP and port is known very late - when sending data. More details on this topic here.

The drawback of unconnected socket in my situation is that each connection blocks exactly one local port. For connected sockets situation would be much better because SO_REUSEADDR flag allows to reuse the same local port for multiple connections as long as tuple (local_ip, local_port, remote_ip, remote_port) is unique. In short I could workaround my problem by simply listening on multiple ports on upstream servers.

[mattklein123]

Hmm TBH I don't remember why I did what I did when I originally wrote this code. It might be possible to move to the first version of what you have above without any implications?

[mzdeb]

Well, I tried to investigate the envoy code, however my knowledge of c++ is too miserable. :( If the same udp socket is not reused for sending to multiple destinations then it should be safe.

[mattklein123]

envoy/source/extensions/filters/udp/udp_proxy/udp_proxy_filter.h

Lines 373 to 377 in a098234

	virtual Network::SocketPtr createSocket(const Upstream::HostConstSharedPtr& host) {
	// Virtual so this can be overridden in unit tests.
	return std::make_unique<Network::SocketImpl>(Network::Socket::Type::Datagram, host->address(),
	nullptr, Network::SocketCreationOptions{});
	}

That calls into code that does the bind. It could probably be swapped to connect instead.