Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add configurable timeouts to TransportServer #1346

Merged
merged 14 commits into from
Feb 9, 2021
Merged

Add configurable timeouts to TransportServer #1346

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
0e0618d
Add configurable timeouts to transportServer
LorcanMcVeigh Jan 21, 2021
871142f
Feedback
LorcanMcVeigh Feb 4, 2021
fb6a841
Apply suggestions from code review
LorcanMcVeigh Feb 4, 2021
259582c
Update test name
LorcanMcVeigh Feb 4, 2021
17177d8
Fix typo
LorcanMcVeigh Feb 4, 2021
4667ca3
fix typo in yaml
LorcanMcVeigh Feb 4, 2021
18aecb7
Address feedback
LorcanMcVeigh Feb 4, 2021
fb1f1cb
Update unit tests
LorcanMcVeigh Feb 4, 2021
a9358fb
Address feedback
LorcanMcVeigh Feb 8, 2021
992b524
Apply suggestions from code review
LorcanMcVeigh Feb 8, 2021
a4c018a
Update crd descriptions
LorcanMcVeigh Feb 8, 2021
9ddff8b
Address feedback 2
LorcanMcVeigh Feb 9, 2021
92954e3
Apply suggestions from code review
LorcanMcVeigh Feb 9, 2021
951d782
Feedback 3
LorcanMcVeigh Feb 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions deployments/common/crds-v1beta1/k8s.nginx.org_transportservers.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,10 +49,24 @@ spec:
type: string
protocol:
type: string
sessionParameters:
description: SessionParameters defines session parameters.
type: object
properties:
timeout:
type: string
upstreamParameters:
description: UpstreamParameters defines parameters for an upstream.
type: object
properties:
connectTimeout:
type: string
nextUpstream:
type: boolean
nextUpstreamTimeout:
type: string
nextUpstreamTries:
type: integer
udpRequests:
type: integer
udpResponses:
Expand Down
14 changes: 14 additions & 0 deletions deployments/common/crds/k8s.nginx.org_transportservers.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,10 +50,24 @@ spec:
type: string
protocol:
type: string
sessionParameters:
description: SessionParameters defines session parameters.
type: object
properties:
timeout:
type: string
upstreamParameters:
description: UpstreamParameters defines parameters for an upstream.
type: object
properties:
connectTimeout:
type: string
nextUpstream:
type: boolean
nextUpstreamTimeout:
type: string
nextUpstreamTries:
type: integer
udpRequests:
type: integer
udpResponses:
Expand Down
14 changes: 14 additions & 0 deletions deployments/helm-chart/crds/k8s.nginx.org_transportservers.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,10 +49,24 @@ spec:
type: string
protocol:
type: string
sessionParameters:
description: SessionParameters defines session parameters.
type: object
properties:
timeout:
type: string
upstreamParameters:
description: UpstreamParameters defines parameters for an upstream.
type: object
properties:
connectTimeout:
type: string
nextUpstream:
type: boolean
nextUpstreamTimeout:
type: string
nextUpstreamTries:
type: integer
udpRequests:
type: integer
udpResponses:
Expand Down
47 changes: 45 additions & 2 deletions docs-web/configuration/transportserver-resource.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ This document is the reference documentation for the TransportServer resource. T
- [Listener](#listener)
- [Upstream](#upstream)
- [UpstreamParameters](#upstreamparameters)
- [SessionParameters](#sessionparameters)
- [Action](#action)
- [Using TransportServer](#using-transportserver)
- [Validation](#validation)
Expand Down Expand Up @@ -182,11 +183,15 @@ port: 8443

### UpstreamParameters

The upstream parameters define various parameters for the upstreams. For now, only UDP-related parameters are supported:
The upstream parameters define various parameters for the upstreams:
```yaml
upstreamParameters:
udpRequests: 1
udpResponses: 1
connectTimeout: 60s
nextUpstream: true
nextUpstreamTimeout: 50s
nextUpstreamTries: 1
```

```eval_rst
Expand All @@ -205,6 +210,44 @@ upstreamParameters:
- The number of datagrams expected from the proxied server in response to a client datagram. See the `proxy_responses <https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_responses>`_ directive. By default, the number of datagrams is not limited.
- ``int``
- No
* - ``connectTimeout``
- The timeout for establishing a connection with a proxied server. See the `proxy_connect_timeout <http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_connect_timeout>`_ directive. The default is ``60s``.
- ``string``
- No
* - ``nextUpstream``
- If a connection to the proxied server cannot be established, determines whether a client connection will be passed to the next server. See the `proxy_next_upstream <http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream>`_ directive. The default is ``true``.
- bool
- No
* - ``nextUpstreamTries``
- The number of tries for passing a connection to the next server. See the `proxy_next_upstream_tries <http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream_tries>`_ directive. The default is ``0``.
- ``int``
- No
* - ``nextUpstreamTimeout``
- The time allowed to pass a connection to the next server. See the `proxy_next_upstream_timeout <http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream_timeout>`_ directive. The default us ``0``.
- ``string``
- No
```

### SessionParameters
LorcanMcVeigh marked this conversation as resolved.
Show resolved Hide resolved

The session parameters define various parameters for TCP connections and UDP sessions.
```yaml
sessionParameters:
timeout: 50s
```

```eval_rst
.. list-table::
:header-rows: 1

* - Field
- Description
- Type
- Required
* - ``timeout``
LorcanMcVeigh marked this conversation as resolved.
Show resolved Hide resolved
- The timeout between two succesive read or write operations on client or proxied server connections. See `proxy_timeout <http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_timeout>`_ directive. The default is ``10m``.
LorcanMcVeigh marked this conversation as resolved.
Show resolved Hide resolved
- ``string``
- No
```

### Action
Expand Down Expand Up @@ -312,4 +355,4 @@ The [ConfigMap](/nginx-ingress-controller/configuration/global-configuration/con
As of Release 1.7, the TransportServer resource is a preview feature. Currently, it comes with the following limitations:
* When using TLS Passthrough, it is not possible to configure [Proxy Protocol](https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/proxy-protocol) for port 443 both for regular HTTPS and TLS Passthrough traffic.
* If multiple TCP (or UDP) TransportServers reference the same listener, only one of them will receive the traffic. Moreover, until there is only one TransportServer, NGINX will fail to reload. If this happens, the IC will report a warning event with the `AddedOrUpdatedWithError` reason for the resource, which caused the problem, and also report the error in the logs.
* If multiple TLS Passthrough TransportServers have the same hostname, only one of them will receive the traffic. If this happens, the IC will report a warning in the logs like `host "app.example.com" is used by more than one TransportServers`.
* If multiple TLS Passthrough TransportServers have the same hostname, only one of them will receive the traffic. If this happens, the IC will report a warning in the logs like `host "app.example.com" is used by more than one TransportServers`.
42 changes: 32 additions & 10 deletions internal/configs/transportserver.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,27 @@ func generateTransportServerConfig(transportServerEx *TransportServerEx, listene
upstreams := generateStreamUpstreams(transportServerEx, upstreamNamer, isPlus)

var proxyRequests, proxyResponses *int
var connectTimeout, nextUpstreamTimeout string
var nextUpstream bool
var nextUpstreamTries int
if transportServerEx.TransportServer.Spec.UpstreamParameters != nil {
proxyRequests = transportServerEx.TransportServer.Spec.UpstreamParameters.UDPRequests
proxyResponses = transportServerEx.TransportServer.Spec.UpstreamParameters.UDPResponses

nextUpstream = transportServerEx.TransportServer.Spec.UpstreamParameters.NextUpstream
if nextUpstream {
nextUpstreamTries = transportServerEx.TransportServer.Spec.UpstreamParameters.NextUpstreamTries
nextUpstreamTimeout = transportServerEx.TransportServer.Spec.UpstreamParameters.NextUpstreamTimeout
LorcanMcVeigh marked this conversation as resolved.
Show resolved Hide resolved
}

connectTimeout = transportServerEx.TransportServer.Spec.UpstreamParameters.ConnectTimeout
}

var proxyTimeout string
if transportServerEx.TransportServer.Spec.SessionParameters != nil {
proxyTimeout = transportServerEx.TransportServer.Spec.SessionParameters.Timeout
LorcanMcVeigh marked this conversation as resolved.
Show resolved Hide resolved
}

statusZone := ""
if transportServerEx.TransportServer.Spec.Listener.Name == conf_v1alpha1.TLSPassthroughListenerName {
statusZone = transportServerEx.TransportServer.Spec.Host
Expand All @@ -48,16 +65,21 @@ func generateTransportServerConfig(transportServerEx *TransportServerEx, listene

return version2.TransportServerConfig{
Server: version2.StreamServer{
TLSPassthrough: transportServerEx.TransportServer.Spec.Listener.Name == conf_v1alpha1.TLSPassthroughListenerName,
UnixSocket: generateUnixSocket(transportServerEx),
Port: listenerPort,
UDP: transportServerEx.TransportServer.Spec.Listener.Protocol == "UDP",
StatusZone: statusZone,
ProxyRequests: proxyRequests,
ProxyResponses: proxyResponses,
ProxyPass: upstreamNamer.GetNameForUpstream(transportServerEx.TransportServer.Spec.Action.Pass),
Name: transportServerEx.TransportServer.Name,
Namespace: transportServerEx.TransportServer.Namespace,
TLSPassthrough: transportServerEx.TransportServer.Spec.Listener.Name == conf_v1alpha1.TLSPassthroughListenerName,
UnixSocket: generateUnixSocket(transportServerEx),
Port: listenerPort,
UDP: transportServerEx.TransportServer.Spec.Listener.Protocol == "UDP",
StatusZone: statusZone,
ProxyRequests: proxyRequests,
ProxyResponses: proxyResponses,
ProxyPass: upstreamNamer.GetNameForUpstream(transportServerEx.TransportServer.Spec.Action.Pass),
Name: transportServerEx.TransportServer.Name,
Namespace: transportServerEx.TransportServer.Namespace,
ProxyConnectTimeout: generateString(connectTimeout, "60s"),
ProxyTimeout: generateString(proxyTimeout, "10m"),
ProxyNextUpstream: nextUpstream,
ProxyNextUpstreamTimeout: generateString(nextUpstreamTimeout, "0"),
ProxyNextUpstreamTries: nextUpstreamTries,
},
Upstreams: upstreams,
}
Expand Down
80 changes: 56 additions & 24 deletions internal/configs/transportserver_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,13 @@ func TestGenerateTransportServerConfigForTCP(t *testing.T) {
Port: 5001,
},
},
UpstreamParameters: &conf_v1alpha1.UpstreamParameters{
ConnectTimeout: "30s",
NextUpstream: false,
},
SessionParameters: &conf_v1alpha1.SessionParameters{
Timeout: "50s",
},
Action: &conf_v1alpha1.Action{
Pass: "tcp-app",
},
Expand Down Expand Up @@ -112,12 +119,17 @@ func TestGenerateTransportServerConfigForTCP(t *testing.T) {
},
},
Server: version2.StreamServer{
Port: 2020,
UDP: false,
StatusZone: "tcp-listener",
ProxyPass: "ts_default_tcp-server_tcp-app",
Name: "tcp-server",
Namespace: "default",
Port: 2020,
UDP: false,
StatusZone: "tcp-listener",
ProxyPass: "ts_default_tcp-server_tcp-app",
Name: "tcp-server",
Namespace: "default",
ProxyConnectTimeout: "30s",
ProxyNextUpstream: false,
ProxyNextUpstreamTries: 0,
ProxyNextUpstreamTimeout: "0",
ProxyTimeout: "50s",
},
}

Expand Down Expand Up @@ -148,6 +160,12 @@ func TestGenerateTransportServerConfigForTLSPasstrhough(t *testing.T) {
Port: 5001,
},
},
UpstreamParameters: &conf_v1alpha1.UpstreamParameters{
ConnectTimeout: "30s",
NextUpstream: false,
NextUpstreamTries: 0,
NextUpstreamTimeout: "",
},
Action: &conf_v1alpha1.Action{
Pass: "tcp-app",
},
Expand Down Expand Up @@ -180,14 +198,19 @@ func TestGenerateTransportServerConfigForTLSPasstrhough(t *testing.T) {
},
},
Server: version2.StreamServer{
TLSPassthrough: true,
UnixSocket: "unix:/var/lib/nginx/passthrough-default_tcp-server.sock",
Port: 2020,
UDP: false,
StatusZone: "example.com",
ProxyPass: "ts_default_tcp-server_tcp-app",
Name: "tcp-server",
Namespace: "default",
TLSPassthrough: true,
UnixSocket: "unix:/var/lib/nginx/passthrough-default_tcp-server.sock",
Port: 2020,
UDP: false,
StatusZone: "example.com",
ProxyPass: "ts_default_tcp-server_tcp-app",
Name: "tcp-server",
Namespace: "default",
ProxyConnectTimeout: "30s",
ProxyNextUpstream: false,
ProxyNextUpstreamTimeout: "0",
ProxyNextUpstreamTries: 0,
ProxyTimeout: "10m",
},
}

Expand Down Expand Up @@ -221,8 +244,12 @@ func TestGenerateTransportServerConfigForUDP(t *testing.T) {
},
},
UpstreamParameters: &conf_v1alpha1.UpstreamParameters{
UDPRequests: &udpRequests,
UDPResponses: &udpResponses,
UDPRequests: &udpRequests,
UDPResponses: &udpResponses,
ConnectTimeout: "30s",
NextUpstream: true,
NextUpstreamTimeout: "",
NextUpstreamTries: 0,
},
Action: &conf_v1alpha1.Action{
Pass: "udp-app",
Expand Down Expand Up @@ -256,14 +283,19 @@ func TestGenerateTransportServerConfigForUDP(t *testing.T) {
},
},
Server: version2.StreamServer{
Port: 2020,
UDP: true,
StatusZone: "udp-listener",
ProxyRequests: &udpRequests,
ProxyResponses: &udpResponses,
ProxyPass: "ts_default_udp-server_udp-app",
Name: "udp-server",
Namespace: "default",
Port: 2020,
UDP: true,
StatusZone: "udp-listener",
ProxyRequests: &udpRequests,
ProxyResponses: &udpResponses,
ProxyPass: "ts_default_udp-server_udp-app",
Name: "udp-server",
Namespace: "default",
ProxyConnectTimeout: "30s",
ProxyNextUpstream: true,
ProxyNextUpstreamTimeout: "0",
ProxyNextUpstreamTries: 0,
ProxyTimeout: "10m",
},
}

Expand Down
9 changes: 9 additions & 0 deletions internal/configs/version2/nginx-plus.transportserver.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,13 @@ server {
{{ end }}

proxy_pass {{ $s.ProxyPass }};

proxy_timeout {{ $s.ProxyTimeout }};
LorcanMcVeigh marked this conversation as resolved.
Show resolved Hide resolved
proxy_connect_timeout {{ $s.ProxyConnectTimeout }};

{{ if $s.ProxyNextUpstream }}
proxy_next_upstream on;
proxy_next_upstream_timeout {{ $s.ProxyNextUpstreamTimeout }};
proxy_next_upstream_tries {{ $s.ProxyNextUpstreamTries }};
{{ end }}
}
9 changes: 9 additions & 0 deletions internal/configs/version2/nginx.transportserver.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,13 @@ server {
{{ end }}

proxy_pass {{ $s.ProxyPass }};

proxy_timeout {{ $s.ProxyTimeout }};
proxy_connect_timeout {{ $s.ProxyConnectTimeout }};

{{ if $s.ProxyNextUpstream }}
proxy_next_upstream on;
proxy_next_upstream_timeout {{ $s.ProxyNextUpstreamTimeout }};
proxy_next_upstream_tries {{ $s.ProxyNextUpstreamTries }};
{{ end }}
}
25 changes: 15 additions & 10 deletions internal/configs/version2/stream.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,16 +20,21 @@ type StreamUpstreamServer struct {

// StreamServer defines a server in the stream module.
type StreamServer struct {
TLSPassthrough bool
UnixSocket string
Port int
UDP bool
StatusZone string
ProxyRequests *int
ProxyResponses *int
ProxyPass string
Name string
Namespace string
TLSPassthrough bool
UnixSocket string
Port int
UDP bool
StatusZone string
ProxyRequests *int
ProxyResponses *int
ProxyPass string
Name string
Namespace string
ProxyTimeout string
ProxyConnectTimeout string
ProxyNextUpstream bool
ProxyNextUpstreamTimeout string
ProxyNextUpstreamTries int
}

// TLSPassthroughHostsConfig defines a mapping between TLS Passthrough hosts and the corresponding unix sockets.
Expand Down
Loading