Does tritonserver Support IPv6 --http-address bind?

[malcolmgreaves]

Description
tritonserver does not appear to support an IPv6 value for --http-address. Cannot connect to server when started with an IPv6 address.

Triton Information
2.21.0

(Also reproduced on 2.30.0)

Are you using the Triton container or did you build it yourself?

Using image tagged 22.04-py3

(Also reproduced on image tagged 23.01-py3)

To Reproduce

1. Start tritonserver with option --http-address ::.
   a. I have also tried --http-address values [::] and [::1]
2. Attempt to hit the health check: curl -w "%{http_code}" -X GET "http://[::1]:8000/v2/health/ready"
3. Expecting 200 but instead get: curl: (7) Failed to connect to ::1 port 8000: Connection refused
   a. in version 2.30.0, I get curl: (7) Couldn't connect to server

Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well).

Agnostic to model. Reproduced using image classifier example.

Expected behavior
Return an HTTP 200 OK status code.

[malcolmgreaves]

This may or may not help, but this is what netstat shows when I run tritonserver --http-address :::

$ netstat -lnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 255.255.255.255:8000    0.0.0.0:*               LISTEN      41/tritonserver
tcp        0      0 255.255.255.255:8002    0.0.0.0:*               LISTEN      41/tritonserver
tcp6       0      0 :::8001                 :::*                    LISTEN      41/tritonserver
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name     Path
unix  2      [ ACC ]     SEQPACKET  LISTENING     1576793  41/tritonserver      @cuda-uvmfd-4026533836-41@

It seems odd to me that the HTTP & metrics server ports (8000 and 8002, respectively) are not reported as tcp6 protocols. Oddly, even though I don't request --grpc-address to be IPv6, it is using the tcp6 protocol.

[malcolmgreaves]

This is what the server log reports on startup:

I0203 01:48:01.534312 75 grpc_server.cc:4819] Started GRPCInferenceService at 0.0.0.0:8001
I0203 01:48:01.534586 75 http_server.cc:3477] Started HTTPService at [::]:8000
I0203 01:48:01.589119 75 http_server.cc:184] Started Metrics Service at [::]:8002

I am only showing it when I tried it with --http-address [::]. The other values show each show :: and [::1] for the HTTPService and Metrics Service log lines.

[malcolmgreaves]

Hi @mc-nv , @tanmayv25, @Tabrizian, @dyastremsky -- if you have some time, I would greatly appreciate if any of you could take a look into this issue. 🙏 Thank you for your work on Triton!

[tanmayv25]

@malcolmgreaves Unfortunately, Triton endpoint does not support ipv6. You can try using nginx proxy in front of Triton endpoint to expose ipv6 endpoint.
This article provides nice tutorial on how to implement this: https://thecustomizewindows.com/2016/11/add-ipv6-ipv4-server-nginx-reverse-proxy/

Triton does support ipv6.

[malcolmgreaves]

Thank you for posting a workaround @tanmayv25. I will look into using this!

[malcolmgreaves]

I think others will be able to find this comment thread to learn. And I know that this is probably one of the lowest priorities. However, it might be useful to have tritonserver fail fast if the supplied --http-address isn't IPv4 compatible.

If I were going to submit a PR for this behavior, do you have any pointers as to where I should start looking?

[tanmayv25]

@malcolmgreaves I got curious and started looking through the code and play around a little bit. It seems like tritonserver can support ipv6.
I looked at the evthp implementation that we use in our http endpoint. They seem to be handling ipv6. See here.

So this is what I did. I updated docker config to enable ipv6 connection and launched tritonserver container with --network host option.
Within the container, I launched tritonserver as:

tritonserver --model-store=/tmp/host/models/ --http-address ipv6:[::1]&

I see the following:

I0215 21:04:11.572305 571 grpc_server.cc:4868] Started GRPCInferenceService at 0.0.0.0:8001
I0215 21:04:11.572528 571 http_server.cc:3477] Started HTTPService at ipv6:[::1]:8000
I0215 21:04:11.614167 571 http_server.cc:184] Started Metrics Service at ipv6:[::1]:8002

The netstat output is :

root@tanmay-X299-A:/opt/tritonserver# netstat -tulpn | grep tritonserver
tcp6       0      0 :::8000                 :::*                    LISTEN      571/tritonserver    
tcp6       0      0 :::8001                 :::*                    LISTEN      571/tritonserver    
tcp6       0      0 :::8002                 :::*                    LISTEN      571/tritonserver 

And the curl command runs as expected:

curl -6 --verbose "http://[::1]:8000/v2/health/ready"
*   Trying ::1:8000...
* TCP_NODELAY set
* Connected to ::1 (::1) port 8000 (#0)
> GET /v2/health/ready HTTP/1.1
> Host: [::1]:8000
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Length: 0
< Content-Type: text/plain
< 
* Connection #0 to host ::1 left intact

These steps might not be too refined as I am not to familiar with ipv6. Let me know if you learn anything new or have any questions. But I was able to run inference on the server using perf_analyzer as well which touches various different HTTP APIs.

 perf_analyzer -m savedmodel_int32_int8_int8
*** Measurement Settings ***
  Batch size: 1
  Service Kind: Triton
  Using "time_windows" mode for stabilization
  Measurement window: 5000 msec
  Using synchronous calls for inference
  Stabilizing using average latency

Request concurrency: 1
  Client: 
    Request count: 18462
    Throughput: 1025.31 infer/sec
    Avg latency: 973 usec (standard deviation 488 usec)
    p50 latency: 954 usec
    p90 latency: 1166 usec
    p95 latency: 1253 usec
    p99 latency: 1448 usec
    Avg HTTP time: 964 usec (send/recv 106 usec + response wait 858 usec)
  Server: 
    Inference count: 18463
    Execution count: 18463
    Successful request count: 18463
    Avg request latency: 594 usec (overhead 37 usec + queue 50 usec + compute input 23 usec + compute infer 466 usec + compute output 16 usec)

Inferences/Second vs. Client Average Batch Latency
Concurrency: 1, throughput: 1025.31 infer/sec, latency 973 usec

[malcolmgreaves]

Wow! Thank you so much for digging into this! ❤️ This looks like it will work well for my use case! I am using Triton in a k8s deployment. The cluster I'm using requires IPv6 networking. I will try this out and let you know how it goes!

[krishung5]

Closing due to inactivity. Please let us know if you would like to reopen the issue for follow-up.

[metavee]

@tanmayv25 's solution of --http-address=ipv6:[::] worked for me on an IPv6 only kubernetes cluster! Thank you for posting it.