Skip to content

triton-inference-server/server

Folders and files

NameName
Last commit message
Last commit date

Latest commit

873e177 · Feb 26, 2020
Oct 28, 2019
Jan 30, 2020
Feb 7, 2020
Feb 7, 2020
Feb 18, 2020
Feb 18, 2020
Feb 7, 2020
Dec 18, 2018
Nov 5, 2018
Jun 5, 2019
Nov 5, 2018
Feb 26, 2020
Feb 18, 2020
Feb 6, 2020
Feb 6, 2020
Jan 30, 2020
Oct 24, 2019
Feb 26, 2020
Feb 26, 2020
Nov 8, 2019

Repository files navigation

License

NVIDIA TensorRT Inference Server

The NVIDIA TensorRT Inference Server provides a cloud inferencing solution optimized for NVIDIA GPUs. The server provides an inference service via an HTTP or GRPC endpoint, allowing remote clients to request inferencing for any model being managed by the server.

What's New in 1.11.0

  • The TensorRT backend is improved to have significantly better performance. Improvements include reducing thread contention, using pinned memory for faster CPU<->GPU transfers, and increasing compute and memory copy overlap on GPUs.
  • Reduce memory usage of TensorRT models in many cases by sharing weights across multiple model instances.
  • Boolean data-type and shape tensors are now supported for TensorRT models.
  • A new model configuration option allows the dynamic batcher to create “ragged” batches for custom backend models. A ragged batch is a batch where one or more of the input/output tensors have different shapes in different batch entries.
  • Local S3 storage endpoints are now supported for model repositories. A local S3 endpoint is specified as s3://host:port/path/to/repository.
  • The Helm chart showing an example Kubernetes deployment is updated to include Prometheus and Grafana support so that inference server metrics can be collected and visualized.
  • The inference server container no longer sets LD_LIBRARY_PATH, instead the server uses RUNPATH to locate its shared libraries.
  • Python 2 is end-of-life so all support has been removed. Python 3 is still supported.

Features

  • Multiple framework support. The server can manage any number and mix of models (limited by system disk and memory resources). Supports TensorRT, TensorFlow GraphDef, TensorFlow SavedModel, ONNX, PyTorch, and Caffe2 NetDef model formats. Also supports TensorFlow-TensorRT integrated models. Variable-size input and output tensors are allowed if supported by the framework. See Capabilities for detailed support information for each framework.
  • Concurrent model execution support. Multiple models (or multiple instances of the same model) can run simultaneously on the same GPU.
  • Batching support. For models that support batching, the server can accept requests for a batch of inputs and respond with the corresponding batch of outputs. The inference server also supports multiple scheduling and batching algorithms that combine individual inference requests together to improve inference throughput. These scheduling and batching decisions are transparent to the client requesting inference.
  • Custom backend support. The inference server allows individual models to be implemented with custom backends instead of by a deep-learning framework. With a custom backend a model can implement any logic desired, while still benefiting from the GPU support, concurrent execution, dynamic batching and other features provided by the server.
  • Ensemble support. An ensemble represents a pipeline of one or more models and the connection of input and output tensors between those models. A single inference request to an ensemble will trigger the execution of the entire pipeline.
  • Multi-GPU support. The server can distribute inferencing across all system GPUs.
  • The inference server provides multiple modes for model management. These model management modes allow for both implicit and explicit loading and unloading of models without requiring a server restart.
  • Model repositories may reside on a locally accessible file system (e.g. NFS), in Google Cloud Storage or in Amazon S3.
  • Readiness and liveness health endpoints suitable for any orchestration or deployment framework, such as Kubernetes.
  • Metrics indicating GPU utilization, server throughput, and server latency.
  • C library inferface allows the full functionality of the inference server to be included directly in an application.

The current release of the TensorRT Inference Server is 1.11.0 and corresponds to the 20.02 release of the tensorrtserver container on NVIDIA GPU Cloud (NGC). The branch for this release is r20.02.

Backwards Compatibility

Continuing in the latest version the following interfaces maintain backwards compatibilty with the 1.0.0 release. If you have model configuration files, custom backends, or clients that use the inference server HTTP or GRPC APIs (either directly or through the client libraries) from releases prior to 1.0.0 you should edit and rebuild those as necessary to match the version 1.0.0 APIs.

The following inferfaces will maintain backwards compatibility for all future 1.x.y releases (see below for exceptions):

As new features are introduced they may temporarily have beta status where they are subject to change in non-backwards-compatible ways. When they exit beta they will conform to the backwards-compatibility guarantees described above. Currently the following features are in beta:

  • The inference server library API as defined in trtserver.h is currently in beta and may undergo non-backwards-compatible changes.
  • The inference server HTTP and GRPC APIs related to system and CUDA shared memory are currently in beta and may undergo non-backwards-compatible changes.
  • The V2 custom backend interface as defined in custom.h is currently in beta and may undergo non-backwards-compatible changes.
  • The C++ and Python client libraries are not stictly included in the inference server compatibility guarantees and so should be considered as beta status.

Documentation

The User Guide, Developer Guide, and API Reference documentation for the current release provide guidance on installing, building, and running the TensorRT Inference Server.

You can also view the documentation for the master branch and for earlier releases.

An FAQ provides answers for frequently asked questions.

READMEs for deployment examples can be found in subdirectories of deploy/, for example, deploy/single_server/README.rst.

The Release Notes and Support Matrix indicate the required versions of the NVIDIA Driver and CUDA, and also describe which GPUs are supported by the inference server.

Other Documentation

Contributing

Contributions to TensorRT Inference Server are more than welcome. To contribute make a pull request and follow the guidelines outlined in the Contributing document.

Reporting problems, asking questions

We appreciate any feedback, questions or bug reporting regarding this project. When help with code is needed, follow the process outlined in the Stack Overflow (https://stackoverflow.com/help/mcve) document. Ensure posted examples are:

  • minimal – use as little code as possible that still produces the same problem
  • complete – provide all parts needed to reproduce the problem. Check if you can strip external dependency and still show the problem. The less time we spend on reproducing problems the more time we have to fix it
  • verifiable – test the code you're about to provide to make sure it reproduces the problem. Remove all other problems that are not related to your request/question.