Skip to content

Commit

Permalink
Update post-23.11 release (#6653)
Browse files Browse the repository at this point in the history
* Update README and versions for 2.40.0 / 23.11 (#6544)

* Removing path construction to use SymLink alternatives

* Update version for PyTorch

* Update windows Dockerfile configuration

* Update triton version to 23.11

* Update README and versions for 2.40.0 / 23.11

* Fix typo

* Ading 'ldconfig' to configure dynamic linking in container (#6602)

* Point to tekit_backend (#6616)

* Point to tekit_backend

* Update version

* Revert tekit changes (#6640)

---------

Co-authored-by: Kris Hung <[email protected]>
  • Loading branch information
mc-nv and krishung5 authored Dec 4, 2023
1 parent b96ae5f commit b9f69b5
Show file tree
Hide file tree
Showing 26 changed files with 74 additions and 54 deletions.
2 changes: 1 addition & 1 deletion Dockerfile.sdk
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
#

# Base image on the minimum Triton container
ARG BASE_IMAGE=nvcr.io/nvidia/tritonserver:23.10-py3-min
ARG BASE_IMAGE=nvcr.io/nvidia/tritonserver:23.11-py3-min

ARG TRITON_CLIENT_REPO_SUBDIR=clientrepo
ARG TRITON_COMMON_REPO_TAG=main
Expand Down
26 changes: 21 additions & 5 deletions Dockerfile.win10.min
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,13 @@ ARG BUILD_TOOLS_SOURCE=https://aka.ms/vs/17/release/vs_buildtools.exe
ADD ${BUILD_TOOLS_SOURCE} vs_buildtools.exe
# Install Build Tools with the Microsoft.VisualStudio.Workload.VCTools workload, including recommended.
ARG VS_INSTALL_PATH_WP="C:\BuildTools"
RUN vs_buildtools.exe --quiet --wait --norestart --nocache install --installPath %VS_INSTALL_PATH_WP% --channelUri "C:\tmp\VisualStudio.chman" --installChannelUri "C:\tmp\VisualStudio.chman" --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended --locale "En-us"
RUN vs_buildtools.exe --quiet --wait --norestart --nocache install \
--installPath %VS_INSTALL_PATH_WP% \
--channelUri "C:\tmp\VisualStudio.chman" \
--installChannelUri "C:\tmp\VisualStudio.chman" \
--add Microsoft.VisualStudio.Workload.VCTools \
--includeRecommended \
--locale "En-us"

LABEL BUILDTOOLS_VERSION=${BUILDTOOLS_VERSION}

Expand All @@ -97,7 +103,17 @@ RUN git clone --single-branch --depth=1 -b %VCPGK_VERSION% https://github.com/mi
WORKDIR /vcpkg
RUN bootstrap-vcpkg.bat
RUN vcpkg.exe update
RUN vcpkg.exe install openssl:x64-windows openssl-windows:x64-windows rapidjson:x64-windows re2:x64-windows boost-filesystem:x64-windows boost-interprocess:x64-windows boost-stacktrace:x64-windows zlib:x64-windows pthread:x64-windows b64:x64-windows
RUN vcpkg.exe install \
b64:x64-windows \
boost-filesystem:x64-windows \
boost-interprocess:x64-windows \
boost-stacktrace:x64-windows \
openssl-windows:x64-windows \
openssl:x64-windows \
pthread:x64-windows \
rapidjson:x64-windows \
re2:x64-windows \
zlib:x64-windows
RUN vcpkg.exe integrate install

LABEL VCPGK_VERSION=${VCPGK_VERSION}
Expand All @@ -108,8 +124,8 @@ WORKDIR /
# Installing CUDA
#
ARG CUDA_MAJOR=12
ARG CUDA_MINOR=2
ARG CUDA_PATCH=1
ARG CUDA_MINOR=3
ARG CUDA_PATCH=0
ARG CUDA_VERSION=${CUDA_MAJOR}.${CUDA_MINOR}.${CUDA_PATCH}
ARG CUDA_PACKAGES="nvcc_${CUDA_MAJOR}.${CUDA_MINOR} \
cudart_${CUDA_MAJOR}.${CUDA_MINOR} \
Expand Down Expand Up @@ -157,7 +173,7 @@ LABEL TENSORRT_VERSION="${TENSORRT_VERSION}"
#
# Installing cuDNN
#
ARG CUDNN_VERSION=8.9.5.27
ARG CUDNN_VERSION=8.9.6.50
ARG CUDNN_ZIP=cudnn-windows-x86_64-${CUDNN_VERSION}_cuda12-archive.zip
ARG CUDNN_SOURCE=${CUDNN_ZIP}

Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@

**LATEST RELEASE: You are currently on the main branch which tracks
under-development progress towards the next release. The current release is
version [2.38.0](https://github.com/triton-inference-server/server/tree/r23.09)
and corresponds to the 23.09 container release on
version [2.40.0](https://github.com/triton-inference-server/server/tree/r23.11)
and corresponds to the 23.11 container release on
[NVIDIA GPU Cloud (NGC)](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver).**

----
Expand Down Expand Up @@ -93,16 +93,16 @@ Inference Server with the

```bash
# Step 1: Create the example model repository
git clone -b r23.10 https://github.com/triton-inference-server/server.git
git clone -b r23.11 https://github.com/triton-inference-server/server.git
cd server/docs/examples
./fetch_models.sh

# Step 2: Launch triton from the NGC Triton container
docker run --gpus=1 --rm --net=host -v ${PWD}/model_repository:/models nvcr.io/nvidia/tritonserver:23.10-py3 tritonserver --model-repository=/models
docker run --gpus=1 --rm --net=host -v ${PWD}/model_repository:/models nvcr.io/nvidia/tritonserver:23.11-py3 tritonserver --model-repository=/models

# Step 3: Sending an Inference Request
# In a separate console, launch the image_client example from the NGC Triton SDK container
docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:23.10-py3-sdk
docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:23.11-py3-sdk
/workspace/install/bin/image_client -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg

# Inference should return the following
Expand Down
10 changes: 5 additions & 5 deletions build.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@
TRITON_VERSION_MAP = {
"2.41.0dev": (
"23.12dev", # triton container
"23.10", # upstream container
"23.11", # upstream container
"1.16.3", # ORT
"2023.0.0", # ORT OpenVINO
"2023.0.0", # Standalone OpenVINO
Expand Down Expand Up @@ -1389,10 +1389,10 @@ def add_cpu_libs_to_linux_dockerfile(backends, target_machine):
COPY --from=min_container /usr/local/cuda/lib64/stubs/libcublasLt.so /usr/local/cuda/lib64/stubs/libcublasLt.so.11
RUN mkdir -p /usr/local/cuda/targets/{cuda_arch}-linux/lib
COPY --from=min_container /usr/local/cuda-12.2/targets/{cuda_arch}-linux/lib/libcudart.so.12 /usr/local/cuda/targets/{cuda_arch}-linux/lib/.
COPY --from=min_container /usr/local/cuda-12.2/targets/{cuda_arch}-linux/lib/libcupti.so.12 /usr/local/cuda/targets/{cuda_arch}-linux/lib/.
COPY --from=min_container /usr/local/cuda-12.2/targets/{cuda_arch}-linux/lib/libnvToolsExt.so.1 /usr/local/cuda/targets/{cuda_arch}-linux/lib/.
COPY --from=min_container /usr/local/cuda-12.2/targets/{cuda_arch}-linux/lib/libnvJitLink.so.12 /usr/local/cuda/targets/{cuda_arch}-linux/lib/.
COPY --from=min_container /usr/local/cuda/lib64/libcudart.so.12 /usr/local/cuda/targets/{cuda_arch}-linux/lib/.
COPY --from=min_container /usr/local/cuda/lib64/libcupti.so.12 /usr/local/cuda/targets/{cuda_arch}-linux/lib/.
COPY --from=min_container /usr/local/cuda/lib64/libnvToolsExt.so.1 /usr/local/cuda/targets/{cuda_arch}-linux/lib/.
COPY --from=min_container /usr/local/cuda/lib64/libnvJitLink.so.12 /usr/local/cuda/targets/{cuda_arch}-linux/lib/.
RUN mkdir -p /opt/hpcx/ucc/lib/ /opt/hpcx/ucx/lib/
COPY --from=min_container /opt/hpcx/ucc/lib/libucc.so.1 /opt/hpcx/ucc/lib/libucc.so.1
Expand Down
2 changes: 1 addition & 1 deletion deploy/aws/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
replicaCount: 1

image:
imageName: nvcr.io/nvidia/tritonserver:23.10-py3
imageName: nvcr.io/nvidia/tritonserver:23.11-py3
pullPolicy: IfNotPresent
modelRepositoryPath: s3://triton-inference-server-repository/model_repository
numGpus: 1
Expand Down
2 changes: 1 addition & 1 deletion deploy/fleetcommand/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@

apiVersion: v1
# appVersion is the Triton version; update when changing release
appVersion: "2.39.0"
appVersion: "2.40.0"
description: Triton Inference Server (Fleet Command)
name: triton-inference-server
# version is the Chart version; update when changing anything in the chart
Expand Down
6 changes: 3 additions & 3 deletions deploy/fleetcommand/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
replicaCount: 1

image:
imageName: nvcr.io/nvidia/tritonserver:23.10-py3
imageName: nvcr.io/nvidia/tritonserver:23.11-py3
pullPolicy: IfNotPresent
numGpus: 1
serverCommand: tritonserver
Expand All @@ -46,13 +46,13 @@ image:
# Model Control Mode (Optional, default: none)
#
# To set model control mode, uncomment and configure below
# See https://github.com/triton-inference-server/server/blob/r23.10/docs/model_management.md
# See https://github.com/triton-inference-server/server/blob/r23.11/docs/model_management.md
# for more details
#- --model-control-mode=explicit|poll|none
#
# Additional server args
#
# see https://github.com/triton-inference-server/server/blob/r23.10/README.md
# see https://github.com/triton-inference-server/server/blob/r23.11/README.md
# for more details

service:
Expand Down
2 changes: 1 addition & 1 deletion deploy/gcp/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
replicaCount: 1

image:
imageName: nvcr.io/nvidia/tritonserver:23.10-py3
imageName: nvcr.io/nvidia/tritonserver:23.11-py3
pullPolicy: IfNotPresent
modelRepositoryPath: gs://triton-inference-server-repository/model_repository
numGpus: 1
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ metadata:
namespace: default
spec:
containers:
- image: nvcr.io/nvidia/tritonserver:23.10-py3-sdk
- image: nvcr.io/nvidia/tritonserver:23.11-py3-sdk
imagePullPolicy: Always
name: nv-triton-client
securityContext:
Expand Down
6 changes: 3 additions & 3 deletions deploy/gke-marketplace-app/server-deployer/build_and_push.sh
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@

export REGISTRY=gcr.io/$(gcloud config get-value project | tr ':' '/')
export APP_NAME=tritonserver
export MAJOR_VERSION=2.39
export MINOR_VERSION=2.39.0
export NGC_VERSION=23.10-py3
export MAJOR_VERSION=2.40
export MINOR_VERSION=2.40.0
export NGC_VERSION=23.11-py3

docker pull nvcr.io/nvidia/$APP_NAME:$NGC_VERSION

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

apiVersion: v1
appVersion: "2.39"
appVersion: "2.40"
description: Triton Inference Server
name: triton-inference-server
version: 2.39.0
version: 2.40.0
Original file line number Diff line number Diff line change
Expand Up @@ -32,13 +32,13 @@ tritonProtocol: HTTP
# HPA GPU utilization autoscaling target
HPATargetAverageValue: 85
modelRepositoryPath: gs://triton_sample_models/23_09
publishedVersion: '2.39.0'
publishedVersion: '2.40.0'
gcpMarketplace: true

image:
registry: gcr.io
repository: nvidia-ngc-public/tritonserver
tag: 23.10-py3
tag: 23.11-py3
pullPolicy: IfNotPresent
# modify the model repository here to match your GCP storage bucket
numGpus: 1
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
x-google-marketplace:
schemaVersion: v2
applicationApiVersion: v1beta1
publishedVersion: '2.39.0'
publishedVersion: '2.40.0'
publishedVersionMetadata:
releaseNote: >-
Initial release.
Expand Down
2 changes: 1 addition & 1 deletion deploy/gke-marketplace-app/server-deployer/schema.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
x-google-marketplace:
schemaVersion: v2
applicationApiVersion: v1beta1
publishedVersion: '2.39.0'
publishedVersion: '2.40.0'
publishedVersionMetadata:
releaseNote: >-
Initial release.
Expand Down
2 changes: 1 addition & 1 deletion deploy/gke-marketplace-app/trt-engine/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
```
docker run --gpus all -it --network host \
--shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 \
-v ~:/scripts nvcr.io/nvidia/tensorrt:23.10-py3
-v ~:/scripts nvcr.io/nvidia/tensorrt:23.11-py3
pip install onnx six torch tf2onnx tensorflow
Expand Down
2 changes: 1 addition & 1 deletion deploy/k8s-onprem/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ tags:
loadBalancing: true

image:
imageName: nvcr.io/nvidia/tritonserver:23.10-py3
imageName: nvcr.io/nvidia/tritonserver:23.11-py3
pullPolicy: IfNotPresent
modelRepositoryServer: < Replace with the IP Address of your file server >
modelRepositoryPath: /srv/models
Expand Down
6 changes: 3 additions & 3 deletions docs/customization_guide/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ $ ./build.py ... --repo-tag=common:<container tag> --repo-tag=core:<container ta

If you are building on a release branch then `<container tag>` will
default to the branch name. For example, if you are building on the
r23.10 branch, `<container tag>` will default to r23.10. If you are
r23.11 branch, `<container tag>` will default to r23.11. If you are
building on any other branch (including the *main* branch) then
`<container tag>` will default to "main". Therefore, you typically do
not need to provide `<container tag>` at all (nor the preceding
Expand Down Expand Up @@ -334,8 +334,8 @@ python build.py --cmake-dir=<path/to/repo>/build --build-dir=/tmp/citritonbuild
If you are building on *main* branch then '<container tag>' will
default to "main". If you are building on a release branch then
'<container tag>' will default to the branch name. For example, if you
are building on the r23.10 branch, '<container tag>' will default to
r23.10. Therefore, you typically do not need to provide '<container
are building on the r23.11 branch, '<container tag>' will default to
r23.11. Therefore, you typically do not need to provide '<container
tag>' at all (nor the preceding colon). You can use a different
'<container tag>' for a component to instead use the corresponding
branch/tag in the build. For example, if you have a branch called
Expand Down
14 changes: 7 additions & 7 deletions docs/customization_guide/compose.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,8 @@ from source to get more exact customization.
The `compose.py` script can be found in the [server repository](https://github.com/triton-inference-server/server).
Simply clone the repository and run `compose.py` to create a custom container.
Note: Created container version will depend on the branch that was cloned.
For example branch [r23.10](https://github.com/triton-inference-server/server/tree/r23.10)
should be used to create a image based on the NGC 23.10 Triton release.
For example branch [r23.11](https://github.com/triton-inference-server/server/tree/r23.11)
should be used to create a image based on the NGC 23.11 Triton release.

`compose.py` provides `--backend`, `--repoagent` options that allow you to
specify which backends and repository agents to include in the custom image.
Expand Down Expand Up @@ -76,19 +76,19 @@ For example, running
```
python3 compose.py --backend tensorflow1 --repoagent checksum
```
on branch [r23.10](https://github.com/triton-inference-server/server/tree/r23.10) pulls:
- `min` container `nvcr.io/nvidia/tritonserver:23.10-py3-min`
- `full` container `nvcr.io/nvidia/tritonserver:23.10-py3`
on branch [r23.11](https://github.com/triton-inference-server/server/tree/r23.11) pulls:
- `min` container `nvcr.io/nvidia/tritonserver:23.11-py3-min`
- `full` container `nvcr.io/nvidia/tritonserver:23.11-py3`

Alternatively, users can specify the version of Triton container to pull from any branch by either:
1. Adding flag `--container-version <container version>` to branch
```
python3 compose.py --backend tensorflow1 --repoagent checksum --container-version 23.10
python3 compose.py --backend tensorflow1 --repoagent checksum --container-version 23.11
```
2. Specifying `--image min,<min container image name> --image full,<full container image name>`.
The user is responsible for specifying compatible `min` and `full` containers.
```
python3 compose.py --backend tensorflow1 --repoagent checksum --image min,nvcr.io/nvidia/tritonserver:23.10-py3-min --image full,nvcr.io/nvidia/tritonserver:23.10-py3
python3 compose.py --backend tensorflow1 --repoagent checksum --image min,nvcr.io/nvidia/tritonserver:23.11-py3-min --image full,nvcr.io/nvidia/tritonserver:23.11-py3
```
Method 1 and 2 will result in the same composed container. Furthermore, `--image` flag overrides the `--container-version` flag when both are specified.

Expand Down
2 changes: 1 addition & 1 deletion docs/customization_guide/test.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ $ ./gen_qa_custom_ops
```

This will create multiple model repositories in /tmp/<version>/qa_*
(for example /tmp/23.10/qa_model_repository). The TensorRT models
(for example /tmp/23.11/qa_model_repository). The TensorRT models
will be created for the GPU on the system that CUDA considers device 0
(zero). If you have multiple GPUs on your system see the documentation
in the scripts for how to target a specific GPU.
Expand Down
6 changes: 3 additions & 3 deletions docs/user_guide/custom_operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ simple way to ensure you are using the correct version of TensorRT is
to use the [NGC TensorRT
container](https://ngc.nvidia.com/catalog/containers/nvidia:tensorrt)
corresponding to the Triton container. For example, if you are using
the 23.10 version of Triton, use the 23.10 version of the TensorRT
the 23.11 version of Triton, use the 23.11 version of the TensorRT
container.

## TensorFlow
Expand Down Expand Up @@ -123,7 +123,7 @@ simple way to ensure you are using the correct version of TensorFlow
is to use the [NGC TensorFlow
container](https://ngc.nvidia.com/catalog/containers/nvidia:tensorflow)
corresponding to the Triton container. For example, if you are using
the 23.10 version of Triton, use the 23.10 version of the TensorFlow
the 23.11 version of Triton, use the 23.11 version of the TensorFlow
container.
## PyTorch
Expand Down Expand Up @@ -167,7 +167,7 @@ simple way to ensure you are using the correct version of PyTorch is
to use the [NGC PyTorch
container](https://ngc.nvidia.com/catalog/containers/nvidia:pytorch)
corresponding to the Triton container. For example, if you are using
the 23.10 version of Triton, use the 23.10 version of the PyTorch
the 23.11 version of Triton, use the 23.11 version of the PyTorch
container.
## ONNX
Expand Down
4 changes: 2 additions & 2 deletions docs/user_guide/performance_tuning.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ with a `tritonserver` binary.

```bash
# Start server container
docker run -ti --rm --gpus=all --network=host -v $PWD:/mnt --name triton-server nvcr.io/nvidia/tritonserver:23.10-py3
docker run -ti --rm --gpus=all --network=host -v $PWD:/mnt --name triton-server nvcr.io/nvidia/tritonserver:23.11-py3

# Start serving your models
tritonserver --model-repository=/mnt/models
Expand Down Expand Up @@ -284,7 +284,7 @@ by setting the `-u` flag, such as `perf_analyzer -m densenet_onnx -u

```bash
# Start the SDK container interactively
docker run -ti --rm --gpus=all --network=host -v $PWD:/mnt --name triton-client nvcr.io/nvidia/tritonserver:23.10-py3-sdk
docker run -ti --rm --gpus=all --network=host -v $PWD:/mnt --name triton-client nvcr.io/nvidia/tritonserver:23.11-py3-sdk

# Benchmark model being served from step 3
perf_analyzer -m densenet_onnx --concurrency-range 1:4
Expand Down
2 changes: 2 additions & 0 deletions qa/L0_infer/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ if [ ! -z "$TEST_REPO_ARCH" ]; then
REPO_VERSION=${REPO_VERSION}_${TEST_REPO_ARCH}
fi

ldconfig || true

export CUDA_VISIBLE_DEVICES=0

TEST_RESULT_FILE='test_results.txt'
Expand Down
2 changes: 2 additions & 0 deletions qa/L0_sequence_batcher/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ TEST_RESULT_FILE='test_results.txt'

# Must run on a single device or else the TRITONSERVER_DELAY_SCHEDULER
# can fail when the requests are distributed to multiple devices.
ldconfig || true

export CUDA_VISIBLE_DEVICES=0

CLIENT_LOG="./client.log"
Expand Down
2 changes: 1 addition & 1 deletion qa/common/gen_jetson_trt_models
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@
# Make all generated files accessible outside of container
umask 0000
# Set the version of the models
TRITON_VERSION=${TRITON_VERSION:=23.10}
TRITON_VERSION=${TRITON_VERSION:=23.11}
# Set the CUDA device to use
CUDA_DEVICE=${RUNNER_ID:=0}
# Set TensorRT image
Expand Down
Loading

0 comments on commit b9f69b5

Please sign in to comment.