Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include jemalloc into the documentation #5760

Merged
merged 1 commit into from
May 10, 2023
Merged

Include jemalloc into the documentation #5760

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 23 additions & 5 deletions docs/user_guide/model_management.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<!--
# Copyright 2018-2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# Copyright 2018-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
Expand Down Expand Up @@ -81,16 +81,34 @@ If you are seeing some memory growth when using the [model control
protocol](../protocol/extension_model_repository.md) for loading and unloading
models, it is possible that it's not an actual memory leak but some system's
malloc heuristics that causes memory to be unable to be released back to the OS
right away. You can try to switch from malloc to tcmalloc for better memory
performance by setting `LD_PRELOAD` as below when running Triton:
right away. To improve memory performance, you can consider switching from
malloc to [tcmalloc](https://github.com/google/tcmalloc) or
[jemalloc](https://github.com/jemalloc/jemalloc) by setting the `LD_PRELOAD`
environment variable when running Triton, as shown below:
```
# Using tcmalloc
LD_PRELOAD=/usr/lib/$(uname -m)-linux-gnu/libtcmalloc.so.4:${LD_PRELOAD} tritonserver --model-repository=/models ...
```
The tcmalloc library is already installed within Triton container. You can also
install tcmalloc using
```
# Using jemalloc
LD_PRELOAD=/usr/lib/$(uname -m)-linux-gnu/libjemalloc.so:${LD_PRELOAD} tritonserver --model-repository=/models ...
```
We recommend experimenting with both tcmalloc and jemalloc to determine which
one works better for your use case, as they have different strategies for
memory allocation and deallocation and may perform differently depending on the
workload.

Both tcmalloc and jemalloc libraries are already installed within the Triton
container. However, if you need to install them, you can do so using the
following commands:
```
# Install tcmalloc
apt-get install gperf libgoogle-perftools-dev
```
```
# Install jemalloc
apt-get install libjemalloc-dev
```

## Model Control Mode POLL

Expand Down