Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the triton inference server backend "turbomind_backend" #1986

Merged
merged 15 commits into from
Jul 17, 2024
Merged

Remove the triton inference server backend "turbomind_backend" #1986

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
e460b60
remove cutlass to BUILD_TEST
lvhan028 Jul 10, 2024
cb5b6d2
remove turbomind_backend
lvhan028 Jul 10, 2024
4fbd39e
get_hf_config_content
lvhan028 Jul 10, 2024
0d520fb
remove profile_serving.py and libfastertransformer
lvhan028 Jul 11, 2024
a7eac26
remove lmdeploy/serve/turbomind/triton_models
lvhan028 Jul 11, 2024
eae2264
remove chatbot.py
lvhan028 Jul 11, 2024
afa0f08
remove triton_server_backend
lvhan028 Jul 16, 2024
20d7c47
remove triton_client CLI
lvhan028 Jul 16, 2024
e14962f
merge main
lvhan028 Jul 16, 2024
b1ae7a5
fix
lvhan028 Jul 17, 2024
e1ba6de
remove triton_utils.hpp
lvhan028 Jul 17, 2024
e6c99ac
fix
lvhan028 Jul 17, 2024
2cb9a85
fix typo
lvhan028 Jul 17, 2024
9a66b2c
update manifest.in
lvhan028 Jul 17, 2024
8d319e2
fix as suggested
lvhan028 Jul 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 12 additions & 11 deletions CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,20 +43,21 @@ option(BUILD_PY_FFI "Build python ffi" ON)
option(BUILD_TEST "Build tests" OFF)

include(FetchContent)
if (BUILD_TEST)
FetchContent_Declare(
repo-cutlass
GIT_REPOSITORY https://github.com/NVIDIA/cutlass.git
GIT_TAG 6f47420213f757831fae65c686aa471749fa8d60
GIT_SHALLOW ON
)

FetchContent_Declare(
repo-cutlass
GIT_REPOSITORY https://github.com/NVIDIA/cutlass.git
GIT_TAG 6f47420213f757831fae65c686aa471749fa8d60
GIT_SHALLOW ON
)

set(CUTLASS_ENABLE_HEADERS_ONLY ON CACHE BOOL "Enable only the header library")
set(CUTLASS_ENABLE_HEADERS_ONLY ON CACHE BOOL "Enable only the header library")

FetchContent_MakeAvailable(repo-cutlass)
FetchContent_MakeAvailable(repo-cutlass)

set(CUTLASS_HEADER_DIR ${PROJECT_SOURCE_DIR}/3rdparty/cutlass/include)
set(CUTLASS_EXTENSIONS_DIR ${PROJECT_SOURCE_DIR}/src/turbomind/cutlass_extensions/include)
set(CUTLASS_HEADER_DIR ${PROJECT_SOURCE_DIR}/3rdparty/cutlass/include)
set(CUTLASS_EXTENSIONS_DIR ${PROJECT_SOURCE_DIR}/src/turbomind/cutlass_extensions/include)
endif()

option(SPARSITY_SUPPORT "Build project with Ampere sparsity feature support" OFF)

Expand Down
260 changes: 0 additions & 260 deletions benchmark/profile_serving.py
View file
Open in desktop

This file was deleted.

58 changes: 0 additions & 58 deletions docs/en/benchmark/profile_triton_server.md
View file
Open in desktop

This file was deleted.

1 change: 0 additions & 1 deletion docs/en/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,6 @@ Documentation
benchmark/profile_generation.md
benchmark/profile_throughput.md
benchmark/profile_api_server.md
benchmark/profile_triton_server.md
benchmark/evaluate_with_opencompass.md

.. _supported_models:
Expand Down
2 changes: 0 additions & 2 deletions docs/en/inference/turbomind.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,6 @@ Our implementation of the LLaMa family models is modified from Gpt-NeoX model in

TurboMind supports a Python API that enables streaming output and tensor parallel mode.

The ability to use [tritonserver](https://github.com/triton-inference-server/server) for serving is also inherited from FasterTransformer. However, to support submitting concurrent requests into our persistent batch model, we no longer use sequence batching or dynamic batching as FasterTransformer does. The bookkeeping of request and sequence states are managed by TurboMind instead.

## Difference between FasterTransformer and TurboMind

Apart of the features described above, there are still many minor differences that we don't cover in this document. Notably, many capabilities of FT are dropped in TurboMind because of the difference in objectives (e.g. prefix prompt, beam search, context embedding, sparse GEMM, GPT/T5/other model families, etc)
Expand Down
Loading
Loading