Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for response sender in the default mode #364

Merged
merged 7 commits into from
Jun 6, 2024
Merged

Add support for response sender in the default mode #364

merged 7 commits into from
Jun 6, 2024

Conversation

kthui
Copy link
Contributor

@kthui kthui commented Jun 1, 2024 •
edited
Loading

Related PR: triton-inference-server/server#7311

Add support for non-decoupled models to use response sender. Removed the non-decoupled data pipeline.

Items to be done on another ticket:

  • Re-implement restart functionality to the decoupled data pipeline. [This functionality is never mentioned on documentation, so it is ok to skip it for now]

Logistics:

  • All commits on this PR came from previously approved PRs (less copyright/documentation updates), the purpose of this PR is to merge the feature into the main branch.

kthui added 4 commits May 31, 2024 15:11
@kthui
Add response sender to non-decoupled models and unify data pipelines (#…
01ba273 
…360)

* Add response sender to non-decoupled model and unify data pipelines

* Rename variable and class name
@kthui
Fix decoupled batch statistics to account for implicit batch size (#361)
4551e04
@kthui
Fix decoupled gpu output error handling (#362)
9f2865d 
* Fix decoupled gpu output error handling

* Return full error string upon exception from model
@kthui
Response sender to check for improper non-decoupled model usage (#363)
4961e24 
* Response sender to check for improper non-decoupled model usage

* Force close response sender on exception

* Rename functions
@kthui kthui mentioned this pull request Jun 4, 2024
Merged
@kthui kthui marked this pull request as ready for review June 4, 2024 19:07
krishung5
krishung5 reviewed Jun 5, 2024
View reviewed changes
src/python_be.cc Outdated Show resolved Hide resolved
@kthui kthui requested review from krishung5 and Tabrizian June 5, 2024 01:28
@kthui kthui merged commit 27f04d1 into main Jun 6, 2024
3 checks passed
@kthui kthui deleted the jacky-res-sender-main branch June 6, 2024 00:54
@krishung5 krishung5 mentioned this pull request Oct 4, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@krishung5 krishung5 krishung5 approved these changes

@Tabrizian Tabrizian Tabrizian approved these changes

@tanmayv25 tanmayv25 Awaiting requested review from tanmayv25

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kthui @Tabrizian @krishung5