Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initialize local CPU server if remote server not available #12

Closed
kpedro88 opened this issue Sep 14, 2020 · 3 comments
Closed

Initialize local CPU server if remote server not available #12

kpedro88 opened this issue Sep 14, 2020 · 3 comments

Comments

@kpedro88
Copy link
Collaborator

kpedro88 commented Sep 14, 2020

Initializing a local CPU server when the requested remote server is not available can serve as a fallback option. Open questions:

  1. At what stage should checking availability and initializing the local server take place? (python config, C++ module initialization, ....)
  2. How to control # threads used by the local server? see Control number of threads used by CPU server triton-inference-server/server#2018

If a local CPU server is used, SonicTriton clients should be forced to use Sync mode, to prevent contention between CMSSW and the local server (otherwise they would both try to use the same threads).

Assigned to: @kpedro88

@kpedro88
Copy link
Collaborator Author

See cms-sw/cmssw@master...kpedro88:TritonService for a functioning draft version of TritonService with automatic fallback support. Notes:

  • currently based on CMSSW_11_2_0_pre9
  • still potentially a work in progress, so the branch may be updated at any moment - don't get attached
  • all other ongoing SONIC-related PRs and branches will have to be rebased on this once it is merged (some API changes), but let's not worry about that yet

@kpedro88
Copy link
Collaborator Author

draft PR now submitted: cms-sw/cmssw#32576

@kpedro88
Copy link
Collaborator Author

minor followup: cms-sw/cmssw#32861

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant