Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to modify provider options at runtime #1048

Merged
merged 15 commits into from
Nov 9, 2024

Conversation

RyanUnderhill
Copy link
Member

Now a provider can be chosen and configured purely at runtime, to override what the genai_config.json specifies.
auto config = OgaConfig::Create("path\to\model"); config->SetProviderOption("cuda", "enable_quantum_drive", "1"); auto model = OgaModel::Create(*config);
vs this if no runtime changes are needed (this is also the current API):
auto model = OgaModel::Create("path\to\model");

This will solve the following use cases:

  • User wants to change a runtime option for a provider already specified in the genai_config.json
  • User wants to switch which provider is used vs what the genai_config.json specifies
  • The genai_config.json specifies no provider, and one is completely chosen at runtime.

The OgaConfig provides these methods:

  • ClearProviders() - Remove all providers currently specified (will default to cpu if nothing else happens)
  • SetProvider("provider_name") - If provider is already exists, does nothing, otherwise adds to end of provider list
  • SetProviderOption("provider_name", "option_name", "option_value") - If provider doesn't exist, adds it to end of list. Then sets an option on the provider.

@RyanUnderhill RyanUnderhill marked this pull request as ready for review November 8, 2024 19:05
src/ort_genai_c.h Outdated Show resolved Hide resolved
src/ort_genai_c.h Outdated Show resolved Hide resolved
test/python/test_onnxruntime_genai_api.py Outdated Show resolved Hide resolved
test/python/test_onnxruntime_genai_api.py Show resolved Hide resolved
test/c_api_tests.cpp Show resolved Hide resolved
src/python/python.cpp Show resolved Hide resolved
src/python/python.cpp Show resolved Hide resolved
src/models/debugging.cpp Show resolved Hide resolved
src/csharp/Model.cs Show resolved Hide resolved
@natke natke self-requested a review November 8, 2024 21:08
Copy link
Contributor

@natke natke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a code sample in the PR for the scenario that is driving this change ie to set the default provider? Did we lose the SetDefaultProvider() API?

@RyanUnderhill
Copy link
Member Author

RyanUnderhill commented Nov 8, 2024

Can you add a code sample in the PR for the scenario that is driving this change ie to set the default provider? Did we lose the SetDefaultProvider() API?

The initial 'SetDefaultProvider' proposed API would rearrange the providers so the one you mention is first in the list. This isn't really what we want since it doesn't match the onnxruntime API for setting providers. With onnxruntime you want just the provider you want to use, unless you're doing something with multiple providers at the same time.

Where should the sample go? This is how it will look:

auto config = OgaConfig::Create("path\to\model");
config.ClearProviders(); // Only needed if config already specifies a provider, but safe to do in both cases
config.SetProvider("dml");
auto model = OgaModel::Create(*config);

Copy link
Contributor

@pranavsharma pranavsharma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving for the C/C++ API changes.

@RyanUnderhill RyanUnderhill merged commit f66e4f5 into main Nov 9, 2024
12 checks passed
@RyanUnderhill RyanUnderhill deleted the ryanunderhill/model_config branch November 9, 2024 03:26
RyanUnderhill added a commit that referenced this pull request Nov 12, 2024
Now a provider can be chosen and configured purely at runtime, to
override what the genai_config.json specifies.
`
  auto config = OgaConfig::Create("path\to\model");
  config->SetProviderOption("cuda", "enable_quantum_drive", "1");
  auto model = OgaModel::Create(*config);
`
vs this if no runtime changes are needed (this is also the current API):
`
  auto model = OgaModel::Create("path\to\model");
`

This will solve the following use cases:
* User wants to change a runtime option for a provider already specified
in the genai_config.json
* User wants to switch which provider is used vs what the
genai_config.json specifies
* The genai_config.json specifies no provider, and one is completely
chosen at runtime.

The OgaConfig provides these methods:
* ClearProviders() - Remove all providers currently specified (will
default to cpu if nothing else happens)
* SetProvider("provider_name") - If provider is already exists, does
nothing, otherwise adds to end of provider list
* SetProviderOption("provider_name", "option_name", "option_value") - If
provider doesn't exist, adds it to end of list. Then sets an option on
the provider.
RyanUnderhill added a commit that referenced this pull request Nov 12, 2024
Cherry picks:
#1059 
#1049 
#1057
#1048
#1047
#1020

---------

Co-authored-by: kunal-vaishnavi <[email protected]>
Co-authored-by: vortex-captain <[email protected]>
Co-authored-by: Yi Ren <[email protected]>
@RyanUnderhill RyanUnderhill restored the ryanunderhill/model_config branch November 20, 2024 03:45
@RyanUnderhill RyanUnderhill deleted the ryanunderhill/model_config branch November 20, 2024 03:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants