-
Notifications
You must be signed in to change notification settings - Fork 132
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ability to modify provider options at runtime #1048
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add a code sample in the PR for the scenario that is driving this change ie to set the default provider? Did we lose the SetDefaultProvider() API?
The initial 'SetDefaultProvider' proposed API would rearrange the providers so the one you mention is first in the list. This isn't really what we want since it doesn't match the onnxruntime API for setting providers. With onnxruntime you want just the provider you want to use, unless you're doing something with multiple providers at the same time. Where should the sample go? This is how it will look:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approving for the C/C++ API changes.
Now a provider can be chosen and configured purely at runtime, to override what the genai_config.json specifies. ` auto config = OgaConfig::Create("path\to\model"); config->SetProviderOption("cuda", "enable_quantum_drive", "1"); auto model = OgaModel::Create(*config); ` vs this if no runtime changes are needed (this is also the current API): ` auto model = OgaModel::Create("path\to\model"); ` This will solve the following use cases: * User wants to change a runtime option for a provider already specified in the genai_config.json * User wants to switch which provider is used vs what the genai_config.json specifies * The genai_config.json specifies no provider, and one is completely chosen at runtime. The OgaConfig provides these methods: * ClearProviders() - Remove all providers currently specified (will default to cpu if nothing else happens) * SetProvider("provider_name") - If provider is already exists, does nothing, otherwise adds to end of provider list * SetProviderOption("provider_name", "option_name", "option_value") - If provider doesn't exist, adds it to end of list. Then sets an option on the provider.
Cherry picks: #1059 #1049 #1057 #1048 #1047 #1020 --------- Co-authored-by: kunal-vaishnavi <[email protected]> Co-authored-by: vortex-captain <[email protected]> Co-authored-by: Yi Ren <[email protected]>
Now a provider can be chosen and configured purely at runtime, to override what the genai_config.json specifies.
auto config = OgaConfig::Create("path\to\model"); config->SetProviderOption("cuda", "enable_quantum_drive", "1"); auto model = OgaModel::Create(*config);
vs this if no runtime changes are needed (this is also the current API):
auto model = OgaModel::Create("path\to\model");
This will solve the following use cases:
The OgaConfig provides these methods: