Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add named models support on AI workers #215

Closed
Angelmmiguel opened this issue Sep 13, 2023 · 0 comments · Fixed by #224
Closed

Add named models support on AI workers #215

Angelmmiguel opened this issue Sep 13, 2023 · 0 comments · Fixed by #224
Assignees
Labels
🚀 enhancement New feature or request
Milestone

Comments

@Angelmmiguel
Copy link
Contributor

Is your feature request related to a problem? Please describe.

Currently, workers that run inference must retrieve the ML model in memory to pass it back to the host. This limits the number of models we can run as Wasm has a memory limit of 4Gb (32-bit).

Describe the solution you'd like

The WASI-NN proposal recently introduced the "Named Models" feature. This feature allows the host to preload a set of models and expose them to the modules using a label. The model no longer need to load the model in memory. It only needs to reference the model using the right label.

wws can take advantage of this feature by adding new configuration parameters to the feature.wasi_nn object. Then, wws will preload those models and expose them to the workers.

The wasmtime-wasi-nn and wasi-nn (see example) crates already supports this feature.

Describe alternatives you've considered

No response

Additional context

No response

@Angelmmiguel Angelmmiguel added the 🚀 enhancement New feature or request label Sep 13, 2023
@Angelmmiguel Angelmmiguel added this to the v1.6.0 milestone Sep 13, 2023
@Angelmmiguel Angelmmiguel self-assigned this Sep 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🚀 enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant