-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add transformers-musicgen backend #1387
feat: add transformers-musicgen backend #1387
Conversation
✅ Deploy Preview for localai canceled.
|
Signed-off-by: Dave <[email protected]>
@dave-gray101 somehow, you seem to hit again #1356 ( see also rhasspy/piper-phonemize#22 ). I'm re-triggering as by rhasspy/espeak-ng#1 (comment) seems the maintainer pinned the hash. I'd prefer we could control the hash somehow, but seems still it's not possible as for now. |
ok - seems the maintainer's last commit broke it again and affects master too. We need to pin this somehow, even if with workarounds |
this looks great @dave-gray101 ! could we also update the docs to have a small example along with this backend? |
Will do - but for my sanity I'm going to add that in a follow up PR after this one passes CI EDIT: build failed, adding some preliminary docs anyway. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fantastic!
…1.0 by renovate (#16284) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [quay.io/go-skynet/local-ai](https://togithub.com/mudler/LocalAI) | minor | `v2.0.0-cublas-cuda11` -> `v2.1.0-cublas-cuda11` | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>mudler/LocalAI (quay.io/go-skynet/local-ai)</summary> ### [`v2.1.0`](https://togithub.com/mudler/LocalAI/releases/tag/v2.1.0) [Compare Source](https://togithub.com/mudler/LocalAI/compare/v2.0.0...v2.1.0) <!-- Release notes generated using configuration in .github/release.yml at master --> ##### What's Changed ##### Breaking Changes 🛠 - feat(alias): alias llama to llama-cpp, update docs by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1448](https://togithub.com/mudler/LocalAI/pull/1448) ##### Bug fixes 🐛 - fix(piper): pin petals, phonemize and espeak by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1393](https://togithub.com/mudler/LocalAI/pull/1393) - update(llama.cpp): update server, correctly propagate LLAMA_VERSION by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1440](https://togithub.com/mudler/LocalAI/pull/1440) ##### Exciting New Features 🎉 - Added Check API KEYs file to API.go by [@​lunamidori5](https://togithub.com/lunamidori5) in [https://github.com/mudler/LocalAI/pull/1381](https://togithub.com/mudler/LocalAI/pull/1381) - exllama(v2): fix exllamav1, add exllamav2 by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1384](https://togithub.com/mudler/LocalAI/pull/1384) - Fix: API Key / JSON Fast Follow [#​1](https://togithub.com/mudler/LocalAI/issues/1) by [@​dave-gray101](https://togithub.com/dave-gray101) in [https://github.com/mudler/LocalAI/pull/1388](https://togithub.com/mudler/LocalAI/pull/1388) - feat: add transformers-musicgen backend by [@​dave-gray101](https://togithub.com/dave-gray101) in [https://github.com/mudler/LocalAI/pull/1387](https://togithub.com/mudler/LocalAI/pull/1387) - feat(diffusers): update, add autopipeline, controlnet by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1432](https://togithub.com/mudler/LocalAI/pull/1432) - feat(img2vid,txt2vid): Initial support for img2vid,txt2vid by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1442](https://togithub.com/mudler/LocalAI/pull/1442) ##### 👒 Dependencies - ⬆️ Update ggerganov/whisper.cpp by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1378](https://togithub.com/mudler/LocalAI/pull/1378) - ⬆️ Update ggerganov/llama.cpp by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1379](https://togithub.com/mudler/LocalAI/pull/1379) - ⬆️ Update ggerganov/whisper.cpp by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1430](https://togithub.com/mudler/LocalAI/pull/1430) - ⬆️ Update mudler/go-piper by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1441](https://togithub.com/mudler/LocalAI/pull/1441) - ⬆️ Update ggerganov/whisper.cpp by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1434](https://togithub.com/mudler/LocalAI/pull/1434) ##### Other Changes - ⬆️ Update ggerganov/llama.cpp by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1385](https://togithub.com/mudler/LocalAI/pull/1385) - docs: site update fixing old image text / How To update updating GPU and CPU docker pages by [@​lunamidori5](https://togithub.com/lunamidori5) in [https://github.com/mudler/LocalAI/pull/1399](https://togithub.com/mudler/LocalAI/pull/1399) - feat: cuda transformers by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1401](https://togithub.com/mudler/LocalAI/pull/1401) - feat(entrypoint): optionally prepare extra endpoints by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1405](https://togithub.com/mudler/LocalAI/pull/1405) - ⬆️ Update ggerganov/whisper.cpp by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1390](https://togithub.com/mudler/LocalAI/pull/1390) - ⬆️ Update mudler/go-piper by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1400](https://togithub.com/mudler/LocalAI/pull/1400) - tests: add diffusers tests by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1419](https://togithub.com/mudler/LocalAI/pull/1419) - ⬆️ Update ggerganov/whisper.cpp by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1418](https://togithub.com/mudler/LocalAI/pull/1418) - How To Updates / Model Used Switched / Removed "docker-compose" (RIP) by [@​lunamidori5](https://togithub.com/lunamidori5) in [https://github.com/mudler/LocalAI/pull/1417](https://togithub.com/mudler/LocalAI/pull/1417) - fix(transformers\*): add sentence-transformers and transformers-musicgen tests, fix musicgen wrapper by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1420](https://togithub.com/mudler/LocalAI/pull/1420) - extras: add vllm,bark,vall-e-x tests, bump diffusers by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1422](https://togithub.com/mudler/LocalAI/pull/1422) - Documentation for Hipblas by [@​sfxworks](https://togithub.com/sfxworks) in [https://github.com/mudler/LocalAI/pull/1425](https://togithub.com/mudler/LocalAI/pull/1425) - ⬆️ Update ggerganov/llama.cpp by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1391](https://togithub.com/mudler/LocalAI/pull/1391) - docs: add aikit to integrations by [@​sozercan](https://togithub.com/sozercan) in [https://github.com/mudler/LocalAI/pull/1412](https://togithub.com/mudler/LocalAI/pull/1412) - ⬆️ Update ggerganov/llama.cpp by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1429](https://togithub.com/mudler/LocalAI/pull/1429) - docs(mixtral): add mixtral example by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1449](https://togithub.com/mudler/LocalAI/pull/1449) ##### New Contributors - [@​sozercan](https://togithub.com/sozercan) made their first contribution in [https://github.com/mudler/LocalAI/pull/1412](https://togithub.com/mudler/LocalAI/pull/1412) **Full Changelog**: mudler/LocalAI@v2.0.0...v2.1.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 10pm on monday" in timezone Europe/Amsterdam, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4xMDIuMCIsInVwZGF0ZWRJblZlciI6IjM3LjEwMi4wIiwidGFyZ2V0QnJhbmNoIjoibWFzdGVyIn0=-->
…1.0@f0b3afa by renovate (#16304) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [quay.io/go-skynet/local-ai](https://togithub.com/mudler/LocalAI) | minor | `v2.0.0-cublas-cuda12-ffmpeg` -> `v2.1.0-cublas-cuda12-ffmpeg` | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>mudler/LocalAI (quay.io/go-skynet/local-ai)</summary> ### [`v2.1.0`](https://togithub.com/mudler/LocalAI/releases/tag/v2.1.0) [Compare Source](https://togithub.com/mudler/LocalAI/compare/v2.0.0...v2.1.0) <!-- Release notes generated using configuration in .github/release.yml at master --> #### What's Changed ##### Breaking Changes 🛠 - feat(alias): alias llama to llama-cpp, update docs by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1448](https://togithub.com/mudler/LocalAI/pull/1448) ##### Bug fixes 🐛 - fix(piper): pin petals, phonemize and espeak by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1393](https://togithub.com/mudler/LocalAI/pull/1393) - update(llama.cpp): update server, correctly propagate LLAMA_VERSION by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1440](https://togithub.com/mudler/LocalAI/pull/1440) ##### Exciting New Features 🎉 - Added Check API KEYs file to API.go by [@​lunamidori5](https://togithub.com/lunamidori5) in [https://github.com/mudler/LocalAI/pull/1381](https://togithub.com/mudler/LocalAI/pull/1381) - exllama(v2): fix exllamav1, add exllamav2 by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1384](https://togithub.com/mudler/LocalAI/pull/1384) - Fix: API Key / JSON Fast Follow [#​1](https://togithub.com/mudler/LocalAI/issues/1) by [@​dave-gray101](https://togithub.com/dave-gray101) in [https://github.com/mudler/LocalAI/pull/1388](https://togithub.com/mudler/LocalAI/pull/1388) - feat: add transformers-musicgen backend by [@​dave-gray101](https://togithub.com/dave-gray101) in [https://github.com/mudler/LocalAI/pull/1387](https://togithub.com/mudler/LocalAI/pull/1387) - feat(diffusers): update, add autopipeline, controlnet by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1432](https://togithub.com/mudler/LocalAI/pull/1432) - feat(img2vid,txt2vid): Initial support for img2vid,txt2vid by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1442](https://togithub.com/mudler/LocalAI/pull/1442) ##### 👒 Dependencies - ⬆️ Update ggerganov/whisper.cpp by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1378](https://togithub.com/mudler/LocalAI/pull/1378) - ⬆️ Update ggerganov/llama.cpp by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1379](https://togithub.com/mudler/LocalAI/pull/1379) - ⬆️ Update ggerganov/whisper.cpp by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1430](https://togithub.com/mudler/LocalAI/pull/1430) - ⬆️ Update mudler/go-piper by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1441](https://togithub.com/mudler/LocalAI/pull/1441) - ⬆️ Update ggerganov/whisper.cpp by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1434](https://togithub.com/mudler/LocalAI/pull/1434) ##### Other Changes - ⬆️ Update ggerganov/llama.cpp by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1385](https://togithub.com/mudler/LocalAI/pull/1385) - docs: site update fixing old image text / How To update updating GPU and CPU docker pages by [@​lunamidori5](https://togithub.com/lunamidori5) in [https://github.com/mudler/LocalAI/pull/1399](https://togithub.com/mudler/LocalAI/pull/1399) - feat: cuda transformers by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1401](https://togithub.com/mudler/LocalAI/pull/1401) - feat(entrypoint): optionally prepare extra endpoints by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1405](https://togithub.com/mudler/LocalAI/pull/1405) - ⬆️ Update ggerganov/whisper.cpp by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1390](https://togithub.com/mudler/LocalAI/pull/1390) - ⬆️ Update mudler/go-piper by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1400](https://togithub.com/mudler/LocalAI/pull/1400) - tests: add diffusers tests by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1419](https://togithub.com/mudler/LocalAI/pull/1419) - ⬆️ Update ggerganov/whisper.cpp by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1418](https://togithub.com/mudler/LocalAI/pull/1418) - How To Updates / Model Used Switched / Removed "docker-compose" (RIP) by [@​lunamidori5](https://togithub.com/lunamidori5) in [https://github.com/mudler/LocalAI/pull/1417](https://togithub.com/mudler/LocalAI/pull/1417) - fix(transformers\*): add sentence-transformers and transformers-musicgen tests, fix musicgen wrapper by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1420](https://togithub.com/mudler/LocalAI/pull/1420) - extras: add vllm,bark,vall-e-x tests, bump diffusers by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1422](https://togithub.com/mudler/LocalAI/pull/1422) - Documentation for Hipblas by [@​sfxworks](https://togithub.com/sfxworks) in [https://github.com/mudler/LocalAI/pull/1425](https://togithub.com/mudler/LocalAI/pull/1425) - ⬆️ Update ggerganov/llama.cpp by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1391](https://togithub.com/mudler/LocalAI/pull/1391) - docs: add aikit to integrations by [@​sozercan](https://togithub.com/sozercan) in [https://github.com/mudler/LocalAI/pull/1412](https://togithub.com/mudler/LocalAI/pull/1412) - ⬆️ Update ggerganov/llama.cpp by [@​localai-bot](https://togithub.com/localai-bot) in [https://github.com/mudler/LocalAI/pull/1429](https://togithub.com/mudler/LocalAI/pull/1429) - docs(mixtral): add mixtral example by [@​mudler](https://togithub.com/mudler) in [https://github.com/mudler/LocalAI/pull/1449](https://togithub.com/mudler/LocalAI/pull/1449) #### New Contributors - [@​sozercan](https://togithub.com/sozercan) made their first contribution in [https://github.com/mudler/LocalAI/pull/1412](https://togithub.com/mudler/LocalAI/pull/1412) **Full Changelog**: mudler/LocalAI@v2.0.0...v2.1.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 10pm on monday" in timezone Europe/Amsterdam, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4xMDIuMCIsInVwZGF0ZWRJblZlciI6IjM3LjEwMi4wIiwidGFyZ2V0QnJhbmNoIjoibWFzdGVyIn0=-->
This PR adds preliminary support for the
transformers-musicgen
models - LocalAI's first text-to-music generation backend!transformers-musicgen is extremely closely related to the existing transformers backend. In order to keep that one simple (I think its a great teaching example), I've duplicated and modified it into its own external backend.
Currently, transformers-musicgen is implemented as a TTS backend - which comes with some pros and cons. The primary advantage is simplicity - the LocalAI code for TTS already works! The primary disadvantage is that the TTS request model is not a perfect fit - in particular, the highly important "Duration" parameter to control how long the generated audio doesn't really have a good analog within our simple TTSRequest model.
I wanted to create this PR as a "first stage" to allow users to start testing the backend out, before determining if it's better to extend the TTS side of LocalAI to be a little more flexible, or if it's worth creating a dedicated music generation endpoint with a unique request model - the latter being more complex and time consuming, but would probably allow some of the cooler audio-conditioned generation to be implemented!