Integrating Mistral.rs with Swiftide

[timonv]

Hey there,

Figured I'd give it another shot to integrate mistral.rs with Swiftide. Tested several text models and it works great. Kudos on the new v4 builder style api, it's very clean 👍

However, github actions is now running into limitations and kills the worker with a 143 (afaik this is usually due to high cpu).

I've been looking at cargo timings and tried some tuning, but no luck so far. Does mistral.rs have to do tuning to be able to run in GA? I couldn't find anything specific. The full PR is at bosun-ai/swiftide#386

Side note: Would it be possible to publish mistralrs as a crate?

[timonv]

@EricLBuehler With a bit of tuning builds are working reliably and the integration is pretty cool bosun-ai/swiftide#421. However, crates.io does not allow publishing crates that rely on unpublished crates. Is there a reason mistral.rs is not published on crates.io? release-plz makes the process rather painless.

[EricLBuehler]

Hi @timonv! Just released v0.3.4 which integrates some nice changes. We have a fork of Candle which I don't want to publish as it'll conflict with the base Candle package. Otherwise, I would absolutely do so.

[timonv]

Hey @EricLBuehler, thanks for reaching out. After some tinkering, I got it to work fine in a pipeline as well.

Besides having erics-candle, is there no way to up stream the changes or extend? I'm not familiar with the candle internals 🤷

Otherwise, I'll probably go for a separate swiftide-mistrals that users can add as a git dependency instead. It's hard to gauche the traction, it got asked for multiple times, but then people stopped asking. So when/if really depends on if that resumes again.

6x is insane! Nice job!