Document using an already install whisper.cpp

[edmundmiller]

I'd love to be able to use the whisper that's already installed, for example the one that's in nixpkgs. https://github.com/NixOS/nixpkgs/blob/nixos-23.05/pkgs/tools/audio/openai-whisper-cpp/default.nix#L51

The installation works great though!

[edmundmiller]

Ah never mind! Support is already there I just couldn't find it in the README?

whisper.el/whisper.el

Lines 111 to 126 in 2d541ac

	(defcustom whisper-install-whispercpp t
	"Specify whether to install whisper.cpp automatically.
	By default whisper.el compiles whisper.cpp automatically. But if you are on a
	platform where our automatic whisper.cpp install doesn't work but you are able
	to do so manually, you can set this to `manual' to skip our try (and failure)
	to install it automatically. Note that in case a functional install is found
	at `whisper-install-directory', we can still do model download, quantization
	automatically.
	But if you are planning to use something other than whisper.cpp entirely, as
	such don't want to install it nor run checks for it, you may opt out of
	whisper.cpp as a whole by setting this to nil. In that case it's your
	responsibility to override `whisper-command' with appropriate function."
	:type '(choice boolean (const manual))
	:group 'whisper)

[natrys]

Actually it is documented in the README (although I will take this as an unwitting proof of the fact that the README had grown too bloated due to too many variables).

In any case, would whisper.el even work with the nix package? iirc nix store paths are read only, whereas whisper.el basically presumes that downloaded models are in the relative models/ directory. I see that they have patched the download script to put models in current directory instead. Other issue is they renamed the binary from main to whisper-cpp.

You would basically have to override whisper-command function to accommodate these two things if you want to use nixpkgs version with whisper.el.

[edmundmiller]

To clarify, I read the part about using a different tool, but not an already installed whisper.

It has a fix where it installs to the local directory. This would need to install to somewhere that whisper.el could find it, so it's probably more pain than it's worth 🤷🏻

[natrys]

but not an already installed whisper.

That part is also handled by the same variable, by setting it to 'manual. But it's really for people who are using things like Window, where Emacs can't compile whisper for various reasons, but they manually can. It still assumes they compile it in whisper.cpp subdir in whisper-install-directory, and assumes Emacs can run shell commands to download models using upstream script. Neither of these apply here, however. The fact that nix changes binary name and patches download script basically makes it as alien as a different tool to whisper.el so it's the other part of the doc about different tool that becomes relevant here.

I could maybe add couple more variables like whispercpp binary name, or custom model directory just for the nix package, but that imo feels like more pain than worth, though might become necessary in future if whisper.cpp is packaged by more distros and people want to use that. In any case if you want to just use the nixpkgs version, it's not that much extra effort today (though you would lose the automatic download and quantization feature of whisper.el).

Assuming you are at ~/models/ and used the nix script to download model like:

$ whisper-cpp-download-ggml-model base

The whisper.el configuration would be:

(setq whisper-install-whispercpp nil)

(defun whisper--nix-command (input-file)
  `("whisper-cpp"
    "--model" ,(expand-file-name (concat "~/models/" "ggml-" whisper-model ".bin"))
    ,@(when whisper-use-threads (list "--threads" (number-to-string whisper-use-threads)))
    ,@(when whisper-translate '("--translate"))
    ,@(when whisper-show-progress-in-mode-line '("--print-progress"))
    "--language" ,whisper-language
    "--no-timestamps"
    "--file" ,input-file))

(advice-add 'whisper-command :override #'whisper--nix-command)

I pushed a commit earlier to recognise "whisper-cpp" from nix as indeed whisper.cpp from name so that you can still see transcription progress in modeline which is whisper.cpp specific feature. But that's probably all I want to do for now, aside from adding the recipe here in wiki and mention it in README somewhere.

[gtusr]

Instructions for how to configure whisper.el after installing the pre-compiled version of whisper.cpp package available from MSYS2 would be helpful for Windows users.

I set whisper-install-whispercpp to nil and whisper-install-directory to the empty string and custom configured whisper-command to point to the binary that MSYS2 installed. But I'm still getting an error: "Set a suitable value for whisper--ffmpeg-input-format."

From looking at the code, I'm not sure how Windows is supposed to get this value since the pcase sets it to nil if format is not pulse, which under Windows it is not. It is dshow.

Describing how to tell whisper.el which model to use and where to find it would also be helpful when using the MSYS2 package for whisper.cpp since the default location (models folder under installation root folder) is no longer relevant.

[natrys]

But I'm still getting an error: "Set a suitable value for whisper--ffmpeg-input-format."

Hi @gtusr, I think we mentioned that this needs to be manually set to whatever is the audio device, unless you are on Linux. I have basically not used Windows in over a decade so don't really know what goes here. The trick with pulse works in Linux because it abstracts away the underlying hardware, otherwise this needs to be the code/id of your specific hardware which is different for everybody.

If the ffmpeg from msys2 or whatever you use supports this, then the following command may show what are the appropriate devices you have:

ffmpeg -list_devices true -f dshow -i dummy

(according to this you may need to add audio= prefix to your device name)

Describing how to tell whisper.el which model to use and where to find it would also be helpful when using the MSYS2 package for whisper.cpp since the default location (models folder under installation root folder) is no longer relevant.

I think the above comment I wrote regarding Nix still applies here. Assuming you manually downloaded the models and put it in ~/models/, I don't see why you shouldn't be able to use that code.

Although since whisper.cpp compilation doesn't require more than a compiler, have you tried just compiling it yourself in msys? Then you can set whisper-install-directory to wherever whisper.cpp is cloned and whisper-install-whispercpp to 'manual.

[gtusr]

ffmpeg -list_devices true -f dshow -i dummy

This command tells me that the microphone is "Microphone Array (2- Realtek High Definition Audio(SST))" but exits by reporting the following error: "Error opening input: Immediate exit requested. Error opening input file dummy." Not sure if this error is at all relevant since file dummy does not exist.

In any event, whisper.el now reports that it is recording audio after I run whisper-run. But invoking whisper-run again does not stop the recording or start the transcribing. Only if I execute whisper-run twice in quick succession can I get it to stop recording. In no case can I get any transcription.

I know that the underlying whisper.cpp from MSYS2 is working because I can transcribe the jfk.wav sample correctly from the command line.

I also explicitly added the full pathname for the model to the --model parameter to no avail.

[natrys]

Not sure if this error is at all relevant since file dummy does not exist.

Indeed the error isn't relevant, you just need the microphone name from it. I presume you tried setting whisper--ffmpeg-input-format to audio=Microphone Array (2- Realtek High Definition Audio(SST))? Maybe also try only the name minus the audio= prefix.

Anyway sounds like ffmpeg is failing for some reason. Unfortunately error reporting is not great here. If you could try instruction from this comment, maybe you can get logs about exactly why ffmpeg is failing.