Introduce llama-run

[ericcurtin]

It's like simple-chat but it uses smart pointers to avoid manual
memory cleanups. Less memory leaks in the code now. Avoid printing
multiple dots. Split code into smaller functions. Uses no exception
handling.

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[ericcurtin]

Some of these builds seem hardcoded to c++11, when we use a feature from c++14.

Any reason we aren't using say c++17

Any reasonable platform should be up-to date with c++17 I think

[ericcurtin]

Converted to C++11 only

[ericcurtin]

@slaren @ggerganov PTAL, I'm hoping to add other features to this example such as, read prompt from '-p' arg and read prompt from stdin

[slaren]

It would be good to have a more elaborated chat example, but the goal of this example is to show in the simplest way possible how to use the llama.cpp API. I don't think that these changes achieve that, I think that users will have a harder time understanding the llama.cpp API with all the extra boilerplate that is being added here.

If you want to use this as the base of a new example that adds more features that would be great, but I think we should keep this example as simple as possible.

[ericcurtin]

Cool sounds good @slaren, mind if I call the new example ramalama-core ?

[ericcurtin]

llama-inferencer? I really don't care too much about the name, just want to agree on one to unblock this PR.

@rhatdan @slp any suggestions on names? I plan on using this as the main program during "ramalama run" but I'm happy for anyone to use it or make changes to it to suit their needs. It's like a drastically simplified version of llama-cli, with one or two additional features, read from stdin, read from -p flag.

But it does seem stabler and less error prone than llama-cli also. And the verbose info is all cleaned up to only spit out errors. It was based on llama-simple-chat initially.

[slaren]

Maybe something like llama-chat. I mentioned before that I think it would be good to have a example focused on chat only, that does that very well, that in time could replace the current llama-cli as the main program of llama.cpp, which at this is point is basically unmaintainable and should be retired.

[ericcurtin]

SGTM

[ericcurtin]

It's also tempting to call this something like run and use a kinda RamaLama, LocalAI, Ollama type CLI interface to interact with models. Kinda like daemonless Ollama:

llama-run file://somedir/somefile.gguf

[ericcurtin]

We could even possibly add https://, http://, ollama://, hf:// as valid syntaxes to pull models since they all are just a http pull in the end

[ericcurtin]

That might be implemented as llama-pull that llama-run can fork/exec (or they share a common library)

[ericcurtin]

I will be afk for 3 weeks, so expect inactivity in this PR. I did the rename in case we want to merge as is and not leave this go stale.

Although the syntax will completely change to:

llama-run [file://]somedir/somefile.gguf [prompt] [flags]

file:// will be optional, but will set up the possibility of adding the pullers discussed above.

[ericcurtin]

This drives the compiler crazy FWIW:

diff --git a/include/llama.h b/include/llama.h
index 5e742642..c3285da3 100644
--- a/include/llama.h
+++ b/include/llama.h
@@ -537,6 +537,13 @@ extern "C" {
                          int32_t   il_start,
                          int32_t   il_end);

+#ifdef __cplusplus
+    // Smart pointers
+    typedef std::unique_ptr<llama_model, decltype(&llama_free_model)> llama_model_ptr;
+    typedef std::unique_ptr<llama_context, decltype(&llama_free)> llama_context_ptr;
+    typedef std::unique_ptr<llama_sampler, decltype(&llama_sampler_free)> llama_sampler;
+#endif
+
     //
     // KV cache
     //

seems like it only wants to build C code in that file or something.

[slaren]

You are probably doing something wrong. Put it at the end of the file, include <memory>, and rename the llama_sampler ptr to llama_sampler_ptr. You should also structs (function objects) as the deleter, see ggml-cpp.h for an example.

[ericcurtin]

Added the smart pointers typedef stuff @slaren

[slaren]

The rest look good, but this one (char_array_ptr) does not belong here.

[ericcurtin]

I also need to use more parameters as references, I notice I still pass raw pointers in places where it could be a reference to a smart pointer

[ericcurtin]

I removed char_array_ptr

[slaren]

Looks like a good start for an improved chat example.

llama-cpp.h also needs to be added to the list of public headers in CMakeLists.txt.

[ericcurtin]

Added as public header

[ericcurtin]

Happy to merge this @slaren @ggerganov

Although there will be breaking changes in a follow on PR, want to change the syntax to this (ramalama/ollama style):

Description:
  Runs a llm

Usage:
  llama-run [options] MODEL [PROMPT]

Options:
  -c, --context-size <value>
      Context size (default: 2048)
  -n, --ngl <value>
      Number of GPU layers (default: 0)
  -h, --help
      Show help message

Examples:
  llama-run ~/.local/share/ramalama/models/ollama/smollm\:135m
  llama-run --ngl 99 ~/.local/share/ramalama/models/ollama/smollm\:135m
  llama-run --ngl 99 ~/.local/share/ramalama/models/ollama/smollm\:135m Hello World

Eventually want to add optional prefixes for the model also (like file://, hf://, maybe we implement http://, https://, ollama:// eventually also, its not that hard), but for no prefix we will just assume the string is a file path I guess.

[slaren]

Sounds good. Btw, to support dynamic loading of backends (#10469), you should add a call to ggml_backend_load_all at the beginning. The goal is to allow projects to distribute a single binary rather than needing different versions for each backend, which I imagine would be relevant to a project like ramalama.