High performance API

[niansa]

Hey!

I'd love to see this project being able to be used through some TCP socket with a very optimized protocol. One it may make use of something like protobuf, or even grpc.
I think everyone agrees HTTP would be a complete overkill specially for a project focused on high performance. 😆

Thanks
Niansa

[LostRuins]

I don't agree that HTTP is overkill. I think it is perfect. You have RESTful APIs for all the large LLM providers like OpenAI. You send your inputs and generation params, the service sends you back the generated response string.

Here's an example of serving llama.cpp over HTTP, as an emulated KoboldAI server.
https://github.com/LostRuins/llamacpp-for-kobold

[tarruda]

Try out the tcp_server branch, more details of how it works in #278

[niansa]

You have RESTful APIs for all the large LLM providers like OpenAI. You send your inputs and generation params, the service sends you back the generated response string.

Sure but none of these are performance oriented like this project and none of these run on the local PC.

Here's an example of serving llama.cpp over HTTP, as an emulated KoboldAI server.

If done correctly building a REST API on top of THAT would be trivial, even from Python if performance isn't much of a point anyways.

Also supporting HTTP would require lots of code or even external dependencies.

[anzz1]

Yes, keep the core lean, portable, fast and free of dependencies while having the option of building things on top of it, as modules. This could be achieved with a C api which simply uses file streams in place of stdin/stdout. A communication layer based on those file streams could then be built upon to achieve whatever functionality necessary. The tcp_server branch is actually almost like that, but not quite.

[LostRuins]

Also supporting HTTP would require lots of code or even external dependencies.

My distributable is a 700kb zip containing literally 3 files, one of which is optional. It requires nothing else except a stock python install to run.

https://github.com/LostRuins/llamacpp-for-kobold/releases/tag/v1.0.3

and if you don't want that, we have a literal 1 file pyinstaller solution with a built-in server.

https://github.com/henk717/llamacpp-for-kobold/releases/tag/v1.0.2-2048

[0x090909]

@tarruda This is great, is it going to be merged?

[tarruda]

@tarruda This is great, is it going to be merged?

I don' t think so, check the discussion in #278 for more context.

[Calandiel]

I don't agree that HTTP is overkill. I think it is perfect. You have RESTful APIs for all the large LLM providers like OpenAI. You send your inputs and generation params, the service sends you back the generated response string.

Here's an example of serving llama.cpp over HTTP, as an emulated KoboldAI server. https://github.com/LostRuins/llamacpp-for-kobold

No idea what your specialization is, but in my part of the programming world TCP is already considered too heavy ^^

A TCP based API would be very much welcome over restful APIs from my point of view.