Our mission is to provide open-source tools that enable the effective deployment and use of LLMs and ML models in production settings for developers and MLOps/LLMOps. Our key projects include:
- Paddler: An open-source load balancer and reverse proxy optimized for servers running llama.cpp. Paddler ensures efficient request distribution with a stateful load balancer that monitors server slots and health, supporting dynamic scaling.
- LLMOps Handbook: (work in progress) Practical and advanced guide to LLMOps. It provides a solid understanding of large language models’ general concepts, deployment techniques, and software engineering practices.
Join our community on Discord to collaborate and stay updated!
