JORA: Jax library for distributed LLM fine-tuning

[aniquetahir]

We created this library for distributed fine-tuning of Llama-based LLMs (more planned to be added). We reduce the lines of code for simple fine-tuning significantly compared to HuggingFace and we also observe 12x improvement in performance compared to HuggingFace/Deepspeed. Memory consumption per GPU is also reduced. Check it out here:
https://github.com/aniquetahir/JORA