Tensor Tools

tensor-tools pre-compiled for Ubuntu (x86) so you don't have to compile the rust in order to use the scripts.

The code comes from the HuggingFace/Candle-Core tensor-tools example.

Usage

Init HF Repo

Script requires new repo and ssh setup (see below).

The model name in this script will become your output directory for following scripts.

bash init-repo.sh

Create Repo (on HuggingFace)

https://hf.co/new

SSH Key Setup

# On your system
ssh-keygen -t ed25519 -C "comment here"

Add SSH public key to user settings

bin2safetensors

If you have .bin (pickle) files first you need to convert them to safetensors in order to quantize.

This requires python:

# Dependencies
sudo apt install python3 python3-pip python-is-python3
pip install torch safetensors numpy
# Run script
python scripts/bin2safetensors.py

Quantization

If you have .safetensors files.

To make one of each quantization type use:

bash scripts/make-all.sh

To make individual quantization types follow this format:

./tensor-tools quantize --quantization <quant_type> \
  <list of .safetensors files> \
  --out-file ./Candle_model_<quant_type>.gguf

Links

Specification

Candle Types