Vicuna problem

[zhound420]

Has anyone got this model to work yet? Running into this:

OSError: anon8231489123/vicuna-13b-GPTQ-4bit-128g does not appear to have a file named pytorch_model-00001-of-00003.bin. Checkout 'https://huggingface.co/anon8231489123/vicuna-13b-GPTQ-4bit-128g/main' for available files.

[jota2rz]

I don't think basaran supports GPTQ pre-quantized models.
https://github.com/oobabooga/text-generation-webui supports this model.
Documentation at https://github.com/oobabooga/text-generation-webui/wiki/GPTQ-models-(4-bit-mode)

Feature request? 👀

[peakji]

Basaran should work with Vicuna models. The model repo seems to contain outdated configs that point to non-existing weight files: https://huggingface.co/anon8231489123/vicuna-13b-GPTQ-4bit-128g/discussions/15

Also, you may want to install safetensors, as the repo only provides weights in safetensors format.

[jota2rz]

Basaran should work with Vicuna models.

Do you know how to make it work?

I get this error.

ValueError: Couldn't instantiate the backend tokenizer from one of:
(1) a `tokenizers` library serialization file,
(2) a slow tokenizer instance to convert or
(3) an equivalent slow tokenizer class to instantiate and convert.
You need to have sentencepiece installed to convert a slow tokenizer to a fast one.

[jota2rz]

Oops, I forgot to install the extra dependencies since I was inside a venv.
I needed transformers, sentenpiece and safetensors.
pip install safetensors transformers[sentencepiece]

Works good!

[fardeon]

We will add safetensors support in the next release: #174 #175

[zhound420]

Oops, I forgot to install the extra dependencies since I was inside a venv. I needed transformers, sentenpiece and safetensors. pip install safetensors transformers[sentencepiece]

Works good!

Hey I'd appreciate it if you could help me out running this model with basaran. Maybe point me to the right huggingface repository? Thanks.

[jota2rz]

@zhound420 https://rentry.org/nur779

[zhound420]

@zhound420 https://rentry.org/nur779

Thank you, you rock.

[karfly]

@fardeon @peakji

Hi, guys!
In the end, I did not understand whether GPTQ 4bit models are supported or not?

[karfly]

@zhound420 did you manage to run GPTQ model?

[zhound420]

@karfly no I did not yet. I'll have to come back to it in a couple days.

[karfly]

@zhound420 looking forward to hear from you!

[advaitdeshmukh]

@karfly no I did not yet. I'll have to come back to it in a couple days.

Did you manage to do it? Kinda stuck on the same.

Edit: I'm trying to use it with docker image(1st option)