Llmblog2 #16

KevinMusgrave · 2024-02-21T17:22:14Z

No description provided.

…ram with padding and truncation

…fields from config files

KevinMusgrave · 2024-02-23T20:44:27Z

The test_model.py script downloads the best checkpoint if exp_id is given, and this was something I was able to run on my laptop for TinyLlama. I also ran it for the pretrained Mistral model, but I don't want to run it for a finetuned Mistral model because I think the checkpoint will be huge.

Part of the problem is that the checkpoint contains the optimizer state, which I don't need to evaluate the model. Maybe there is some way to download only part of the checkpoint? I'm not sure.

blog/llm-finetuning-2/README.md

blog/llm-finetuning-2/deepspeed.yaml

blog/llm-finetuning-2/distributed.yaml

blog/llm-finetuning-2/finetune.py

Co-authored-by: Agnieszka Ciborowska <[email protected]>

aciborowska

LGTM!

aciborowska and others added 26 commits February 5, 2024 18:09

working deepspeed

c7bfdba

Trying out mistral

67a9654

Move some chat formatting logic to chat_format

047b607

minor changes

7a86d96

adding scrap.py

34ff373

get max_length

9ba7e6b

Get rid of max_length function

788b170

test with batch size

8608815

Test with a batch

a90a835

Add get_tokenize_fn

341542e

Added test_model back in

5ea0939

Add max_length back in

c614bbe

lora seems to work

b80fa15

include system prompt in user prompt for mistral

cbde3df

check if response is in decoded

96abc27

Use WarmupDecayLR

b20e987

Add lora flag to configs

2f2d38f

delete test_model, move into inference.py

0032b48

plot token histogram

bd45e48

plot for num tokens before response

7df117a

right-side padding and truncation. max_length 2048. Plot token histog…

ed845ce

…ram with padding and truncation

profiling

7696c20

scrap->validate_tokenizer. Remove profiler stuff. Remove unnecessary …

f700eda

…fields from config files

Separate folder for new blog post

e9329d5

Updated readme

70a8d06

Manually set title

1d0c924