Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify EOS token handling in tokenize function #12866

Closed
wants to merge 1 commit into from
Closed

Simplify EOS token handling in tokenize function #12866

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 4 additions & 12 deletions docker/llm/finetune/lora/cpu/docker/lora_finetune.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,26 +137,18 @@ def train(
tokenizer.padding_side = "left" # Allow batched inference

def tokenize(prompt, add_eos_token=True):
# there's probably a way to do this with the tokenizer settings
# but again, gotta move fast
result = tokenizer(
prompt,
truncation=True,
max_length=cutoff_len,
padding=False,
return_tensors=None,
)
if (
result["input_ids"][-1] != tokenizer.eos_token_id
and len(result["input_ids"]) < cutoff_len
and add_eos_token
):
result["input_ids"].append(tokenizer.eos_token_id)
result["attention_mask"].append(1)
add_special_tokens=add_eos_token,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @DreadPirate09 ,

Thanks for your contribution. The code previously manually appends EOS token because add_special_token=True will add both a BOS and EOS at the beginning and end respectively, as illustrated here. Therefore, I think add_special_token does not exactly match add_eos_token's semantics.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok ... I understand now

)

result["labels"] = result["input_ids"].copy()
result["labels"] = result["input_ids"].copy()

return result
return result

def generate_and_tokenize_prompt(data_point):
full_prompt = generate_prompt(data_point)
Expand Down