Train without eos token? #990
-
|
How to train with ignore eos token? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 2 replies
-
|
Could you elaborate? |
Beta Was this translation helpful? Give feedback.
-
|
I am trying to train model without eos token attatched. |
Beta Was this translation helpful? Give feedback.
-
|
Hey, this would be an edge case. In such case, we support Bringing your own tokenized dataset. Please check readme but in essence, you’ll need to create your own prompt then tokenize it, and save as Dataset. Afterward, you can pass the dataset path. |
Beta Was this translation helpful? Give feedback.
Hey, this would be an edge case. In such case, we support Bringing your own tokenized dataset. Please check readme but in essence, you’ll need to create your own prompt then tokenize it, and save as Dataset.
Afterward, you can pass the dataset path.