Ability encode files to tokens separately from fine-tuning?

[bob80333]

I have a dataset of ~160MB fine-tuning in google colab, but a dataset of ~180MB causes the runtime to crash while loading the dataset, due to using all available RAM. However, while fine-tuning, I noticed that the VRAM has 6GB available, and the RAM has ~10GB available.

My dataset was originally many smaller files that are combined, if I could encode each of these separately into tokens, then combine the encoded dataset, and skip the encoding process when loading the dataset, I think I could use larger datasets while avoiding running out of RAM.

I did notice that on line 27 of load_dataset.py it seems to be able to load pre-encoded files.

[bob80333]

Update: I was able to use use nsheppard's encode.py to encode a 544MB dataset into a tokenized 160MB dataset, which loaded in seconds on Google colab, and would have previously crashed it. The tokenization process did use > 20GB of RAM on my computer, and read over 37GB from disk, so I think there is probably some room for optimization there.

However, I am currently successfully training a dataset that was ~544MB with 137968501 tokens, that previously crashed google colab.

[minimaxir]

Huh, TIL. I wasn't sure what encode.py was in that repo; pre-encoding makes sense! Thanks for the technical breakdown!

I should probably port that to this repo.

[bob80333]

Yeah, I can't claim full credit for that, I read about it in this blog post on retraining gpt-2 for poetry, it had some interesting ideas for improving output like beam search or tree search instead of just a greedy search. I did find a tensorflow beam search decoder, but I couldn't find any information on how to use it.

[minimaxir]

Added with gpt2.encode_dataset().

[saippuakauppias]

maybe someone will try to use https://github.com/Blosc/bcolz ?