splitter is a tiny tool to split a single-file dataset into several distinct parts, typically train, test, and dev.
splitter requires Python 3, and the click library.
splitter -d dataset train:80 test:10 dev:10
This creates three new files: dataset.train, containing roughly 80% of the
lines from dataset, and dataset.test/dataset.dev, each containing about 10%
of the lines. Because this setting is so common, there's a shortcut:
splitter -d dataset --standard-splits
You can make this even shorter by using the shorthand -s, so one could also
just write
splitter -sd dataset
All the three above examples are equivalent.
The names "test", "dev", and "train" are not special in any way (apart from
being used by --standard-splits); you can use any names. You are also not
limited to having three parts, you can specify two, or seven, or even one part
(although that doesn't make much sense).
The numbers have to sum up to 100. Because most datasets can't exactly be split as per the split specification, we give extra examples to (one of) the smallest dataset(s).
If your files aren't seperated by newlines but by double newlines (i.e. empty
lines), you can switch the --mode to paragraph (the default being line):
splitter -sd dataset --mode paragraph
Instead of paragraph, you can use P, instead of line L.
By default, examples are shuffled before being divided into parts. To disable
that, use the switch --no-shuffle:
splitter -sd dataset --no-shuffle
Finally, when in paragraph mode, there's an option to --trim-empty-lines,
which will cause the output to remove the empty lines seperating examples.