Add support for multiple datasets in the trainer. The datasets are in…

[Sopel97]

…terleaved at binpack chunk granularity.

Handling cyclic binpack reading was moved from BinpackSfenInputParallelStream to CompressedTrainingDataEntryParallelReader to allow for cycling each dataset individually.

train.py accepts any number of positional arguments - paths to the datasets to use. Validation will use the same datasets unless overriden by --validation-data, which can be present multiple times (or have multiple values) to specify multiple datasets.

easy_train.py now can have multiple instances of --training-dataset and --validation-dataset (or accept multiple values for each).

C API changed, additional helper functions were made to wrap conversion of string list to array of char* for create_fen_batch_stream and create_sparse_batch_stream.

Also kinda deprecate .bin, currently won't work with multiple datasets. No one was using it anyway, we should remove it (at least don't allow it in the trainer) next time.

---

I tested easy_train.py briefly, but I would like someone to do a full training run and compare the results with a manually interleaved dataset. The semantics should be similar, but there are some changes to the data loader that could manifest issues when re-reading datasets (requires a long training session), and also to make sure the interleaving is similar enough.

@linrock

[linrock]

getting this error running easy_train.py when --validation-datasets is not specified:

File "/root/easy-train-data/experiments/experiment_L1-2048-new-117G/nnue-pytorch/train.py", line 172, in <module>
main()
File "/root/easy-train-data/experiments/experiment_L1-2048-new-117G/nnue-pytorch/train.py", line 68, in main
args.validation_datasets = flatten_once(args.validation_datasets)
File "/root/easy-train-data/experiments/experiment_L1-2048-new-117G/nnue-pytorch/train.py", line 37, in flatten_once
return sum(lst, [])
TypeError: 'NoneType' object is not iterable

[Sopel97]

should be fixed now

[linrock]

how do you specify multiple training datasets with easy_train.py? if i use the same flag multiple times, only the last arg gets sent to train.py:

python3 easy_train.py \
  --training-dataset /data/UHO.e1fb1ade.binpack \
  --training-dataset /data/multinet_pv-2_diff-100_nodes-5000.binpack \
  --training-dataset /data/nodes5000pv2_UHO.binpack \
  ...

log output shows:

Training with: ['/data/nodes5000pv2_UHO.binpack', '/data/nodes5000pv2_UHO.binpack']
Validating with: ['/data/nodes5000pv2_UHO.binpack', '/data/nodes5000pv2_UHO.binpack']

[Sopel97]

how do you specify multiple training datasets with easy_train.py? if i use the same flag multiple times, only the last arg gets sent to train.py:

python3 easy_train.py \
  --training-dataset /data/UHO.e1fb1ade.binpack \
  --training-dataset /data/multinet_pv-2_diff-100_nodes-5000.binpack \
  --training-dataset /data/nodes5000pv2_UHO.binpack \
  ...

log output shows:

Training with: ['/data/nodes5000pv2_UHO.binpack', '/data/nodes5000pv2_UHO.binpack']
Validating with: ['/data/nodes5000pv2_UHO.binpack', '/data/nodes5000pv2_UHO.binpack']

Hmm. I'm unable to replicate this.

    --training-dataset=c:/dev/nnue-pytorch/noob_master_leaf_static_d12_85M_0.binpack ^
    --training-dataset=c:/dev/nnue-pytorch/d8_100000.binpack ^
    --training-dataset=c:/dev/nnue-pytorch/10m_d3_2.binpack ^
    --validation-dataset=c:/dev/nnue-pytorch/d8_100000.binpack ^

Training with: ['c:\\dev\\nnue-pytorch\\noob_master_leaf_static_d12_85M_0.binpack', 'c:\\dev\\nnue-pytorch\\d8_100000.binpack', 'c:\\dev\\nnue-pytorch\\10m_d3_2.binpack']
Validating with: ['c:\\dev\\nnue-pytorch\\d8_100000.binpack']

[linrock]

oops my bad, i forgot to update the branch from which i was invoking easy_train.py. i'm seeing the same results as you now. looks to be working as expected!

[linrock]

i started training runs with both: a manually interleaved binpack + using multiple training datasets as runtime flags. i'll report back later with how they compare

[linrock]

current status of trainings from scratch:

• L1-2048-new-117G - (blue) 2x runs with manually interleaved binpacks using interleave_binpacks.py
• L1-2048-new-117G-individual- (orange) 2x runs using this branch and several --training-dataset flags

117G total size of binpacks of different sizes, all compressed:

 38G nodes5000pv2_UHO.binpack
 26G multinet_pv-2_diff-100_nodes-5000.binpack
 21G UHO.e1fb1ade.binpack
9.7G dfrc-n5000.L1-2048.binpack
9.2G UHO_4060_v2.1.binpack
8.2G test60-2019-2tb7p.binpack
3.1G UHO.L1-2048.binpack
3.1G UHO-4060-v2.L1-2048.binpack

default LR and gamma. lambda 1.0 for the first 500 epochs, then end-lambda set to 0.75 at epoch 500.

[Sopel97]

I did a test with farseerT74.binpack, farseerT75.binpack, farseerT76.binpack. Results:

440 - interleaved in the data loader
441 - interleaved by interleave_binpacks

It looks like they are close to equivalent. I will, however remove the requirement for filesystem because it's unnecessary and causes issues on older gcc.

[vondele]

This looks ready to merge, OK?

[Sopel97]

yes, tests seem to indicate it works correctly