FT optimization utility. Integrate with serialization.

[Sopel97]

This is a small rewrite of https://github.com/Ergodice/nnue-pytorch/tree/ftperm. Big thanks to @Ergodice for the great FT weight permutation optimization algorithm. I kept it as is, only reducing batch size in one place - needs revisiting later to clean up and potentially support cpu-only optimization.

ftperm.py can be used standalone to go through the optimization process step by step. serialize.py integration allows optionally including the full process (without intermediate files) to be performed during serialization. The optimization is NOT performed by default, as it takes very long and requires additional parameters.

Usage notes copied from notes in ftperm.py:

NOTE: This script uses CUDA and may requires large amounts of VRAM. Decrease --count if encountering problems.

Example use:

1. Generate the activation matrix for some sample dataset.

python ftperm.py gather --data=data\fishpack32.binpack --net=networks\nn-5af11540bbfe.nnue --count=1000000 --features=HalfKAv2_hm --out ftact1m.npy

python ftperm.py gather --data=noob_master_leaf_static_d12_85M_0.binpack --net=nn-5af11540bbfe.nnue --count=10000 --features=HalfKAv2_hm --out ftact1m.npy

2. Find a permutation

python ftperm.py find_perm --data=ftact1m.npy --out=ftact.perm

3. Test the permutation against the baseline

python ftperm.py eval_perm --data=ftact1m.npy --perm=ftact.perm

4. Apply permutation and save
python serialize.py nn-5af11540bbfe.nnue permuted.nnue --features=HalfKAv2_hm --ft_perm=ftact.perm

----------------------------------------------------------------

OR do the whole process in one step

python serialize.py networks\nn-5af11540bbfe.nnue permuted.nnue --features=HalfKAv2_hm --ft_optimize --ft_optimize_data=data\fishpack32.binpack --ft_optimize_count=1000000

python serialize.py nn-5af11540bbfe.nnue permuted.nnue --features=HalfKAv2_hm --ft_optimize --ft_optimize_data=noob_master_leaf_static_d12_85M_0.binpack --ft_optimize_count=10000

After this is merged the following things should be looked at:

1. How many data points we actually need. I got good results with 10000, 1M maybe excessive.
2. Find out ideal batch size in get_score_change. Should be smallest possible as it affects VRAM usage. I reduced it to 10000 already. With 1GB of VRAM limit I had to reduce it to 100, didn't notice any issues.
3. Figure out if make_swaps_3 can be optimized with cupy. It takes by far the longest.
4. Add a cpu-only mode for VRAM limited systems.

[Sopel97]

I have improved the readability and maintainability of the algorithm code. I also made sure it works both with cupy and numpy. Initially I was wrong, cupy was being used for make_swaps_3, it was numpy that wasn't.

@Ergodice could you please check if the comments I added make sense? I wasn't sure in all places.

I also believe that make_swaps_3 could be made faster by reusing old max_values and only updating it along the changed blocks (as this is quite sparse given the size of it). But this is for another time to expore.

[Ergodice]

Very elegant refactor!

A couple things:
On lines 95 and 159 the shape of actmat is not (N, L1) but (N, L1 // 2).
The zeroing out on lines 265-268 is not done to prevent redundancy but because the entries are only computed correctly when they are all from different blocks.

[vondele]

this seems to have conflicts, and unaddressed review comments?

[Sopel97]

yes, I'll address them later

[Sopel97]

Resolved conflics and addressed the comments.