[RFC] [WIP] Dual net NNUE

[mstembera]

This allows SF to use two nets of different sizes. The intention is that the second net will be smaller/faster and used to lazy evaluate positions w/ high scores.

I need help w/ two things:

1. I commented out the net download section of the Makefile because it doesn't handle multiple nets. We need this before we can start testing on fishtest. Could someone w/ the right skills please fix it?
2. The current small net is just a compressed version of the last L1=1024 wide master net. This is way too big but even so local tests are close to elo neutral. Could someone please train a very small net? Maybe L1=256 L2=15 L3=16? This net could also be trained on mostly(only) positions with high scores to make it even better.

Per @vondele this idea has been around so credit to whoever thought of it first. Thanks in advance for any help and feedback.

bench: 1449578

[linrock]

nn-ecb35f70ff2a.nnue is a L1=256 L2=15 L3=32 net from a 2-stage training that may be worth trying:
https://tests.stockfishchess.org/api/nn/nn-ecb35f70ff2a.nnue

About -67 elo vs master at the time:
https://tests.stockfishchess.org/tests/view/64b6b769dc56e1650abab4fc

And +60 elo vs. a previous small net:
https://tests.stockfishchess.org/tests/view/64b6b6abdc56e1650abab4e8

[linrock]

after this works on fishtest, i can help with training stronger L1-256 nets for testing

[mstembera]

@linrock ok "patch -i" did the job. https://tests.stockfishchess.org/tests/view/656c0f90f6bf9a79b891d587 started. Thanks!

If you or anyone want to run any tests please don't wait for me. Just fork and go ahead. The two places that will likely need some tweaking along w/ the nets are 1) in evaluate.cpp line 188 where we decide which net to use 2) in evaluate_nnue.cpp around line 179 we decide which feature transformer to update.

[linrock]

Results of the LTC above with L1-256 small net:
https://tests.stockfishchess.org/tests/view/656c0f90f6bf9a79b891d587

Elo: -1.29 ± 1.6 (95%) LOS: 6.2%
Total: 40000 W: 9760 L: 9908 D: 20332
Ptnml(0-2): 18, 4627, 10862, 4471, 22
nElo: -2.67 ± 3.4 (95%) PairsRatio: 0.97

I started a few STCs:
https://tests.stockfishchess.org/tests/view/656d1abb6980e15f69c74cec (baseline)
https://tests.stockfishchess.org/tests/view/656d278c6980e15f69c74e3d (hint threshold reduced to 2000)
https://tests.stockfishchess.org/tests/view/656d2fdc6980e15f69c74ef6 (closer match between hint+lazy thresholds)

[cj5716]

I wonder if it is possible to make the NNUE choice dependent solely on root eval instead (using the big net) to decide the net we use during search so that we save on redundant hinting/accumulator updates

[mstembera]

@cj5716 Seems possible but maybe by the time the root has a high enough eval the game has already been decided? I don't mean to discourage you from trying it.

[mstembera]

@linrock I checked an currently the fastest net we can have is L1=128. (Because the avx512 implementation doesn't support anything smaller.) What do you think about the idea of training one of these exclusively on positions with high simple_eval() similar to what it will be used to evaluate?

[vdbergh]

I guess the purpose of the small net would be to recognize positions where the actual eval is substantially different from the simple eval. This should be reflected in the training data.

[linrock]

sure, i'll prepare L1-128 and a couple other small architectures. and then train with filtered datasets with only high simple eval positions. i'll post results later

[mstembera]

Another idea I just had is that w/ the small net in place maybe we can let the default net be much bigger. Say L1=4096. I don't know if training such a net is actually feasible in practice?

[Dboingue]

is there any where we can check back on the training loop scheme, this is modifying.. I thought I would jump right where it gets interesting. not interested in fight with dispersed documentation about the big picture training loop.

remaining questions.. is there still a non-NN moderate search depth from training input position for some master network? sub-question: is that some derived version of the classical evaluation function? or still the same function with same parameter choices (I had heard or read it had been frozen from SF12 on). Also, maybe that has changed in past since SF16 came out, but SF12 blog statement about that basic ML story of input training data and output training data going in the training process, was only vaguely modified in SF16 blog to say that leela's data was now used in training. It sounded mysterious, and I tried to ask. and gave up. here is me trying again, where such question should be fresh in mind, since we are here talking about something high level.

I understand notions of master NN, and transformers.. What I seem to need that this conversation and its backreferenced issue does not weed out, in my imagination (and ignorance of certain things possibly now easy to find, but I have been burned before, so asking right where it should be on the mind of people), is what is the evaluation function that provide an a priori high material imbalance (as defined in SF, not per Silman, AFAIU).

Can I keep assuming that blog SF12 not specificially changed explicitely by blog SF16 (leela's data!), is still what is determining the training target of the master networks.

Otherwise. I find it nice that ensemble of NN are being considered. I wonder what is the long term plan for how to keep sorting evaluation work by different hand-crafted functions versus hand-crafted input features NNue of various sizes. hand-crafted is not necessarily a bad thing, when done with method, I suggest.

so where is simple eval defined or explained not in code.. issues are fine.. one level of abstraction from code can be induce from discussoins. As I might have done some here.. Thanks you anyone who took the time to read this. sorry I have gotten lazy on the documentation hunting.. I ran out of juice a while back.

[linrock]

Another idea I just had is that w/ the small net in place maybe we can let the default net be much bigger. Say L1=4096. I don't know if training such a net is actually feasible in practice?

it's possible a larger default net would work.

Here are some trained larger nets that can be tried later:

• L1-3072: https://tests.stockfishchess.org/tests/view/6553020f136acbc57353555a
• L1-2560 L2-31: https://tests.stockfishchess.org/tests/view/6551b9fe136acbc573533a55

Training L1-4096 would take a while. In practice, maybe if L1-3072 can work with dual nnue, then larger L1 will be worth a closer look. Larger net sizes haven't been doing well recently with a single default net.

[cj5716]

Another idea I just had is that w/ the small net in place maybe we can let the default net be much bigger. Say L1=4096. I don't know if training such a net is actually feasible in practice?

it's possible a larger default net would work.

Here are some trained larger nets that can be tried later:

• L1-3072: https://tests.stockfishchess.org/tests/view/6553020f136acbc57353555a
• L1-2560 L2-31: https://tests.stockfishchess.org/tests/view/6551b9fe136acbc573533a55

Training L1-4096 would take a while. In practice, maybe if L1-3072 can work with dual nnue, then larger L1 will be worth a closer look. Larger net sizes haven't been doing well recently with a single default net.

In the same vein I believe we can also try those same nets but with lower lazy threshold

[linrock]

@mstembera this dual nnue variant passed both stc and ltc:
https://tests.stockfishchess.org/tests/view/6574cb9d95ea6ba1fcd49e3b
https://tests.stockfishchess.org/tests/view/6575da2d4d789acf40aaac6e

do you want to update this PR or should we open a new PR with details of the changes? the main differences are:

• L1-128 net trained only on positions with high simple eval
• constant lazy thresholds based on simple eval: return simple eval > 2300, smallnet > 1100

[mstembera]

@linrock Congrats! It may be cleaner to open a new PR. BTW, I pushed a couple of small changes to fix sanitizer issues but there are a couple of other checks still failing for reasons I don't understand. I can compile and run successfully using GCC, Clang, and MSVC.

[linrock]

Sounds good, i'll open a new PR later and take a closer look at the failing CI checks

[github-actions]

clang-format 17 needs to be run on this PR.
If you do not have clang-format installed, the maintainer will run it when merging.
For the exact version please see https://packages.ubuntu.com/mantic/clang-format-17.

(execution 7218251317 / attempt 1)

[mstembera]

Closing in favor of #4915