Erratic eval @CCC

[mstembera]

Describe the issue

In game https://www.chess.com/computer-chess-championship#event=ccc21-blitz-semifinals&game=337 starting on move 88 the eval suddenly jumps and stays erratic till move 106. Given only 7 pieces and the pre and post evals this seems erroneous.

Expected behavior

Eval isn't erratic.

Steps to reproduce

See https://www.chess.com/computer-chess-championship#event=ccc21-blitz-semifinals&game=337 starting on move 88.

Anything else?

No response

Operating system

All

Stockfish version

dev-20230729-65ece7d9

[Clausable]

Here is another example: https://www.chess.com/computer-chess-championship#event=ccc21-rapid-alt-finals&game=27

[reggie1337]

I don't think the eval is behaving unexpectedly since white stays generally positive with two minor pieces in play vs blacks one, the eval bar seems relatively consistent based off moves made.

[cmwetherell]

The evaluation function/network is not perfect. Maybe positions like this could be compiled with the "known" answer, i.e., it is a draw and the data added to the training data for the nets. I doubt this is a "bug" that can/should be targeted aside from improving training data.

If you view the position in Lichess for SF 16, it gives a more reasonable eval: https://lichess.org/analysis/3bk3/8/6p1/3PK3/1B6/7P/8/8_b_-_-_0_1

[cj5716]

positions like this could be compiled with the "known" answer, i.e., it is a draw and the data added to the training data for the nets

I highly doubt that will work given that the nets are trained on over 100GB of data (in compressed form)

[peregrineshahin]

it's been a long long time since this happened.