Add --set-high-watermark and --remove-high-watermark to watermark-repair subcommand

[siladu]

Final part of #696

Usage: web3signer eth2 watermark-repair [-hV] [--remove-high-watermark]
                                        [--set-high-watermark]
                                        [--epoch=<epoch>] [--slot=<slot>]
                                        [COMMAND]
Updates the slashing protection low watermark or high watermark for all
validators. This will not move the low watermark lower, the low watermark can
only be increased.If setting the high watermark, care should be taken to set
this to a future epoch and slot.
      --epoch=<epoch>        Low watermark to set the attestation source and
                               target to. (Sets high watermark epoch when
                               --set-high-watermark=true).
  -h, --help                 Show this help message and exit.
      --remove-high-watermark
                             Removes high watermark. When set to true, all
                               other subcommand options are ignored. (Default:
                               false)
      --set-high-watermark   Sets high watermark to given epoch and slot. (Sets
                               low watermark when --set-high-watermark=false).
                               (Default: false)
      --slot=<slot>          Low watermark to set the block slot to. (Sets high
                               watermark slot when --set-high-watermark=true).
  -V, --version              Print version information and exit.

Set high watermark:
web3signer --config-file=config.yaml eth2 watermark-repair --epoch=1 --slot=3 --set-high-watermark

Remove high watermark:
web3signer --config-file=config.yaml eth2 watermark-repair --remove-high-watermark

Set low watermark remains as:
web3signer --config-file=config.yaml eth2 watermark-repair --epoch=1 --slot=3
but this is also valid:
web3signer --config-file=config.yaml eth2 watermark-repair --epoch=1 --slot=3 --set-high-watermark=false

Documentation

• I thought about documentation and added the doc-change-required label to this PR if updates are required.

Changelog

• I thought about adding a changelog entry, and added one if I deemed necessary.

Testing

• I thought about testing these changes in a realistic/non-local environment.

[siladu]

Note, if the low watermark conflicts with the high watermark, this fails fast (on the first validator). This might seem like different behaviour when compared to the slashing import, which fails per validator.

However, since we're updating all validators to the same low watermark, they should all fail anyway so this is fine.

[siladu]

Carried out some end to end performance testing and can see negligible difference to loading, signing and importing when using high watermark feature. I have not tested pruning.

I did several runs and am showing the worst case from each.

1. Importing slashing data: 1 block + 1 attestation for 100K validators

time ./build/install/web3signer/bin/web3signer --config-file=tmp/config.yaml eth2 import --from=100_000_validators_interchange.json

- CONTROL (web3signer 23.9.0)
50.52s user 25.49s system 70% cpu 1:47.17 total

- PR
50.57s user 26.03s system 74% cpu 1:42.74 total

2. Update low watermark for 100K validators

$ time ./build/install/web3signer/bin/web3signer --config-file=tmp/config.yaml eth2 watermark-repair --epoch=100000 --slot=100000

- CONTROL (web3signer 23.9.0)
7.69s user 4.88s system 55% cpu 22.463 total

- PR with high watermark set
7.70s user 5.11s system 55% cpu 23.134 total

3. ./gradlew gatlingRun

- CONTROL (web3signer 23.9.0)

---- Global Information --------------------------------------------------------
> request count                                       6000 (OK=6000   KO=0     )
> min response time                                      2 (OK=2      KO=-     )
> max response time                                    154 (OK=154    KO=-     )
> mean response time                                     5 (OK=5      KO=-     )
> std deviation                                          6 (OK=6      KO=-     )
> response time 50th percentile                          4 (OK=4      KO=-     )
> response time 75th percentile                          4 (OK=4      KO=-     )
> response time 95th percentile                          8 (OK=8      KO=-     )
> response time 99th percentile                         20 (OK=20     KO=-     )
> mean requests/sec                                    100 (OK=100    KO=-     )
---- Response Time Distribution ------------------------------------------------
> t < 800 ms                                          6000 (100%)
> 800 ms <= t < 1200 ms                                  0 (  0%)
> t >= 1200 ms                                           0 (  0%)
> failed                                                 0 (  0%)
================================================================================

- PR with triggers enabled but no high watermark set

================================================================================
---- Global Information --------------------------------------------------------
> request count                                       6000 (OK=6000   KO=0     )
> min response time                                      2 (OK=2      KO=-     )
> max response time                                    189 (OK=189    KO=-     )
> mean response time                                     5 (OK=5      KO=-     )
> std deviation                                          6 (OK=6      KO=-     )
> response time 50th percentile                          5 (OK=5      KO=-     )
> response time 75th percentile                          6 (OK=6      KO=-     )
> response time 95th percentile                          8 (OK=8      KO=-     )
> response time 99th percentile                         15 (OK=15     KO=-     )
> mean requests/sec                                    100 (OK=100    KO=-     )
---- Response Time Distribution ------------------------------------------------
> t < 800 ms                                          6000 (100%)
> 800 ms <= t < 1200 ms                                  0 (  0%)
> t >= 1200 ms                                           0 (  0%)
> failed                                                 0 (  0%)
================================================================================

4. Custom AT to load and sign 1 block + 1 attestation for 10000 validators

- CONTROL (web3signer 23.9.0)
BUILD SUCCESSFUL in 4m 31s


- PR with high watermark set
BUILD SUCCESSFUL in 4m 36s

[jframe]

This is only checking that the watermarks didn't get updated. We should check that signed attestations and blocks data are not updated

[siladu]

done