Expose additional builder booster related flags in the vc (#5086)

Squashed commit of the following:

commit 4df4663
Author: Paul Hauner <[email protected]>
Date:   Wed Jan 24 18:23:20 2024 +1100

    Fix issue with PATCH request

commit 45d7ea5
Author: Jimmy Chen <[email protected]>
Date:   Wed Jan 24 17:26:25 2024 +1100

    Add http test for builder boost factor with process defaults.

commit 800e29c
Merge: b225328 2401fd7
Author: Jimmy Chen <[email protected]>
Date:   Wed Jan 24 12:29:28 2024 +1100

    Merge branch 'expose-builder-booster-flags-vc' of github.com:eserilev/lighthouse into fork/eserilev/expose-builder-booster-flags-vc

commit b225328
Author: Jimmy Chen <[email protected]>
Date:   Wed Jan 24 11:37:14 2024 +1100

    Prioritise per validator builder boost configurations over CLI flags.

commit 2401fd7
Author: Mac L <[email protected]>
Date:   Wed Jan 24 11:03:31 2024 +1100

    Fix CLI help text

commit c852a51
Author: Eitan Seri-Levi <[email protected]>
Date:   Tue Jan 23 03:34:39 2024 +0200

    typo

commit 59af312
Author: Eitan Seri-Levi <[email protected]>
Date:   Tue Jan 23 03:32:56 2024 +0200

    typo

commit 24a522b
Author: Eitan Seri-Levi <[email protected]>
Date:   Tue Jan 23 02:55:33 2024 +0200

    fmt

commit f3a1b30
Author: Eitan Seri-Levi <[email protected]>
Date:   Tue Jan 23 02:02:45 2024 +0200

    assume builder-proosals flag if one of other two vc builder flags are present

commit 2b2c9af
Author: Eitan Seri-Levi <[email protected]>
Date:   Mon Jan 22 19:43:27 2024 +0200

    fix typos

commit d1591fe
Author: Eitan Seri-Levi <[email protected]>
Date:   Mon Jan 22 19:43:20 2024 +0200

    fix typos

commit dc7a870
Author: Eitan Seri-Levi <[email protected]>
Date:   Mon Jan 22 19:25:51 2024 +0200

    remove unneeded val, change how boost factor flag logic in the vc, add some additional documentation

commit 7a214f7
Author: Eitan Seri-Levi <[email protected]>
Date:   Sat Jan 20 21:57:36 2024 +0200

    fix issues related to CreateConfig and MoveConfig

commit 89ade68
Author: Eitan Seri-Levi <[email protected]>
Date:   Sat Jan 20 01:52:42 2024 +0200

    resolve failing test

commit 5641d24
Merge: 03d68f5 185646a
Author: Eitan Seri-Levi <[email protected]>
Date:   Fri Jan 19 04:30:54 2024 +0200

    Merge branch 'unstable' of https://github.com/sigp/lighthouse into expose-builder-booster-flags-vc

commit 03d68f5
Author: Eitan Seri-Levi <[email protected]>
Date:   Fri Jan 19 04:30:39 2024 +0200

    expose builder booster flags in vc, enable options in validator endpoints, update tests

account_manager/src/validator/import.rs

@@ -284,6 +284,8 @@ pub fn cli_run(matches: &ArgMatches, validator_dir: PathBuf) -> Result<(), Strin
             suggested_fee_recipient,
             None,
             None,
+            None,
+            None,
         )
         .map_err(|e| format!("Unable to create new validator definition: {:?}", e))?;
 

book/src/api-vc-endpoints.md

@@ -427,7 +427,7 @@ Example Response Body
 
 ## `PATCH /lighthouse/validators/:voting_pubkey`
 
-Update some values for the validator with `voting_pubkey`. Possible fields: `enabled`, `gas_limit`, `builder_proposals`, 
+Update some values for the validator with `voting_pubkey`. Possible fields: `enabled`, `gas_limit`, `builder_proposals`, `builder_boost_factor`, `prefer_builder_proposals`
 and `graffiti`.  The following example updates a validator from `enabled: true` to `enabled: false`.
 
 ### HTTP Specification

book/src/builders.md

@@ -31,6 +31,18 @@ blinded blocks, you should use the following flag:
 lighthouse vc --builder-proposals
 ```
 With the `--builder-proposals` flag, the validator client will ask for blinded blocks for all validators it manages.
+
+```
+lighthouse vc --prefer-builder-proposals
+```
+With the `--prefer-builder-proposals` flag, the validator client will always prefer blinded blocks, regardless of the payload value, for all validators it manages.
+
+```
+lighthouse vc --builder-boost-factor <INTEGER>
+```
+With the `--builder-boost-factor` flag, a percentage multiplier is applied to the builder's payload value when choosing between a
+builder payload header and payload from the paired execution node.
+
 In order to configure whether a validator queries for blinded blocks check out [this section.](#validator-client-configuration)
 
 ## Multiple builders
@@ -46,9 +58,9 @@ relays, run one of the following services and configure lighthouse to use it wit
 In the validator client you can configure gas limit and fee recipient on a per-validator basis. If no gas limit is
 configured, Lighthouse will use a default gas limit of 30,000,000, which is the current default value used in execution
 engines.  You can also enable or disable use of external builders on a per-validator basis rather than using
-`--builder-proposals`, which enables external builders for all validators. In order to manage these configurations
-per-validator, you can either make updates to the `validator_definitions.yml` file or you can use the HTTP requests
-described below.
+`--builder-proposals`, `--builder-boost-factor` or `--prefer-builder-proposals`, which apply builder related preferences for all validators.
+In order to manage these configurations per-validator, you can either make updates to the `validator_definitions.yml` file
+or you can use the HTTP requests described below.
 
 Both the gas limit and fee recipient will be passed along as suggestions to connected builders. If there is a discrepancy
 in either, it will *not* keep you from proposing a block with the builder. This is because the bounds on gas limit are

book/src/help_vc.md

@@ -56,6 +56,9 @@ FLAGS:
             machine. Note that logs can often contain sensitive information about your validator and so this flag should
             be used with caution. For Windows users, the log file permissions will be inherited from the parent folder.
         --metrics                                Enable the Prometheus metrics HTTP server. Disabled by default.
+        --prefer-builder-proposals
+            If this flag is set, Lighthouse will always prefer blocks constructed by builders, regardless of payload
+            value.
         --produce-block-v3
             Enable block production via the block v3 endpoint for this validator client. This should only be enabled
             when paired with a beacon node that has this endpoint implemented. This flag will be enabled by default in
@@ -80,6 +83,9 @@ OPTIONS:
             Comma-separated list of beacon API topics to broadcast to all beacon nodes. Possible values are: none,
             attestations, blocks, subscriptions, sync-committee. Default (when flag is omitted) is to broadcast
             subscriptions only.
+        --builder-boost-factor <UINT64>
+            Defines the boost factor, a percentage multiplier to apply to the builder's payload value when choosing
+            between a builder payload header and payload from the local execution node.
         --builder-registration-timestamp-override <builder-registration-timestamp-override>
             This flag takes a unix timestamp value that will be used to override the timestamp used in the builder api
             registration

book/src/help_vm_create.md

@@ -42,6 +42,9 @@ OPTIONS:
             A HTTP(S) address of a beacon node using the beacon-API. If this value is provided, an error will be raised
             if any validator key here is already known as a validator by that beacon node. This helps prevent the same
             validator being created twice and therefore slashable conditions.
+        --builder-boost-factor <UINT64>
+            Defines the boost factor, a percentage multiplier to apply to the builder's payload value when choosing
+            between a builder payload header and payload from the local execution node.
         --builder-proposals <builder-proposals>
             When provided, all created validators will attempt to create blocks via builder rather than the local EL.
             [possible values: true, false]
@@ -93,13 +96,18 @@ OPTIONS:
         --logfile-max-size <SIZE>
             The maximum size (in MB) each log file can grow to before rotating. If set to 0, background file logging is
             disabled. [default: 200]
-        --mnemonic-path <MNEMONIC_PATH>                         If present, the mnemonic will be read in from this file.
+        --mnemonic-path <MNEMONIC_PATH>
+            If present, the mnemonic will be read in from this file.
+
         --network <network>
             Name of the Eth2 chain Lighthouse will sync and follow. [possible values: mainnet, prater, goerli, gnosis,
             chiado, sepolia, holesky]
         --output-path <DIRECTORY>
             The path to a directory where the validator and (optionally) deposits files will be created. The directory
             will be created if it does not exist.
+        --prefer-builder-proposals <prefer-builder-proposals>
+            If this flag is set, Lighthouse will always prefer blocks constructed by builders, regardless of payload
+            value. [possible values: true, false]
         --safe-slots-to-import-optimistically <INTEGER>
             Used to coordinate manual overrides of the SAFE_SLOTS_TO_IMPORT_OPTIMISTICALLY parameter. This flag should
             only be used if the user has a clear understanding that the broad Ethereum community has elected to override

book/src/help_vm_move.md

@@ -26,10 +26,13 @@ FLAGS:
     -V, --version                        Prints version information
 
 OPTIONS:
+        --builder-boost-factor <UINT64>
+            Defines the boost factor, a percentage multiplier to apply to the builder's payload value when choosing
+            between a builder payload header and payload from the local execution node.
         --builder-proposals <builder-proposals>
             When provided, all created validators will attempt to create blocks via builder rather than the local EL.
             [possible values: true, false]
-        --count <VALIDATOR_COUNT>                               The number of validators to move.
+        --count <VALIDATOR_COUNT>                                The number of validators to move.
     -d, --datadir <DIR>
             Used to specify a custom root data directory for lighthouse keys and databases. Defaults to
             $HOME/.lighthouse/{network} where network is the value of the `network` flag Note: Users should specify
@@ -75,6 +78,9 @@ OPTIONS:
         --network <network>
             Name of the Eth2 chain Lighthouse will sync and follow. [possible values: mainnet, prater, goerli, gnosis,
             chiado, sepolia, holesky]
+        --prefer-builder-proposals <prefer-builder-proposals>
+            If this flag is set, Lighthouse will always prefer blocks constructed by builders, regardless of payload
+            value. [possible values: true, false]
         --safe-slots-to-import-optimistically <INTEGER>
             Used to coordinate manual overrides of the SAFE_SLOTS_TO_IMPORT_OPTIMISTICALLY parameter. This flag should
             only be used if the user has a clear understanding that the broad Ethereum community has elected to override

common/account_utils/src/validator_definitions.rs

@@ -157,6 +157,12 @@ pub struct ValidatorDefinition {
     #[serde(skip_serializing_if = "Option::is_none")]
     pub builder_proposals: Option<bool>,
     #[serde(default)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub builder_boost_factor: Option<u64>,
+    #[serde(default)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub prefer_builder_proposals: Option<bool>,
+    #[serde(default)]
     pub description: String,
     #[serde(flatten)]
     pub signing_definition: SigningDefinition,
@@ -169,13 +175,16 @@ impl ValidatorDefinition {
     /// ## Notes
     ///
     /// This function does not check the password against the keystore.
+    #[allow(clippy::too_many_arguments)]
     pub fn new_keystore_with_password<P: AsRef<Path>>(
         voting_keystore_path: P,
         voting_keystore_password_storage: PasswordStorage,
         graffiti: Option<GraffitiString>,
         suggested_fee_recipient: Option<Address>,
         gas_limit: Option<u64>,
         builder_proposals: Option<bool>,
+        builder_boost_factor: Option<u64>,
+        prefer_builder_proposals: Option<bool>,
     ) -> Result<Self, Error> {
         let voting_keystore_path = voting_keystore_path.as_ref().into();
         let keystore =
@@ -196,6 +205,8 @@ impl ValidatorDefinition {
             suggested_fee_recipient,
             gas_limit,
             builder_proposals,
+            builder_boost_factor,
+            prefer_builder_proposals,
             signing_definition: SigningDefinition::LocalKeystore {
                 voting_keystore_path,
                 voting_keystore_password_path,
@@ -344,6 +355,8 @@ impl ValidatorDefinitions {
                     suggested_fee_recipient: None,
                     gas_limit: None,
                     builder_proposals: None,
+                    builder_boost_factor: None,
+                    prefer_builder_proposals: None,
                     signing_definition: SigningDefinition::LocalKeystore {
                         voting_keystore_path,
                         voting_keystore_password_path,

common/eth2/src/lighthouse_vc/http_client.rs

@@ -483,12 +483,15 @@ impl ValidatorClientHttpClient {
     }
 
     /// `PATCH lighthouse/validators/{validator_pubkey}`
+    #[allow(clippy::too_many_arguments)]
     pub async fn patch_lighthouse_validators(
         &self,
         voting_pubkey: &PublicKeyBytes,
         enabled: Option<bool>,
         gas_limit: Option<u64>,
         builder_proposals: Option<bool>,
+        builder_boost_factor: Option<u64>,
+        prefer_builder_proposals: Option<bool>,
         graffiti: Option<GraffitiString>,
     ) -> Result<(), Error> {
         let mut path = self.server.full.clone();
@@ -505,6 +508,8 @@ impl ValidatorClientHttpClient {
                 enabled,
                 gas_limit,
                 builder_proposals,
+                builder_boost_factor,
+                prefer_builder_proposals,
                 graffiti,
             },
         )

common/eth2/src/lighthouse_vc/types.rs

@@ -32,6 +32,12 @@ pub struct ValidatorRequest {
     #[serde(default)]
     #[serde(skip_serializing_if = "Option::is_none")]
     pub builder_proposals: Option<bool>,
+    #[serde(default)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub builder_boost_factor: Option<u64>,
+    #[serde(default)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub prefer_builder_proposals: Option<bool>,
     #[serde(with = "serde_utils::quoted_u64")]
     pub deposit_gwei: u64,
 }
@@ -86,6 +92,12 @@ pub struct ValidatorPatchRequest {
     #[serde(default)]
     #[serde(skip_serializing_if = "Option::is_none")]
     pub graffiti: Option<GraffitiString>,
+    #[serde(default)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub builder_boost_factor: Option<u64>,
+    #[serde(default)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub prefer_builder_proposals: Option<bool>,
 }
 
 #[derive(Clone, PartialEq, Serialize, Deserialize)]
@@ -105,6 +117,12 @@ pub struct KeystoreValidatorsPostRequest {
     #[serde(default)]
     #[serde(skip_serializing_if = "Option::is_none")]
     pub builder_proposals: Option<bool>,
+    #[serde(default)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub builder_boost_factor: Option<u64>,
+    #[serde(default)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub prefer_builder_proposals: Option<bool>,
 }
 
 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
@@ -135,6 +153,12 @@ pub struct Web3SignerValidatorRequest {
     pub client_identity_path: Option<PathBuf>,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub client_identity_password: Option<String>,
+    #[serde(default)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub builder_boost_factor: Option<u64>,
+    #[serde(default)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub prefer_builder_proposals: Option<bool>,
 }
 
 #[derive(Debug, Clone, PartialEq, Deserialize, Serialize)]

lighthouse/tests/account_manager.rs

@@ -492,6 +492,8 @@ fn validator_import_launchpad() {
         suggested_fee_recipient: None,
         gas_limit: None,
         builder_proposals: None,
+        builder_boost_factor: None,
+        prefer_builder_proposals: None,
         voting_public_key: keystore.public_key().unwrap(),
         signing_definition: SigningDefinition::LocalKeystore {
             voting_keystore_path,
@@ -614,6 +616,8 @@ fn validator_import_launchpad_no_password_then_add_password() {
         suggested_fee_recipient: None,
         gas_limit: None,
         builder_proposals: None,
+        builder_boost_factor: None,
+        prefer_builder_proposals: None,
         voting_public_key: keystore.public_key().unwrap(),
         signing_definition: SigningDefinition::LocalKeystore {
             voting_keystore_path,
@@ -640,6 +644,8 @@ fn validator_import_launchpad_no_password_then_add_password() {
         suggested_fee_recipient: None,
         gas_limit: None,
         builder_proposals: None,
+        builder_boost_factor: None,
+        prefer_builder_proposals: None,
         voting_public_key: keystore.public_key().unwrap(),
         signing_definition: SigningDefinition::LocalKeystore {
             voting_keystore_path: dst_keystore_dir.join(KEYSTORE_NAME),
@@ -742,6 +748,8 @@ fn validator_import_launchpad_password_file() {
         suggested_fee_recipient: None,
         gas_limit: None,
         builder_proposals: None,
+        builder_boost_factor: None,
+        prefer_builder_proposals: None,
         signing_definition: SigningDefinition::LocalKeystore {
             voting_keystore_path,
             voting_keystore_password_path: None,

lighthouse/tests/validator_client.rs

@@ -464,6 +464,32 @@ fn builder_proposals_flag() {
         .with_config(|config| assert!(config.builder_proposals));
 }
 #[test]
+fn builder_boost_factor_flag() {
+    CommandLineTest::new()
+        .flag("builder-boost-factor", Some("150"))
+        .run()
+        .with_config(|config| assert_eq!(config.builder_boost_factor, Some(150)));
+}
+#[test]
+fn no_builder_boost_factor_flag() {
+    CommandLineTest::new()
+        .run()
+        .with_config(|config| assert_eq!(config.builder_boost_factor, None));
+}
+#[test]
+fn prefer_builder_proposals_flag() {
+    CommandLineTest::new()
+        .flag("prefer-builder-proposals", None)
+        .run()
+        .with_config(|config| assert!(config.prefer_builder_proposals));
+}
+#[test]
+fn no_prefer_builder_proposals_flag() {
+    CommandLineTest::new()
+        .run()
+        .with_config(|config| assert!(!config.prefer_builder_proposals));
+}
+#[test]
 fn no_builder_registration_timestamp_override_flag() {
     CommandLineTest::new()
         .run()