lidofinance · F4ever · Sep 23, 2024 · Oct 3, 2024 · Oct 11, 2024 · Nov 28, 2024
diff --git a/README.md b/README.md
@@ -3,7 +3,8 @@
 [![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
 [![Tests](https://github.com/lidofinance/lido-oracle/workflows/Tests/badge.svg?branch=daemon_v2)](https://github.com/lidofinance/lido-oracle/actions)
 
-Oracle daemon for Lido decentralized staking service: Monitoring the state of the protocol across both layers and submitting regular update reports to the Lido smart contracts.
+Oracle daemon for Lido decentralized staking service: Monitoring the state of the protocol across both layers and submitting regular update
+reports to the Lido smart contracts.
 
 ## How it works
 
@@ -15,7 +16,8 @@ There are 3 modules in the oracle:
 
 ### Accounting module
 
-Accounting module updates the protocol TVL, distributes node-operator rewards, updates information about the number of exited and stuck validators and processes user withdrawal requests.
+Accounting module updates the protocol TVL, distributes node-operator rewards, updates information about the number of exited and stuck
+validators and processes user withdrawal requests.
 Also Accounting module makes decision to turn on/off the bunker.
 
 **Flow**
@@ -37,7 +39,8 @@ The frame includes these stages:
 
 ### Ejector module
 
-Ejector module requests Lido validators to eject via events in Execution Layer when the protocol requires additional funds to process user withdrawals.
+Ejector module requests Lido validators to eject via events in Execution Layer when the protocol requires additional funds to process user
+withdrawals.
 
 **Flow**
 
@@ -58,14 +61,16 @@ Only Oracle:
 - Memory - 8 GB
 
 Oracle + KAPI:
+
 - vCPU - 4
 - Memory - 16 GB
 
 ## Dependencies
 
 ### Execution Client Node
 
-To prepare the report, Oracle fetches up to 10 days old events, makes historical requests for balance data and makes simulated reports on historical blocks. This requires an [archive](https://ethereum.org/en/developers/docs/nodes-and-clients/#archive-node) execution node.
+To prepare the report, Oracle fetches up to 10 days old events, makes historical requests for balance data and makes simulated reports on
+historical blocks. This requires an [archive](https://ethereum.org/en/developers/docs/nodes-and-clients/#archive-node) execution node.
 Oracle needs two weeks of archived data.
 
 | Client                                          | Tested | Notes                                                                                                                                                                           |
@@ -77,19 +82,21 @@ Oracle needs two weeks of archived data.
 
 ### Consensus Client Node
 
-Also, to calculate some metrics for bunker mode Oracle needs [archive](https://ethereum.org/en/developers/docs/nodes-and-clients/#archive-node) consensus node.
+Also, to calculate some metrics for bunker mode Oracle
+needs [archive](https://ethereum.org/en/developers/docs/nodes-and-clients/#archive-node) consensus node.
 
-| Client                                            | Tested | Notes                                                                                                                                           |
-|---------------------------------------------------|:------:|-------------------------------------------------------------------------------------------------------------------------------------------------|
-| [Lighthouse](https://lighthouse.sigmaprime.io/)   |   🟢   | Use `--reconstruct-historic-states` param                                                                                                       |
-| [Lodestar](https://lodestar.chainsafe.io)         |   🔴   | Not tested yet                                                                                                                                  |
-| [Nimbus](https://nimbus.team)                     |   🔴   | Not tested yet                                                                                                                                  |
-| [Prysm](https://github.com/prysmaticlabs/prysm)   |   🟢   | Use <br> `--grpc-max-msg-size=104857600` <br> `--enable-historical-state-representation=true` <br> `--slots-per-archive-point=1024` <br> params |
-| [Teku](https://docs.teku.consensys.net)           |   🟢   | Use <br> `--data-storage-mode=archive` <br>`--data-storage-archive-frequency=1024`<br> `--reconstruct-historic-states=true`<br> params          |
+| Client                                          | Tested | Notes                                                                                                                                           |
+|-------------------------------------------------|:------:|-------------------------------------------------------------------------------------------------------------------------------------------------|
+| [Lighthouse](https://lighthouse.sigmaprime.io/) |   🟢   | Use `--reconstruct-historic-states` param                                                                                                       |
+| [Lodestar](https://lodestar.chainsafe.io)       |   🔴   | Not tested yet                                                                                                                                  |
+| [Nimbus](https://nimbus.team)                   |   🔴   | Not tested yet                                                                                                                                  |
+| [Prysm](https://github.com/prysmaticlabs/prysm) |   🟢   | Use <br> `--grpc-max-msg-size=104857600` <br> `--enable-historical-state-representation=true` <br> `--slots-per-archive-point=1024` <br> params |
+| [Teku](https://docs.teku.consensys.net)         |   🟢   | Use <br> `--data-storage-mode=archive` <br>`--data-storage-archive-frequency=1024`<br> `--reconstruct-historic-states=true`<br> params          |
 
 ### Keys API Service
 
-This is a separate service that uses Consensus and Execution Clients to fetch all lido keys. It stores the latest state of lido keys in database.
+This is a separate service that uses Consensus and Execution Clients to fetch all lido keys. It stores the latest state of lido keys in
+database.
 
 [Lido Keys API repository.](https://github.com/lidofinance/lido-keys-api)
 
@@ -102,9 +109,11 @@ Pull the image using the following command:
 docker pull lidofinance/oracle:{tag}
 ```
 
-Where `{tag}` is a version of the image. You can find the latest version in the [releases](https://github.com/lidofinance/lido-oracle/releases)
+Where `{tag}` is a version of the image. You can find the latest version in
+the [releases](https://github.com/lidofinance/lido-oracle/releases)
 **OR**\
-You can build it locally using the following command (make sure build it from latest [release](https://github.com/lidofinance/lido-oracle/releases)):
+You can build it locally using the following command (make sure build it from
+latest [release](https://github.com/lidofinance/lido-oracle/releases)):
 
 ```bash
 docker build -t lidofinance/oracle .
@@ -124,16 +133,17 @@ Full variables list could be found [here](https://github.com/lidofinance/lido-or
    and your environment is ready to run the oracle.
 
 ## Run the oracle
+
 1. By default, the oracle runs in *dry mode*. It means that it will not send any transactions to the Ethereum network.
-    To run Oracle in *production mode*, set `MEMBER_PRIV_KEY` or `MEMBER_PRIV_KEY_FILE` environment variable:
+   To run Oracle in *production mode*, set `MEMBER_PRIV_KEY` or `MEMBER_PRIV_KEY_FILE` environment variable:
     ```
     MEMBER_PRIV_KEY={value}
     ```
-    Where `{value}` is a private key of the Oracle member account or:
+   Where `{value}` is a private key of the Oracle member account or:
     ```
     MEMBER_PRIV_KEY_FILE={path}
     ```
-    Where `{path}` is a path to the private key of the Oracle member account.
+   Where `{path}` is a path to the private key of the Oracle member account.
 2. Run the container using the following command:
 
    ```bash
@@ -203,6 +213,7 @@ In manual mode all sleeps are disabled and `ALLOW_REPORTING_IN_BUNKER_MODE` is T
 | `CACHE_PATH`                                           | Directory to store cache for CSM module                                                                                                                                  | False    | `.`                                     |
 
 ### Mainnet variables
+
 > LIDO_LOCATOR_ADDRESS=0xC1d0b3DE6792Bf6b4b37EccdcC24e45978Cfd2Eb
 > ALLOW_REPORTING_IN_BUNKER_MODE=False
 
@@ -281,9 +292,12 @@ Special metrics for ejector oracle:
 
 Special metrics for CSM oracle:
 
-| Metric name                       | Description                                 | Labels |
-|-----------------------------------|---------------------------------------------|--------|
-| TBD                               | TBD                                         |        |
+| Metric name                     | Description                            | Labels |
+|---------------------------------|----------------------------------------|--------|
+| csm_current_frame_range_l_epoch | Left epoch of the current frame range  |        |
+| csm_current_frame_range_r_epoch | Right epoch of the current frame range |        |
+| csm_unprocessed_epochs_count    | Unprocessed epochs count               |        |
+| csm_min_unprocessed_epoch       | Minimum unprocessed epoch              |        |
 
 # Development
 

diff --git a/fixtures/tests/modules/accounting/test_withdrawal_integration.py/test_happy_path.json b/fixtures/tests/modules/accounting/test_withdrawal_integration.py/test_happy_path.json
@@ -6,7 +6,7 @@
         "to": "0x8D49f1b4AF30598679D4D37Be4B094da1b459b82",
         "data": "0xa3a3fd5d"
       },
-      "0x8ee61584b9d3e010c55f1fa77a803051f5f783385ec75b4e3fc71e199a86184d"
+      "latest"
     ],
     "response": {
       "jsonrpc": "2.0",
@@ -195,7 +195,7 @@
         "to": "0x4c1F6cA213abdbc19b27f2562d7b1A645A019bD9",
         "data": "0x29fd065d"
       },
-      "0x8ee61584b9d3e010c55f1fa77a803051f5f783385ec75b4e3fc71e199a86184d"
+      "latest"
     ],
     "response": {
       "jsonrpc": "2.0",
@@ -218,4 +218,4 @@
       "result": "0x0000000000000000000000000000000000000000000000037d3047cdfd698705000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000030af00000000000000000000000000000000000000000000000000000000000030c000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002"
     }
   }
-]
+]
diff --git a/src/constants.py b/src/constants.py
@@ -1,11 +1,13 @@
 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#misc
+from src.types import Gwei
+
 FAR_FUTURE_EPOCH = 2 ** 64 - 1
 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#time-parameters-1
-MIN_VALIDATOR_WITHDRAWABILITY_DELAY = 2**8
+MIN_VALIDATOR_WITHDRAWABILITY_DELAY = 2 ** 8
 SHARD_COMMITTEE_PERIOD = 256
 MAX_SEED_LOOKAHEAD = 4
 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#state-list-lengths
-EPOCHS_PER_SLASHINGS_VECTOR = 2**13
+EPOCHS_PER_SLASHINGS_VECTOR = 2 ** 13
 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#rewards-and-penalties
 PROPORTIONAL_SLASHING_MULTIPLIER_BELLATRIX = 3
 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#gwei-values
@@ -15,12 +17,18 @@
 MAX_WITHDRAWALS_PER_PAYLOAD = 2 ** 4
 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#withdrawal-prefixes
 ETH1_ADDRESS_WITHDRAWAL_PREFIX = '0x01'
+# https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#withdrawal-prefixes
+COMPOUNDING_WITHDRAWAL_PREFIX = '0x02'
 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#validator-cycle
 MIN_PER_EPOCH_CHURN_LIMIT = 2 ** 2
 CHURN_LIMIT_QUOTIENT = 2 ** 16
 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#time-parameters
 SLOTS_PER_HISTORICAL_ROOT = 8192
 
+# https://github.com/ethereum/consensus-specs/blob/dev/specs/electra/beacon-chain.md#gwei-values
+MIN_ACTIVATION_BALANCE = Gwei(2 ** 5 * 10 ** 9)
+MAX_EFFECTIVE_BALANCE_ELECTRA = Gwei(2 ** 11 * 10 ** 9)
+
 # Local constants
 GWEI_TO_WEI = 10 ** 9
 SHARE_RATE_PRECISION_E27 = 10 ** 27

diff --git a/src/main.py b/src/main.py
@@ -31,7 +31,6 @@
 
 from src.web3py.contract_tweak import tweak_w3_contracts
 
-
 logger = logging.getLogger(__name__)
 
 
@@ -42,22 +41,10 @@ def main(module_name: OracleModule):
         'variables': {
             **build_info,
             'module': module_name,
-            'ACCOUNT': variables.ACCOUNT.address if variables.ACCOUNT else 'Dry',
-            'LIDO_LOCATOR_ADDRESS': variables.LIDO_LOCATOR_ADDRESS,
-            'CSM_MODULE_ADDRESS': variables.CSM_MODULE_ADDRESS,
-            'FINALIZATION_BATCH_MAX_REQUEST_COUNT': variables.FINALIZATION_BATCH_MAX_REQUEST_COUNT,
-            'EL_REQUESTS_BATCH_SIZE': variables.EL_REQUESTS_BATCH_SIZE,
-            'MAX_CYCLE_LIFETIME_IN_SECONDS': variables.MAX_CYCLE_LIFETIME_IN_SECONDS,
+            **variables.PUBLIC_ENV_VARS,
         },
     })
-    ENV_VARIABLES_INFO.info({
-        "ACCOUNT": str(variables.ACCOUNT.address) if variables.ACCOUNT else 'Dry',
-        "LIDO_LOCATOR_ADDRESS": str(variables.LIDO_LOCATOR_ADDRESS),
-        "CSM_MODULE_ADDRESS": str(variables.CSM_MODULE_ADDRESS),
-        "FINALIZATION_BATCH_MAX_REQUEST_COUNT": str(variables.FINALIZATION_BATCH_MAX_REQUEST_COUNT),
-        "EL_REQUESTS_BATCH_SIZE": str(variables.EL_REQUESTS_BATCH_SIZE),
-        "MAX_CYCLE_LIFETIME_IN_SECONDS": str(variables.MAX_CYCLE_LIFETIME_IN_SECONDS),
-    })
+    ENV_VARIABLES_INFO.info(variables.PUBLIC_ENV_VARS)
     BUILD_INFO.info(build_info)
 
     logger.info({'msg': f'Start healthcheck server for Docker container on port {variables.HEALTHCHECK_SERVER_PORT}'})

diff --git a/src/modules/accounting/accounting.py b/src/modules/accounting/accounting.py
@@ -326,7 +326,7 @@ def get_extra_data(self, blockstamp: ReferenceBlockStamp) -> ExtraData:
         logger.info({'msg': 'Calculate stuck validators.', 'value': stuck_validators})
         exited_validators = self.lido_validator_state_service.get_lido_newly_exited_validators(blockstamp)
         logger.info({'msg': 'Calculate exited validators.', 'value': exited_validators})
-        orl = self.w3.lido_contracts.oracle_report_sanity_checker.get_oracle_report_limits(blockstamp.block_hash)
+        orl = self.w3.lido_contracts.oracle_report_sanity_checker.get_oracle_report_limits()
 
         if consensus_version == 1:
             return ExtraDataService.collect(
@@ -350,7 +350,7 @@ def _get_generic_extra_data(self, blockstamp: ReferenceBlockStamp) -> GenericExt
         logger.info({'msg': 'Calculate stuck validators.', 'value': stuck_validators})
         exited_validators = self.lido_validator_state_service.get_lido_newly_exited_validators(blockstamp)
         logger.info({'msg': 'Calculate exited validators.', 'value': exited_validators})
-        orl = self.w3.lido_contracts.oracle_report_sanity_checker.get_oracle_report_limits(blockstamp.block_hash)
+        orl = self.w3.lido_contracts.oracle_report_sanity_checker.get_oracle_report_limits()
         return stuck_validators, exited_validators, orl
 
     def _calculate_report_v1(self, blockstamp: ReferenceBlockStamp) -> ReportData:

diff --git a/src/providers/consensus/client.py b/src/providers/consensus/client.py
@@ -22,7 +22,6 @@
 
 logger = logging.getLogger(__name__)
 
-
 LiteralState = Literal['head', 'genesis', 'finalized', 'justified']
 
 
@@ -145,10 +144,10 @@ def get_attestation_committees(
     @lru_cache(maxsize=1)
     def get_state_block_roots(self, state_id: SlotNumber) -> list[BlockRoot]:
         streamed_json = cast(TransientStreamingJSONObject, self._get(
-                self.API_GET_STATE,
-                path_params=(state_id,),
-                stream=True,
-            ))
+            self.API_GET_STATE,
+            path_params=(state_id,),
+            stream=True,
+        ))
         return list(streamed_json['data']['block_roots'])
 
     @lru_cache(maxsize=1)
@@ -159,6 +158,12 @@ def get_validators(self, blockstamp: BlockStamp) -> list[Validator]:
     @list_of_dataclasses(Validator.from_response)
     def get_validators_no_cache(self, blockstamp: BlockStamp, pub_keys: str | tuple | None = None) -> list[dict]:
         """Spec: https://ethereum.github.io/beacon-APIs/#/Beacon/getStateValidators"""
+        logger.info({
+            'msg': 'Getting validators...',
+            'url': self.API_GET_VALIDATORS,
+            'slot_number': blockstamp.slot_number,
+            'state_root': blockstamp.state_root,
+        })
         try:
             data, _ = self._get(
                 self.API_GET_VALIDATORS,
@@ -168,6 +173,7 @@ def get_validators_no_cache(self, blockstamp: BlockStamp, pub_keys: str | tuple
             )
             if not isinstance(data, list):
                 raise ValueError("Expected list response from getStateValidators")
+            logger.info({'msg': f'Fetched {len(data)} validators'})
             return data
         except NotOkResponse as error:
             if self.PRYSM_STATE_NOT_FOUND_ERROR in error.text:

diff --git a/src/providers/execution/contracts/lido.py b/src/providers/execution/contracts/lido.py
@@ -89,7 +89,7 @@ def _handle_oracle_report(
         state_override: dict[ChecksumAddress, CallOverrideParams] = {
             accounting_oracle_address: {
                 # Fix: insufficient funds for gas * price + value
-                'balance': Wei(10**18),
+                'balance': Wei(100 * 10**18),
                 # Fix: Sanity checker uses `lastProcessingRefSlot` from AccountingOracle to
                 # properly process negative rebase sanity checks. Since current simulation skips call to AO,
                 # setting up `lastProcessingRefSlot` directly.