[WIP] PoV friendly, multi-block Election Provider Multi-Phase pallet (#…

…2504)

> highly WIP, opening draft PR for early feedback.

This PR implements a PoV friendly, multi-block EPM to be used by the
staking parachain. It is split into multiple sub-pallets for better
logic and storage encapsulation and better readability. The pallet split
consists of a main pallet and several other sub-pallets that implement
the logic for different sub-systems, namely:

- **main pallet**: 
    - implements the `trait ElectionProvider`
- `fn elect(remaining_pages)` basically fetches the queued page from the
`pallet::verifier`, which keeps the valid solutions in its storage.
- manages current election `Phase` in `on_initialize`. The current phase
signals other sub-pallets what to do.
    - stores and manages the (paged) target and voter snapshots.
- *note*: the staking pallet needs to return/interpret a paged fetching
of the snapshot data for both voters and targes.
- **signed pallet**: 
- implements the `trait SolutionDataProvider`, which provides a paged
solution for the current best score (the `pallet::verifier` is the
consumer, when trying to fetch the best queued solution to verify).
    - keeps track of the best solution commitments from submitters.
- exposes `Call::register` for submitters to submit a commitment score
for their solution.
- exposes callable `Call::submit_page` for submitters to submit a page
of their solution.
- upon the verifier pallet finalizing the paged solution verification,
it handles the submission deposit/rewards based on the reported
`VerificationResult` (from `pallet::signed`).
- **verifier pallet**: 
- implements the `trait Verifier`: verifies one solution page on-call.
The inputs for the verification are provided in-place.
- implements the `trait AsyncVerifier`: fetches pages from the
implementor of `SolutionDataProvider` (implemented by `pallet::signed`)
and verifies the paged solution.
- `on_initialize`, it checks if the verification is ongoing and proceeds
with it
- it has it's own `VerificationStatus` which signals the current state
of the verification
- for each successfully verified page, add it to the `QueuedSolution`
storage.
- at the end of verifying a solution, it reports the results of the
verification back to the implementor of `trait SolutionDataProvider`
(`pallet::signed`)
- **unsigned pallet**:
- `on_initialize` checks if on `UnsignedPhase` and no queued solution;
compute a solution with offchain Miner.
  - implements the off-chain unsigned (paged) miner. 
  - implements the inherent call that processes unsigned submissions.

---

### Todo/discussion

- [x] E2E multi-page election with staking and EPM-MB pallet
integration.
- [ ] refactor the current `on_initialize` across all pallets to make
explicit calls depending on the current phase, rather than relying on
the pallet's `on_initialize` and current phase to decide what to do at a
given block (TBD).
- [ ] remove the `Emergency` phase and instead just keep trying the
election in case of failure.
- [ ] refactor current `SignedValidation` phase to have enough
blockspace to verify all queued signed submissions, for security
purposes (ie. at least `max_num_queued_submissions * T::Pages` blocks
allocated to signed verification, return early if a submission is valid
and accepted).
- [x] implement the paged ingestion of the election results in the
staking pallet. How to convert from multiple `BoundedSupports` to
`Exposures` in the staking pallet in a nice way (add integration tests).
- idea: if each page contains up to `N` targets and a validator may
appear only in one page, we can process the pages in serie in the
staking side, keeping track of the state of `Exposures` across the
pages.
- [ ] allow the validator to replace the current submission if their
submission has better score than the accepted queued submission.
- [ ] mutations to both the target and voter lists need to "freeze"
while the snapshot is being generated (now multi-block). what's the best
approach?
 
Closes: #2199
Related to: #491
Inspiration from
https://github.com/paritytech/substrate/tree/kiz-multi-block-election

Cargo.toml

@@ -323,6 +323,8 @@ members = [
 	"substrate/frame/conviction-voting",
 	"substrate/frame/core-fellowship",
 	"substrate/frame/democracy",
+	"substrate/frame/election-provider-multi-block",
+	"substrate/frame/election-provider-multi-block/integration-tests",
 	"substrate/frame/election-provider-multi-phase",
 	"substrate/frame/election-provider-multi-phase/test-staking-e2e",
 	"substrate/frame/election-provider-support",

substrate/frame/election-provider-multi-block/Cargo.toml

@@ -0,0 +1,61 @@
+[package]
+name = "pallet-election-provider-multi-block"
+version = "4.0.0-dev"
+authors.workspace = true
+edition.workspace = true
+license = "Apache-2.0"
+homepage = "https://substrate.dev"
+repository.workspace = true
+description = "FRAME pallet election provider multi-block"
+readme = "README.md"
+
+[package.metadata.docs.rs]
+targets = ["x86_64-unknown-linux-gnu"]
+
+[dependencies]
+codec = { package = "parity-scale-codec", version = "3.6.1", default-features = false, features = [
+	"derive",
+] }
+scale-info = { version = "2.10.0", default-features = false, features = [
+	"derive",
+] }
+log = { version = "0.4.17", default-features = false }
+
+frame-support = { version = "4.0.0-dev", default-features = false, path = "../support" }
+frame-system = { version = "4.0.0-dev", default-features = false, path = "../system" }
+
+sp-io = { path = "../../primitives/io", default-features = false}
+sp-std = { path = "../../primitives/std", default-features = false}
+sp-core = { path = "../../primitives/core", default-features = false}
+sp-runtime = { path = "../../primitives/runtime", default-features = false}
+
+frame-election-provider-support = { default-features = false, path = "../election-provider-support" }
+sp-npos-elections = { default-features = false, path = "../../primitives/npos-elections" }
+
+[dev-dependencies]
+sp-tracing = { path = "../../primitives/tracing" }
+pallet-balances = { path = "../balances", default-features = false}
+
+[features]
+default = [ "std" ]
+std = [
+	"codec/std",
+	"scale-info/std",
+	"log/std",
+	"frame-support/std",
+	"frame-system/std",
+	"sp-std/std",
+	"sp-core/std",
+	"sp-io/std",
+	"sp-runtime/std",
+	"frame-election-provider-support/std",
+	"sp-npos-elections/std",
+	"pallet-balances/std",
+]
+try-runtime = [
+	"frame-election-provider-support/try-runtime",
+	"frame-support/try-runtime",
+	"frame-system/try-runtime",
+	"pallet-balances/try-runtime",
+	"sp-runtime/try-runtime",
+]

substrate/frame/election-provider-multi-block/README.md

@@ -0,0 +1 @@
+# Election Provider multi-block

substrate/frame/election-provider-multi-block/integration-tests/Cargo.toml

@@ -0,0 +1,39 @@
+[package]
+name = "pallet-election-tests"
+version = "1.0.0"
+authors.workspace = true
+edition.workspace = true
+license = "Apache-2.0"
+homepage = "https://substrate.io"
+repository.workspace = true
+description = "FRAME election provider multi block pallet tests with staking pallet, bags-list and session pallets"
+publish = false
+
+[package.metadata.docs.rs]
+targets = ["x86_64-unknown-linux-gnu"]
+
+[dev-dependencies]
+parking_lot = "0.12.1"
+codec = { package = "parity-scale-codec", version = "3.6.1", features = ["derive"] }
+scale-info = { version = "2.10.0", features = ["derive"] }
+log = { version = "0.4.17", default-features = false }
+substrate-test-utils = { path = "../../../test-utils" }
+
+sp-runtime = { path = "../../../primitives/runtime" }
+sp-io = { path = "../../../primitives/io" }
+sp-std = { path = "../../../primitives/std" }
+sp-staking = { path = "../../../primitives/staking" }
+sp-core = { path = "../../../primitives/core" }
+sp-npos-elections = { path = "../../../primitives/npos-elections", default-features = false}
+sp-tracing = { path = "../../../primitives/tracing" }
+
+frame-system = { path = "../../system" }
+frame-support = { path = "../../support" }
+frame-election-provider-support = { path = "../../election-provider-support" }
+
+pallet-election-provider-multi-block = { path = ".." }
+pallet-staking = { path = "../../staking" }
+pallet-bags-list = { path = "../../bags-list" }
+pallet-balances = { path = "../../balances" }
+pallet-timestamp = { path = "../../timestamp" }
+pallet-session = { path = "../../session" }

substrate/frame/election-provider-multi-block/integration-tests/src/lib.rs

@@ -0,0 +1,125 @@
+// This file is part of Substrate.
+
+// Copyright (C) Parity Technologies (UK) Ltd.
+// SPDX-License-Identifier: Apache-2.0
+
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 	http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#![cfg(test)]
+mod mock;
+
+pub(crate) const LOG_TARGET: &str = "integration-tests::epm-staking";
+
+use mock::*;
+
+use frame_election_provider_support::{bounds::ElectionBoundsBuilder, ElectionDataProvider};
+
+use frame_support::assert_ok;
+
+// syntactic sugar for logging.
+#[macro_export]
+macro_rules! log {
+	($level:tt, $patter:expr $(, $values:expr)* $(,)?) => {
+		log::$level!(
+			target: crate::LOG_TARGET,
+			concat!("🛠️  ", $patter)  $(, $values)*
+		)
+	};
+}
+
+fn log_current_time() {
+	log!(
+		info,
+		"block: {:?}, session: {:?}, era: {:?}, EPM phase: {:?} ts: {:?}",
+		System::block_number(),
+		Session::current_index(),
+		Staking::current_era(),
+		ElectionProvider::current_phase(),
+		Timestamp::now()
+	);
+}
+
+#[test]
+fn block_progression_works() {
+	let (mut ext, _pool_state, _) = ExtBuilder::default().build_offchainify();
+	ext.execute_with(|| {})
+}
+
+#[test]
+fn verify_snapshot() {
+	ExtBuilder::default().build_and_execute(|| {
+		assert_eq!(Pages::get(), 3);
+
+		// manually get targets and voters from staking to see the inspect the issue with the
+		// DataProvider.
+		let bounds = ElectionBoundsBuilder::default()
+			.targets_count((TargetSnapshotPerBlock::get() as u32).into())
+			.voters_count((VoterSnapshotPerBlock::get() as u32).into())
+			.build();
+
+		assert_ok!(<Staking as ElectionDataProvider>::electable_targets(bounds.targets, 2));
+		assert_ok!(<Staking as ElectionDataProvider>::electing_voters(bounds.voters, 2));
+	})
+}
+
+mod staking_integration {
+	use super::*;
+	use pallet_election_provider_multi_block::Phase;
+
+	#[test]
+	fn call_elect_multi_block() {
+		ExtBuilder::default().build_and_execute(|| {
+			assert_eq!(Pages::get(), 3);
+			assert_eq!(ElectionProvider::current_round(), 0);
+			assert_eq!(Staking::current_era(), Some(0));
+
+			let export_starts_at = election_prediction() - Pages::get();
+
+			assert!(Staking::election_data_lock().is_none());
+
+			// check that the election data provider lock is set during the snapshot phase and
+			// released afterwards.
+			roll_to_phase(Phase::Snapshot(Pages::get() - 1), false);
+			assert!(Staking::election_data_lock().is_some());
+
+			roll_one(None, false);
+			assert!(Staking::election_data_lock().is_some());
+			roll_one(None, false);
+			assert!(Staking::election_data_lock().is_some());
+			// snapshot phase done, election data lock was released.
+			roll_one(None, false);
+			assert_eq!(ElectionProvider::current_phase(), Phase::Signed);
+			assert!(Staking::election_data_lock().is_none());
+
+			// last block where phase is waiting for unsignned submissions.
+			roll_to(election_prediction() - 4, false);
+			assert_eq!(ElectionProvider::current_phase(), Phase::Unsigned(17));
+
+			// staking prepares first page of exposures.
+			roll_to(export_starts_at, false);
+			assert_eq!(ElectionProvider::current_phase(), Phase::Export(export_starts_at));
+
+			// staking prepares second page of exposures.
+			roll_to(election_prediction() - 2, false);
+			assert_eq!(ElectionProvider::current_phase(), Phase::Export(export_starts_at));
+
+			// staking prepares third page of exposures.
+			roll_to(election_prediction() - 1, false);
+
+			// election successfully, round & era progressed.
+			assert_eq!(ElectionProvider::current_phase(), Phase::Off);
+			assert_eq!(ElectionProvider::current_round(), 1);
+			assert_eq!(Staking::current_era(), Some(1));
+		})
+	}
+}