Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Staking runtime apis to help with paged reward payouts #426

Closed
4 tasks done
Ank4n opened this issue Feb 22, 2023 · 0 comments · Fixed by #4301
Closed
4 tasks done

Staking runtime apis to help with paged reward payouts #426

Ank4n opened this issue Feb 22, 2023 · 0 comments · Fixed by #4301
Assignees
@Ank4n

Comments

@Ank4n
Copy link
Contributor

Ank4n commented Feb 22, 2023 •
edited
Loading

Once we start supporting paged reward payouts (#1189), we should expose runtime apis for staking pallet to help with the following use cases.

  • As a nominator, I want to know which page my exposure belongs to for a given era, so I could payout rewards only for that page. (this is not possible on on-chain)
  • As a validator, I want to know how many pages of exposure I have so I can payout all my nominators.
  • Api to get full exposure of a validator potentially unsafe as one api will call multiple storage items, also potentially bad practice to return such a huge chunk of data via runtime api, better for front ends to query storage directly.
  • Api to check if there is pending pages of rewards to be claimed for an era.
This was referenced Feb 22, 2023
Closed
Closed
Open
@juangirini juangirini transferred this issue from paritytech/substrate Aug 24, 2023
@Ank4n Ank4n mentioned this issue Aug 26, 2023
Merged
11 tasks
@Ank4n Ank4n self-assigned this Nov 1, 2023
Ank4n added a commit that referenced this issue Nov 1, 2023
@Ank4n @pepoviola
[NPoS] Paging reward payouts in order to scale rewardable nominators (#…
00b85c5 
…1189)

helps #439.
closes #473.

PR link in the older substrate repository:
paritytech/substrate#13498.

# Context
Rewards payout is processed today in a single block and limited to
`MaxNominatorRewardedPerValidator`. This number is currently 512 on both
Kusama and Polkadot.

This PR tries to scale the nominators payout to an unlimited count in a
multi-block fashion. Exposures are stored in pages, with each page
capped to a certain number (`MaxExposurePageSize`). Starting out, this
number would be the same as `MaxNominatorRewardedPerValidator`, but
eventually, this number can be lowered through new runtime upgrades to
limit the rewardeable nominators per dispatched call instruction.

The changes in the PR are backward compatible.

## How payouts would work like after this change
Staking exposes two calls, 1) the existing `payout_stakers` and 2)
`payout_stakers_by_page`.

### payout_stakers
This remains backward compatible with no signature change. If for a
given era a validator has multiple pages, they can call `payout_stakers`
multiple times. The pages are executed in an ascending sequence and the
runtime takes care of preventing double claims.

### payout_stakers_by_page
Very similar to `payout_stakers` but also accepts an extra param
`page_index`. An account can choose to payout rewards only for an
explicitly passed `page_index`.

**Lets look at an example scenario**
Given an active validator on Kusama had 1100 nominators,
`MaxExposurePageSize` set to 512 for Era e. In order to pay out rewards
to all nominators, the caller would need to call `payout_stakers` 3
times.

- `payout_stakers(origin, stash, e)` => will pay the first 512
nominators.
- `payout_stakers(origin, stash, e)` => will pay the second set of 512
nominators.
- `payout_stakers(origin, stash, e)` => will pay the last set of 76
nominators.
...
- `payout_stakers(origin, stash, e)` => calling it the 4th time would
return an error `InvalidPage`.

The above calls can also be replaced by `payout_stakers_by_page` and
passing a `page_index` explicitly.

## Commission note
Validator commission is paid out in chunks across all the pages where
each commission chunk is proportional to the total stake of the current
page. This implies higher the total stake of a page, higher will be the
commission. If all the pages of a validator's single era are paid out,
the sum of commission paid to the validator across all pages should be
equal to what the commission would have been if we had a non-paged
exposure.

### Migration Note
Strictly speaking, we did not need to bump our storage version since
there is no migration of storage in this PR. But it is still useful to
mark a storage upgrade for the following reasons:

- New storage items are introduced in this PR while some older storage
items are deprecated.
- For the next `HistoryDepth` eras, the exposure would be incrementally
migrated to its corresponding paged storage item.
- Runtimes using staking pallet would strictly need to wait at least
`HistoryDepth` eras with current upgraded version (14) for the migration
to complete. At some era `E` such that `E >
era_at_which_V14_gets_into_effect + HistoryDepth`, we will upgrade to
version X which will remove the deprecated storage items.
In other words, it is a strict requirement that E<sub>x</sub> -
E<sub>14</sub> > `HistoryDepth`, where
E<sub>x</sub> = Era at which deprecated storages are removed from
runtime,
E<sub>14</sub> = Era at which runtime is upgraded to version 14.
- For Polkadot and Kusama, there is a [tracker
ticket](#433) to clean
up the deprecated storage items.

### Storage Changes

#### Added
- ErasStakersOverview
- ClaimedRewards
- ErasStakersPaged

#### Deprecated
The following can be cleaned up after 84 eras which is tracked
[here](#433).

- ErasStakers.
- ErasStakersClipped.
- StakingLedger.claimed_rewards, renamed to
StakingLedger.legacy_claimed_rewards.

### Config Changes
- Renamed MaxNominatorRewardedPerValidator to MaxExposurePageSize.

### TODO
- [x] Tracker ticket for cleaning up the old code after 84 eras.
- [x] Add companion.
- [x] Redo benchmarks before merge.
- [x] Add Changelog for pallet_staking.
- [x] Pallet should be configurable to enable/disable paged rewards.
- [x] Commission payouts are distributed across pages.
- [x] Review documentation thoroughly.
- [x] Rename `MaxNominatorRewardedPerValidator` ->
`MaxExposurePageSize`.
- [x] NMap for `ErasStakersPaged`.
- [x] Deprecate ErasStakers.
- [x] Integrity tests.

### Followup issues
[Runtime api for deprecated ErasStakers storage
item](#426)

---------

Co-authored-by: Javier Viola <[email protected]>
Co-authored-by: Ross Bulat <[email protected]>
Co-authored-by: command-bot <>
This was referenced Nov 1, 2023
Closed
Closed
bgallois pushed a commit to duniter/duniter-polkadot-sdk that referenced this issue Mar 25, 2024
@Ank4n @pepoviola
[NPoS] Paging reward payouts in order to scale rewardable nominators (p…
870ce7f 
…aritytech#1189)

helps paritytech#439.
closes paritytech#473.

PR link in the older substrate repository:
paritytech/substrate#13498.

# Context
Rewards payout is processed today in a single block and limited to
`MaxNominatorRewardedPerValidator`. This number is currently 512 on both
Kusama and Polkadot.

This PR tries to scale the nominators payout to an unlimited count in a
multi-block fashion. Exposures are stored in pages, with each page
capped to a certain number (`MaxExposurePageSize`). Starting out, this
number would be the same as `MaxNominatorRewardedPerValidator`, but
eventually, this number can be lowered through new runtime upgrades to
limit the rewardeable nominators per dispatched call instruction.

The changes in the PR are backward compatible.

## How payouts would work like after this change
Staking exposes two calls, 1) the existing `payout_stakers` and 2)
`payout_stakers_by_page`.

### payout_stakers
This remains backward compatible with no signature change. If for a
given era a validator has multiple pages, they can call `payout_stakers`
multiple times. The pages are executed in an ascending sequence and the
runtime takes care of preventing double claims.

### payout_stakers_by_page
Very similar to `payout_stakers` but also accepts an extra param
`page_index`. An account can choose to payout rewards only for an
explicitly passed `page_index`.

**Lets look at an example scenario**
Given an active validator on Kusama had 1100 nominators,
`MaxExposurePageSize` set to 512 for Era e. In order to pay out rewards
to all nominators, the caller would need to call `payout_stakers` 3
times.

- `payout_stakers(origin, stash, e)` => will pay the first 512
nominators.
- `payout_stakers(origin, stash, e)` => will pay the second set of 512
nominators.
- `payout_stakers(origin, stash, e)` => will pay the last set of 76
nominators.
...
- `payout_stakers(origin, stash, e)` => calling it the 4th time would
return an error `InvalidPage`.

The above calls can also be replaced by `payout_stakers_by_page` and
passing a `page_index` explicitly.

## Commission note
Validator commission is paid out in chunks across all the pages where
each commission chunk is proportional to the total stake of the current
page. This implies higher the total stake of a page, higher will be the
commission. If all the pages of a validator's single era are paid out,
the sum of commission paid to the validator across all pages should be
equal to what the commission would have been if we had a non-paged
exposure.

### Migration Note
Strictly speaking, we did not need to bump our storage version since
there is no migration of storage in this PR. But it is still useful to
mark a storage upgrade for the following reasons:

- New storage items are introduced in this PR while some older storage
items are deprecated.
- For the next `HistoryDepth` eras, the exposure would be incrementally
migrated to its corresponding paged storage item.
- Runtimes using staking pallet would strictly need to wait at least
`HistoryDepth` eras with current upgraded version (14) for the migration
to complete. At some era `E` such that `E >
era_at_which_V14_gets_into_effect + HistoryDepth`, we will upgrade to
version X which will remove the deprecated storage items.
In other words, it is a strict requirement that E<sub>x</sub> -
E<sub>14</sub> > `HistoryDepth`, where
E<sub>x</sub> = Era at which deprecated storages are removed from
runtime,
E<sub>14</sub> = Era at which runtime is upgraded to version 14.
- For Polkadot and Kusama, there is a [tracker
ticket](paritytech#433) to clean
up the deprecated storage items.

### Storage Changes

#### Added
- ErasStakersOverview
- ClaimedRewards
- ErasStakersPaged

#### Deprecated
The following can be cleaned up after 84 eras which is tracked
[here](paritytech#433).

- ErasStakers.
- ErasStakersClipped.
- StakingLedger.claimed_rewards, renamed to
StakingLedger.legacy_claimed_rewards.

### Config Changes
- Renamed MaxNominatorRewardedPerValidator to MaxExposurePageSize.

### TODO
- [x] Tracker ticket for cleaning up the old code after 84 eras.
- [x] Add companion.
- [x] Redo benchmarks before merge.
- [x] Add Changelog for pallet_staking.
- [x] Pallet should be configurable to enable/disable paged rewards.
- [x] Commission payouts are distributed across pages.
- [x] Review documentation thoroughly.
- [x] Rename `MaxNominatorRewardedPerValidator` ->
`MaxExposurePageSize`.
- [x] NMap for `ErasStakersPaged`.
- [x] Deprecate ErasStakers.
- [x] Integrity tests.

### Followup issues
[Runtime api for deprecated ErasStakers storage
item](paritytech#426)

---------

Co-authored-by: Javier Viola <[email protected]>
Co-authored-by: Ross Bulat <[email protected]>
Co-authored-by: command-bot <>
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 8, 2024
@dependabot-preview @serban300
Bump serde_json from 1.0.58 to 1.0.59 (paritytech#426)
db37bab
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 8, 2024
@dependabot-preview @serban300
Bump serde_json from 1.0.58 to 1.0.59 (paritytech#426)
21b11e9
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 9, 2024
@dependabot-preview @serban300
Bump serde_json from 1.0.58 to 1.0.59 (paritytech#426)
8000f37
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 9, 2024
@dependabot-preview @serban300
Bump serde_json from 1.0.58 to 1.0.59 (paritytech#426)
8bbbb0e
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 9, 2024
@dependabot-preview @serban300
Bump serde_json from 1.0.58 to 1.0.59 (paritytech#426)
24e9683
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 9, 2024
@dependabot-preview @serban300
Bump serde_json from 1.0.58 to 1.0.59 (paritytech#426)
f159a73
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 9, 2024
@dependabot-preview @serban300
Bump serde_json from 1.0.58 to 1.0.59 (paritytech#426)
d3f65e0
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 9, 2024
@dependabot-preview @serban300
Bump serde_json from 1.0.58 to 1.0.59 (paritytech#426)
67e4a7f
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 10, 2024
@dependabot-preview @serban300
Bump serde_json from 1.0.58 to 1.0.59 (paritytech#426)
dd36372
serban300 pushed a commit to serban300/polkadot-sdk that referenced this issue Apr 10, 2024
@dependabot-preview @serban300
Bump serde_json from 1.0.58 to 1.0.59 (paritytech#426)
00de668
jonathanudd pushed a commit to jonathanudd/polkadot-sdk that referenced this issue Apr 10, 2024
@dependabot-preview @bkchr
Bump serde_json from 1.0.58 to 1.0.59 (paritytech#426)
31c90f1
@Ank4n Ank4n mentioned this issue Apr 25, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Apr 28, 2024
@Ank4n
[Staking] Runtime api if era rewards are pending to be claimed (#4301)
73b9a83 
closes #426.
related to #1189.

Would help offchain programs to query if there are unclaimed pages of
rewards for a given era.

The logic could look like below

```js
// loop as long as all era pages are claimed.
while (api.call.stakingApi.pendingRewards(era, validator_stash)) {
  api.tx.staking.payout_stakers(validator_stash, era)
}
```
@github-project-automation github-project-automation bot moved this from ⌛️ Sometime-soon to ✅ Done in (Nominated) Proof of Stake Apr 28, 2024
Morganamilo pushed a commit that referenced this issue May 2, 2024
@Ank4n @Morganamilo
[Staking] Runtime api if era rewards are pending to be claimed (#4301)
9c8eef8 
closes #426.
related to #1189.

Would help offchain programs to query if there are unclaimed pages of
rewards for a given era.

The logic could look like below

```js
// loop as long as all era pages are claimed.
while (api.call.stakingApi.pendingRewards(era, validator_stash)) {
  api.tx.staking.payout_stakers(validator_stash, era)
}
```
Ank4n added a commit that referenced this issue May 20, 2024
@Ank4n
[Staking] Runtime api if era rewards are pending to be claimed (#4301)
3c46576 
closes #426.
related to #1189.

Would help offchain programs to query if there are unclaimed pages of
rewards for a given era.

The logic could look like below

```js
// loop as long as all era pages are claimed.
while (api.call.stakingApi.pendingRewards(era, validator_stash)) {
  api.tx.staking.payout_stakers(validator_stash, era)
}
```
This was referenced Jun 5, 2024
Closed
Closed
TarekkMA pushed a commit to moonbeam-foundation/polkadot-sdk that referenced this issue Aug 2, 2024
@Ank4n @TarekkMA
[Staking] Runtime api if era rewards are pending to be claimed (parit…
4c88cb9 
…ytech#4301)

closes paritytech#426.
related to paritytech#1189.

Would help offchain programs to query if there are unclaimed pages of
rewards for a given era.

The logic could look like below

```js
// loop as long as all era pages are claimed.
while (api.call.stakingApi.pendingRewards(era, validator_stash)) {
  api.tx.staking.payout_stakers(validator_stash, era)
}
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Ank4n Ank4n

Labels
None yet
Projects
(Nominated) Proof of Stake
Status: ✅ Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Staking] Runtime api if era rewards are pending to be claimed paritytech/polkadot-sdk
1 participant
@Ank4n