fix(content): storage mining update

This PR updates the Storage Mining subsystem excluding references to old algorithmics and mechanics and including updated content. * touches several parts of description of storage mining subsystem * update to filecoin mining index * standardize naming and minor fixes * remove miner post state * Apply suggestions from code review Co-authored-by: Irene <[email protected]> * updates faults description * new update on skipped fault description * skipped fault clarification. * adding deadlines table * removes future work and figures * changes subsection structure Co-authored-by: zixuanzh <[email protected]> Co-authored-by: Irene <[email protected]>
filecoin-project · Oct 6, 2020 · 0d4ebc3 · 0d4ebc3
1 parent 873ee5d
commit 0d4ebc3
Show file tree

Hide file tree

Showing 5 changed files with 103 additions and 64 deletions.
diff --git a/content/systems/filecoin_mining/_index.md b/content/systems/filecoin_mining/_index.md
@@ -3,25 +3,26 @@ title: Storage Mining
 bookCollapseSection: true
 weight: 6
 dashboardWeight: 2
-dashboardState: wip
+dashboardState: reliable
 dashboardAudit: wip
 dashboardTests: 0
 ---
 
-# Storage Mining System - proving storage for producing blocks
+# Storage Mining
 
 The Storage Mining System is the part of the Filecoin Protocol that deals with storing Client's
-data, producing proof artifacts that demonstrate correct storage behavior, and managing the work
-involved.
+data and producing proof artifacts that demonstrate correct storage behavior.
+
+Storage Mining is one of the most central parts of the Filecoin protocol overall, as it provides all the required consensus algorithms based on proven _storage power_ in the network. Miners are selected to mine blocks and extend the blockchain based on the storage power that they have committed to the network. Storage is added in unit of sectors and sectors are promises to the network that some storage will remain for a promised duration. In order to participate in Storage Mining, the storage miners have to: i) Add storage to the system, and ii) Prove that they maintain a copy of the data they have agreed to throughout the sector's lifetime.
 
 Storing data and producing proofs is a complex, highly optimizable process, with lots of tunable
 choices. Miners should explore the design space to arrive at something that (a) satisfies protocol
 and network-wide constraints, (b) satisfies clients' requests and expectations (as expressed in
-`Deals`), and \(c) gives them the most cost-effective operation. This part of the Filecoin Spec
+`Deals`), and (c) gives them the most cost-effective operation. This part of the Filecoin Spec
 primarily describes in detail what MUST and SHOULD happen here, and leaves ample room for
 various optimizations for implementers, miners, and users to make. In some parts, we describe
 algorithms that could be replaced by other, more optimized versions, but in those cases it is
 important that the **protocol constraints** are satisfied. The **protocol constraints** are
-spelled out in clear detail (an unclear, unmentioned constraint is a "spec error").  It is up
+spelled out in clear detail.  It is up
 to implementers who deviate from the algorithms presented here to ensure their modifications
-satisfy those constraints, especially those relating to protocol security.
+satisfy those constraints, especially those relating to protocol security.
diff --git a/content/systems/filecoin_mining/sector/_index.md b/content/systems/filecoin_mining/sector/_index.md
@@ -1,6 +1,6 @@
 ---
 title: Sector
-weight: 2
+weight: 1
 bookCollapseSection: true
 dashboardWeight: 2
 dashboardState: stable

diff --git a/content/systems/filecoin_mining/storage_mining/_index.md b/content/systems/filecoin_mining/storage_mining/_index.md
@@ -1,65 +1,114 @@
 ---
 title: Storage Miner
+weight: 2
 bookCollapseSection: true
-weight: 1
 dashboardWeight: 2
-dashboardState: incorrect
+dashboardState: reliable
 dashboardAudit: wip
 dashboardTests: 0
 ---
 
 # Storage Miner
 
-## Filecoin Storage Mining Subsystem
+## Storage Mining Subsystem
 
 The Filecoin Storage Mining Subsystem ensures a storage miner can effectively commit storage to the Filecoin protocol in order to both:
 
 - Participate in the Filecoin [Storage Market](storage_market) by taking on client data and participating in storage deals.
-- Participate in Filecoin [Storage Power Consensus](storage_power_consensus), verifying and generating blocks to grow the Filecoin blockchain and earning block rewards and fees for doing so.
+- Participate in Filecoin [Storage Power Consensus](storage_power_consensus) by verifying and generating blocks to grow the Filecoin blockchain and earning block rewards and fees for doing so.
 
 The above involves a number of steps to putting on and maintaining online storage, such as:
 
-- Committing new storage (see Sealing and PoRep)
-- Continuously proving storage (see [Election PoSt](election_post))
-- Declaring storage faults and recovering from them.
+- Committing new storage (see [Sector](sector), [Sector Sealing](sector#sealing) and [PoRep](sdr))
+- Continuously proving storage (see [WinningPoSt](expected_consensus#winning-a-block) and Window PoSt)
+- Declaring [storage faults](sector#sector-faults) and recovering from them.
 
-### Sector Types
 
-There are two types of sectors, Regular Sectors with storage deals in them and Committed Capacity (CC) Sectors with no deals. All sectors require an expiration epoch that is declared upon PreCommit and sectors are assigned a StartEpoch at ProveCommit. Start and Expiration epoch collectively define the lifetime of a Sector. Length and size of active deals in a sector's lifetime determine the `DealWeight` of the sector. `SectorSize`, `Duration`, and `DealWeight` statically determine the power assigned to a sector that will remain constant throughout its lifetime. More details on cost and reward for different sector types will be announced soon.
+## Filecoin Proofs
 
-### Sector States
+### Proof of Replication
 
-When managing their storage [Sector](sector) as part of Filecoin mining, storage providers will account for where in the [Mining Cycle](mining_cycle) their sectors are. For instance, has a sector been committed? Does it need a new PoSt? Most of these operations happen as part of cycles of chain epochs called `Proving Period`s each of which yield high confidence that every miner in the chain has proven their power (see [Election PoSt](election_post)).
+A [Proof of Replication (PoRep)](sdr) is a proof that a Miner has correctly generated a unique _replica_ of some underlying data.
 
-There are three states that an individual sector can be in:
+In practice, the underlying data is the _raw data_ contained in an Unsealed Sector, and a PoRep is a SNARK proof that the _sealing process_ was performed correctly to produce a Sealed Sector (See [Sealing a Sector](#Sealing-a-Sector)).
 
-- `PreCommit` when a sector has been added through a PreCommit message.
-- `Active` when a sector has been proven through a ProveCommit message and when a sector's TemporaryFault period has ended.
-- `TemporaryFault` when a miner declares fault on a particular sector.
+It is important to note that the replica should not only be unique to the miner, but also to the time when a miner has actually created the replica, i.e., sealed the sector. This means that if the same miner produces a sealed sector out of the same raw data twice, then this would count as a different replica.
 
-Sectors enter `Active` from `PreCommit` through a ProveCommit message that serves as the first proof for the sector. PreCommit requires a PreCommit deposit which will be returned upon successful and timely ProveCommit. However, if there is no matching ProveCommit for a particular PreCommit message, the deposit will be burned at PreCommit expiration.
+When Miners commit to storing data, they must first produce a valid Proof of Replication.
 
-A particular sector enters `TemporaryFault` from `Active` through `DeclareTemporaryFault` with a specified period. Power associated with the sector will be lost immediately and miner needs to pay a `TemporaryFaultFee` determined by the power suspended and the duration of suspension. At the end of the declared duration, faulted sectors automatically regain power and enter `Active`. Miners are expected to prove over this recovered sector. Failure to do so may result in failing ElectionPoSt or `DetectedFault` from failing SurprisePoSt. 
+### Proof of Spacetime
 
-![Sector State Machine](diagrams/sector_state_machine.dot)
+A [Proof of Spacetime (aka PoSt)](post) is a long-term assurance of a Miner's continuous storage of their Sectors' data. _This is not a single proof,_ but a collection of proofs the Miner has submitted over time. Periodically, a Miner must add to these proofs by submitting a **WindowPoSt**:
+* Fundamentally, a WindowPoSt is a collection of merkle proofs over the underlying data in a Miner's Sectors.
+* WindowPoSts bundle proofs of various leaves across groups of Sectors (called **Partitions**).
+* These proofs are submitted as a single SNARK.
 
-![Sector State Machine Legend](diagrams/sector_state_machine_legend.dot)
+The historical and ongoing submission of WindowPoSts creates assurance that the Miner has been storing, and continues to store the Sectors they agreed to store in the storage deal.
 
-#### Miner PoSt State
+Once a Miner successfully adds and ProveCommits a Sector, the Sector is assigned to a Deadline: a specific window of time during which PoSts must be submitted. The day is broken up into 48 individual Deadlines of 30 minutes each, and ProveCommitted Sectors are assigned to one of these 48 Deadlines.
+* PoSts may only be submitted for the currently-active Deadline. Deadlines are open for 30 minutes, starting from the Deadline's "Open" epoch and ending at its "Close" epoch.
+* PoSts must incorporate randomness pulled from a random beacon. This randomness becomes publicly available at the Deadline's "Challenge" epoch, which is 20 epochs prior to its "Open" epoch.
+* Deadlines also have a `FaultCutoff` epoch, 70 epochs prior to its "Open" epoch. After this epoch, Faults can no longer be declared for the Deadline's Sectors.
 
-`MinerPoStState` keeps track of a miner's state in responding to PoSt and there are three states in `MinerPoStState`:
+## Miner Accounting
 
-- `OK` miner has passed either a ElectionPoSt or a SurprisePoSt sufficiently recently.
-- `Challenged` miner has been selected to prove its storage via SurprisePoSt and is currently in the Challenged state
-- `DetectedFault` miner has failed at least one SurprisePoSt, indicating that all claimed storage may not be proven. Miner has lost power on its sector and recovery can only proceed by a successful response to a subsequent SurprisePoSt challenge, up until the limit of number of consecutive failures.
+A Miner's financial gain or loss is affected by the following three actions:
+1. Miners deposit tokens to act as collateral for their `PreCommitted` and `ProveCommitted` Sectors
+2. Miners earn tokens from block rewards, when they are elected to mine a new block and extend the blockchain.
+3. Miners lose tokens if they fail to prove storage of a sector and are given penalties as a result.
 
-`DetectedFault` is a miner-wide PoSt state when all sectors are considered inactive. All power is lost immediately and pledge collateral is slashed. If a miner remains in `DetectedFault` for more than MaxConsecutiveFailures, all sectors will be terminated, both power and market actors will be notified for slashing and return of client deal collateral.
+### Balance Requirements
 
-`ProvingSet` consists of sectors that miners are required to generate proofs against and is what counts towards miners' power. In other words, `ProvingSet` is a set of all `Active` sectors for a particular miner. `ProvingSet` is only relevant when the miner is in OK stage of its `MinerPoStState`. When a miner is in the `Challenged` state, `ChallengedSectors` specify the list of sectors to be challenged which is the `ProvingSet` before the challenge is issued thus allowing more sectors to be added while it is in the `Challenged` state.
+A Miner's token balance MUST cover ALL of the following:
+* **PreCommit Deposits**: When a Miner PreCommits a Sector, they must supply a "precommit deposit" for the Sector, which acts as collateral. If the Sector is not ProveCommitted on time, this deposit is removed and burned.
+* **Initial Pledge**: When a Miner ProveCommits a Sector, they must supply an "initial pledge" for the Sector, which acts as collateral. If the Sector is terminated, this deposit is removed and burned along with rewards earned by this sector up to a limit.
+* **Locked Funds**: When a Miner receives tokens from block rewards, the tokens are locked and added to the Miner's vesting table to be unlocked linearly over some future epochs.
 
-Miners can call ProveCommit to commit a sector and add to their Claimed Power. However, a miner's Nominal Power and Consensus Power will be zero when it is in either Challenged or DetectedFault state. Note also that miners can call DeclareTemporaryFault when they are in Challenged or DetectedFault state. This does not change the list of  sectors that are currently challenged which is a snapshot of all active sectors (ProvingSet) at the time of challenge.
+### Faults, Penalties and Fee Debt
 
-![Miner PoSt State Machine](diagrams/miner_post_state_machine.dot)
+**Faults**
 
-![Miner PoSt State Machine Legend](diagrams/miner_post_state_machine_legend.dot)
+A Sector's PoSts must be submitted on time, or that Sector is marked "faulty." There are three types of faults:
+* **Declared Fault**: When the Miner explicitly declares a Sector "faulty" _before_ its Deadline's FaultCutoff. Recall that `WindowPoSt` proofs are submitted per partition for a specific `ChallengeWindow`. A miner has to declare the sector as faulty before the `ChallengeWindow` for the particular partition opens. Until the sectors are recovered they will be masked from proofs in subsequent proving periods.
+* **Detected Fault**: Partitions of sectors without PoSt proof verification records, which have not been declared faulty before the `FaultCutoff` epoch's deadline are marked as detected faults.
+* **Skipped Fault**: If a sector is currently in active or recovering state and has not been declared faulty before, but the miner's PoSt submission does not include a proof for this sector, then this is a "skipped fault" sector (also referred to as "skipped undeclared fault"). In other words, when a miner submits PoSt proofs for a partition but does not include proofs for some sectors in the partition, then these sectors are in "skipped fault" state. This is in contrast to the "detected fault" state, where the miner does not submit a PoSt proof for any section in the partition at all. The skipped fault is helpful in case a sector becomes faulty after the `FaultCutoff` epoch.
 
+Note that the "skipped fault" allows for sector-wise fault penalties, as compared to partition-wide faults and penalties, as is the case with "detected faults".
+
+**Deadlines**
+
+A *deadline* is a period of `WPoStChallengeWindow` epochs that divides a proving period.
+Sectors are assigned to a deadline on `miner.ProveCommitSector` and will remain assigned to it throughout their lifetime. Recall that Sectors are also assigned to a partition.
+
+A miner must submit a `miner.SubmitWindowedPoSt` for each deadline.
+
+There are four relevant epochs associated to a deadline:
+
+| Name          | Distance from `Open`      | Description                                                                                                                   |
+|---------------|---------------------------|-------------------------------------------------------------------------------------------------------------------------------|
+| `Open`        | `0`                       | Epoch from which a PoSt Proof for this deadline can be submitted.                                                             |
+| `Close`       | `WPoStChallengeWindow`    | Epoch after which a PoSt Proof for this deadline will be rejected.                                                            |
+| `FaultCutoff` | `-FaultDeclarationCutoff` | Epoch after which a `miner.DeclareFault` and `miner.DeclareFaultRecovered` for sectors in the upcoming deadline are rejected. |
+| `Challenge`   | `-WPoStChallengeLookback` | Epoch at which the randomness for the challenges is available.                                                                |
+
+
+**Fault Recovery**
+
+Regardless of how a fault first becomes known (declared, skipped, detected), the sector stays faulty and is excluded from future proofs until the miner explicitly declares it recovered. The declaration of recovery restores the sector to the proving set at the start of the subsequent proving period. When a PoSt for a just-recovered sector is received, power for that sector is restored.
+
+**Penalties**
+
+A Miner may accrue penalties for many reasons:
+* **PreCommit Expiry Penalty**: Occurs if a Miner fails to `ProveCommit` a PreCommitted Sector in time. This happens the first time that a miner declares that it proves a sector and falls into the PoRep consensus.
+* **Undeclared Fault Penalty**: Occurs if a Miner fails to submit a PoSt for a Sector on time. Depending on whether the "Skipped Fault" option is implemented, this penalty applies to either a sector or a whole partition.
+* **Declared Fault Penalty**: Occurs if a Miner fails to submit a PoSt for a Sector on time, but they declare the Sector faulty before the system finds out (in which case the fault falls in the "Undeclared Fault Penalty" above). **This penalty fee should be lower than the undeclared fault penalty**, in order to incentivize Miners to declare faults early.
+* **Ongoing Fault Penalty**: Occurs every Proving Period a Miner fails to submit a PoSt for a Sector.
+* **Termination Penalty**: Occurs if a Sector is terminated before its expiration.
+* **Consensus Fault Penalty**: Occurs if a Miner commits a consensus fault and is reported.
+
+When a Miner accrues penalties, the amount penalized is tracked as "Fee Debt." If a Miner has Fee Debt, they are restricted from certain actions until the amount owed is paid off. Miners with Fee Debt may not:
+* PreCommit new Sectors
+* Declare faulty Sectors "recovered"
+* Withdraw balance
+
+Faults are implied to be "temporary" - that is, a Miner that temporarily loses internet connection may choose to declare some Sectors for their upcoming proving period as faulty, because the Miner knows they will eventually regain the ability to submit proofs for those Sectors. This declaration allows the Miner to still submit a valid proof for their Deadline (minus the faulty Sectors). This is very important for Miners, as missing a Deadline's PoSt entirely incurs a high penalty.