Skip to content

Commit

Permalink
fix(content): post and porep section update
Browse files Browse the repository at this point in the history
* post and porep major update

* section sequence restructuring to bring post and porep above sdr proofs

* fixing broken references to old election post

* addresses Nicola's comments 

* glossary update to point to updated section
  • Loading branch information
yiannisbot authored Oct 14, 2020
1 parent 12718a8 commit 2ba3209
Show file tree
Hide file tree
Showing 26 changed files with 149 additions and 335 deletions.
File renamed without changes.
File renamed without changes.
27 changes: 27 additions & 0 deletions content/algorithms/pos/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
---
title: Proof-of-Storage
bookCollapseSection: true
weight: 2
dashboardWeight: 2
dashboardState: reliable
dashboardAudit: wip
dashboardTests: 0
---

# Proof-of-Storage

## Preliminaries

Storage miners in the Filecoin network have to prove that they hold a copy of the data at any given point in time. This is realised through the [Storage Miner Actor](storage_mining#storage_miner_actor) who is the main player in the [Storage Mining](filecoin_mining#storage_mining) subsystem. The proof that a storage miner indeed keeps a copy of the data they have promised to store is achieved through "challenges", that is, by providing answers to specific questions posed by the system. In order for the system to be able to prove that a challenge indeed proves that the miner stores the data, the challenge has to: i) target a random part of the data and ii) be requested at a time interval such that it is not possible, profitable, or rational for the miner to discard the copy of data and refetch it when challenged.

General _Proof-of-Storage_ schemes allow a user to check if a storage provider is storing the outsourced data at the time
of the challenge. How can we use **PoS** schemes to prove that some data was being stored throughout a period
of time? A natural answer to this question is to require the user to repeatedly (e.g. every minute) send
challenges to the storage provider. However, the communication complexity required in each interaction can
be the bottleneck in systems such as Filecoin, where storage providers are required to submit their proofs to
the blockchain network.

To address this question, we introduce a new proof, called _Proof-of-Spacetime_, where a verifier can check if a prover
has indeed stored the outsourced data they committed to over (storage) Space and over a period of Time (hence, the name Spacetime).

Recall that in the Filecoin network, miners are storing data in fixed-size [sectors](filecoin_mining#sector). Sectors are filled with client data agreed through regular deals in the [Storage Market](filecoin_markets#storage_market), through [verified deals](algorithms#verified_clients), or with random client data in case of [Committed Capacity sectors](filecoin_mining#sector).
22 changes: 22 additions & 0 deletions content/algorithms/pos/porep.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
---
title: PoRep
weight: 1
dashboardWeight: 2
dashboardState: reliable
dashboardAudit: wip
dashboardTests: 0
---


# Proof-of-Replication (PoRep)

In order to register a sector with the Filecoin network, the sector has to be sealed. Sealing is a computation-heavy process that produces a unique representation of the data in the form of a proof, called **_Proof-of-Replication_** or PoRep.

The PoRep proof ties together: i) the data itself, ii) the miner actor that performs the sealing and iii) the time when the specific data has been sealed by the specific miner. In other words, if the same miner attempts to seal the same data at a later time, then this will result in a different PoRep proof. Time is included as the blockchain height when sealing took place and the corresponding chain reference is called `SealRandomness`.

Once the proof has been generated, the miner runs a SNARK on the proof in order to compress it and submits the result to the blockchain. This constitutes a certification that the miner has indeed replicated a copy of the data they agreed to store.

The PoRep process includes the following two phases:

- **Sealing preCommit phase 1.** In this phase, PoRep SDR [encoding](sdr#encoding) and [replication](sdr#replication) takes place.
- **Sealing preCommit phase 2.** In this phase, [Merkle proof and tree generation](sdr#merkle-proofs) is performed using the Poseidon hashing algorithm.
90 changes: 90 additions & 0 deletions content/algorithms/pos/post.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
---
title: PoSt
weight: 2
dashboardWeight: 2
dashboardState: reliable
dashboardAudit: wip
dashboardTests: 0
---

# Proof-of-Spacetime (PoSt)

From this point onwards, miners have to prove that they continuously store the data they pledged to store. Proof-of-Spacetime (PoSt) is a procedure during which miners are given cryptographic challenges that can only be correctly answered if the miner is actually storing a copy of the sealed data.

There are two types of challenges (and their corresponding mechanisms) that are realised as part of the PoSt process, namely, _WinningPoSt_ and _WindowPoSt_, each of which serve a different purpose.

- **_WinningPoSt_** is used to prove that the miner has a replica of the data at the specific time when they were asked. A _WinningPoSt_ challenge is issued to a miner only if the miner has been selected by (i.e., wins in) the Secret Leader Election algorithm (of [Expected Consensus](algorithms#expected_consensus)) to mine the next block. The answer to the _WinningPoSt_ challenge has to be submitted within a short deadline, making it impossible for the miner to seal and find the answer to the challenge on demand. This guarantees that at the time of the challenge the miner maintains a copy of the data.
- **_WindowPoSt_** is used as a proof that a copy of the data has been continuously maintained over time. This involves submitting proofs regularly (see details below) and makes it irrational for a miner to _not_ keep a sealed copy of the data (i.e., it is more expensive to seal a copy of the data every time they are asked to submit a WindowPoSt challenge).

In summary, _WinningPoSt_ guarantees that the miner maintains a copy of the data _at some specific point in time_ (i.e., when chosen by the Expected Consensus algorithm to mine the next block), while _WindowPoSt_ guarantees that the miner _continuously maintains a copy of the data over time_.

## Constants & Terminology

Before continuing into more details of the _WinnningPoSt_ and _WindowPoSt_ algorithms, it is worth clarifying the following terms.

- **partition:** a group of 2349 sectors proven simultaneously.
- **proving period:** average period for proving all sectors maintained by a miner (currently set to 24 hours).
- **deadline:** one of multiple points during a proving period when the proofs for some partitions are due.
- **challenge window:** the period immediately before a deadline during which a challenge can be generated by the chain and the requisite proofs computed.
- **miner size:** the amount of proven storage maintained by a single miner actor.

## WinningPoSt

At the beginning of each epoch, a small number of storage miners (up to five, currently) are elected to mine new blocks, by Filecoin's [Expected Consensus](algorithms#expected_consensus) algorithm. Recall that the Filecoin blockchain operates on the basis of _tipsets_, which are groups of up to five blocks. This means that multiple blocks (up to five) can be mined at the same height.

Each of the miners that are elected to mine a block have to submit a proof that they keep a sealed copy of the data which they have included in their proposed block, before the end of the current epoch. Successful submission of this proof is the _WinningPoSt_, which in turn grants the miner the Filecoin Block Reward, as well as the opportunity to charge other nodes fees in order to include their messages in the block. If a miner misses the epoch-end deadline, then the miner misses the opportunity to mine a block and get a Block Reward. No penalty is incurred in this case.

Recall, that the probability of a storage miner being elected to mine a block is governed by Filecoin's [Expected Consensus](algorithms#expected_consensus) algorithm and guarantees that miners will be chosen (on expectation) proportionally to their _Quality Adjusted Power_ in the network, as reported in the power table `WinningPoStSectorLookback` epochs before the election.

## WindowPoSt

WindowPoSt is the mechanism by which the commitments made by storage miners are audited. In _WindowPoSt_ every 24-hour period is called a _"proving period"_ and is broken down into a series of 30min, non-overlapping _deadlines_, making a total of 48 deadlines within any given 24-hour proving period. Every miner must demonstrate availability of all claimed sectors on a 24hr basis. Constraints on individual proof computations limit a single proof to 2349 sectors (a partition), with 10 challenges each.

In particular, the sectors that a miner has pledged to store are: i) assigned to _deadlines_, and ii) grouped in _partitions_. It is important to highlight that although sectors are assigned to deadlines, sectors are proven in partitions - not individually. In other words, upon every deadline, a miner has to prove a whole partition.

For each partition, the miner will have to produce a SNARK-compressed proof and publish it to the blockchain as a message in a block. This proves that the miner has indeed stored the pledged sector. In this way, _every sector of pledged storage is audited (as part of the partition it belongs to) at least once in any 24-hour period_, and a permanent, verifiable, and public record attesting to each storage miner’s continued commitment is kept.

It naturally follows that the more the sectors a miner has pledged to store, the more the partitions of sectors that the miner will need to prove per deadline. This requires ready access to sealed copies of each of the challenged sectors and makes it irrational for the miner to seal data every time they need to provide a WindowPoSt proof.

The Filecoin network expects constant availability of stored files. Failing to submit WindowPoSt for a sector will result in a fault, and the storage miner supplying the sector will be slashed – that is, a portion of their [pledge collateral](filecoin_mining#miner_collaterals) will be forfeited, and their storage power will be reduced (see [Storage Power Consensus](filecoin_blockchain#storage_power_consensus).

## Design

Each miner actor is allocated a 24-hr proving period at random upon creation. This proving period is divided into 48 non-overlapping half-hour deadlines. Each sector is assigned to one of these deadlines when proven to the chain, i.e., when `ProveCommit` completes and never changes deadline. The sets of sectors due at each deadline is recorded in a collection of 48 bitfields.

Generally, sectors are first allocated to fill any deadline up to the next whole-partition multiple of (2349) sectors; next a new partition is started on the deadline with the fewest partitions. If all deadlines have the same number of sectors, a new partition is opened at deadline 0.

The per-deadline sector sets are set at the beginning of each proving period as proving set bitfields and never change. The sector IDs are then (logically) divided sequentially into partitions, and the partitions across all deadlines for the miner logically numbered sequentially. Thus, a sector may move between partitions at the same deadline as other sectors fault or expire, but never changes deadline.

If a miner adds 48 partitions worth of sectors (~3.8 PiB), they will have one partition due for each deadline. When a miner has more than 48 partitions, some deadlines will have multiple partitions due at the same deadline. The proofs (i.e., one SNARK proof per partition) for these simultaneous partitions are expected to be computed and submitted together in a single message, at least up to 10-20 partitions per message, but can be split arbitrarily between messages (which, however, will cost more gas).

A WindowPoSt proof submission must indicate which deadline it targets and which partition indices the proofs represent for that particular deadline. The actor code receiving a submission maps the partition numbers through the deadline’s proving set bitfields to obtain the sector numbers. Faulty sectors are masked from the proving set by substituting a non-faulty sector number. The actor records successful proof verification for each of the partitions in a bitfield of partition indices (or records nothing if verification fails).

There are currently three types of Faults, the _Declared Fault_, the _Detected Fault_ and the _Skipped Fault_. They are discussed in more detail as part of the [Storage Mining subystem](storage_mining#faults-penalties-and-fee-debt).

Summarising:

- A miner maintains its sectors *active* by generating Proofs-of-Spacetime (PoSt) and submit `miner.SubmitWindowedPoSt` for their sectors in a timely manner.
- A WindowPoSt proves that sectors are persistently stored through time.
- Each miner proves all of its sectors once per *proving period*; each sector must be proven by a particular time called _deadline_.
- A *proving period* is a period of `WPoStProvingPeriod` epochs in which a `Miner` actor is scheduled to prove its storage.
- A *proving period* is evenly divided in `WPoStPeriodDeadlines` *deadlines*.
- Each miner has a different start of proving period `ProvingPeriodStart` that is assigned at `Power.CreateMiner`.
- A *deadline* is a period of `WPoStChallengeWindow` epochs that divides a proving period.
- Sectors are assigned to a deadline on `miner.ProveCommitSector` and will remain assigned to it throughout their lifetime.
- In order to prove that they continuously store a sector, a miner must submit a `miner.SubmitWindowedPoSt` for each deadline.
- Sectors are assigned to partitions. A partition is a set of sectors that is not larger than the Seal Proof allowed number of sectors `sp.WindowPoStPartitionSectors`.
- Sectors are assigned to a partition at `miner.ProveCommitSector` and they can be re-arranged via `CompactPartitions`.
- Partitions are a by-product of our current proof mechanism. There is a limit in the number of sectors (`sp.WindowPoStPartitionSectors`) that can be proven in a single SNARK proof. If more than this amount is required to be proven, more than one SNARK proof is required, given that each SNARK proof represents a partition.


There are four relevant epochs associated to a deadline, shown in the table below:

| Name | Distance from `Open` | Description |
|---------------|---------------------------|-------------------------------------------------------------------------------------------------------------------------------|
| `Open` | `0` | Epoch from which a PoSt Proof for this deadline can be submitted. |
| `Close` | `WPoStChallengeWindow` | Epoch after which a PoSt Proof for this deadline will be rejected. |
| `FaultCutoff` | `-FaultDeclarationCutoff` | Epoch after which a `miner.DeclareFault` and `miner.DeclareFaultRecovered` for sectors in the upcoming deadline are rejected. |
| `Challenge` | `-WPoStChallengeLookback` | Epoch at which the randomness for the challenges is available. |


24 changes: 0 additions & 24 deletions content/algorithms/post/_index.md

This file was deleted.

Loading

0 comments on commit 2ba3209

Please sign in to comment.