中文请点击这里

Sector Repair

Filecoin may face sector data loss during sealing or mining, and then destroy PreCommit pre-pledged FILs, or terminate sectors with a maximum loss of 90 days of sector reward. Sector repair can repair the lost files to reduce or avoid the loss.

Causes of sector loss

1. Bad storage disk

Miners are forced to use bare disks for storage to reduce costs and improve their competitiveness in order to reduce packaging costs, and will often use bare disks directly for sector storage. 16T disk, can store more than 130 sectors of 32GiB, if the hard disk is damaged, data can not be recovered and have to terminate the sector, the maximum loss is 90-day of average revenue of sectors across the network.

2. Bad NVMe cache disk

In this case, there are 2 states of the sector that can cause a loss.

• A sector that has submitted a PreCommit message, but has not submitted a ProveCommit message within 30 days will have its PreCommit pre-pledge FIL destroyed.

• Setting FinalizeEarly=false, using ProveCommit first before dropping to storage, is equivalent to losing a sector requiring termination of the sector.

How sector repair works

From the reasons can be analyzed, all sectors that need to be recovered are sectors that have already been submitted PreCommit message, once the data is lost, only the original sealing data can be reassembled and resealing the sectors .

p1o, err := ffi.SealPreCommitPhase1(
    sector.ProofType,
    paths.Cache,
    paths.Unsealed,
    paths.Sealed,
    sector.ID.Number,
    sector.ID.Miner,
    ticket,
    ID.Number, sector.ID.Miner, ticket, pieces,
)

p2...

Resealing the CC sector (without deals), pieces is just generated using the official default method, Only the ticket and ProofType need to be obtained additionally

This is the PreCommit message body recorded on the chain.

{
  "RegisteredProof": 9,
  "SectorNumber": 1322006,
  "SealedCid": "bagboea4b5abcasqanjadumno7blvgx4k5pk765cki6vurnpgs2q3trt2trkznhj3",
  "SealRandEpoch": 925221,
  "DealIds": [],
  "Expiration": 2480426,
  "ReplaceCapacity": false,
  "ReplaceSectorDeadline": 0,
  "ReplaceSectorPartition": 0,
  "ReplaceSector": 0
}

ProofType is different for different network versions and sector sizes, detailed code. It is more convenient to use the RegisteredProof directly in the message body.

Using the SealRandEpoch in the precommit message body as the sector's ticketEpoch, the random number is extracted again from the chain server.

ticket, err := fullNodeApi.StateGetRandomnessFromTickets(ctx, crypto.DomainSeparationTag_SealRandomness, ticketEpoch, buf.Bytes(), ts.Key ())
if err ! = nil {
    return nil, nil, err
}

Compare the SealedCid in the PreCommit message body recorded on the chain with the result of the repair procedure PreCommit2 calculation storage.SectorCids, if the result cid is the same the repair is successful!

Install Go

To build filecoin-sealer-recover, you need to install Go 1.16.4 or higher:

wget -c https://golang.org/dl/go1.16.4.linux-amd64.tar.gz -O - | sudo tar -xz -C /usr/local

Build

1. To build you need to download some Go modules. These are usually hosted on Github, which has a low bandwidth from China. To work around this, use a local proxy before running by setting the following variables.

export GOPROXY=https://goproxy.cn

2. Depending on your CPU model, you will want to export additional environment variables:

a. If you have an AMD Zen or Intel Ice Lake CPU (or later), enable the use of SHA extensions by adding these two environment variables:

export RUSTFLAGS="-C target-cpu=native -g"
export FFI_BUILD_FROM_SOURCE=1

See the Filecoin FFI section for more details about this process.

b. Some older Intel and AMD processors without the ADX instruction support may panic with illegal instruction errors. To solve this, add the CGO_CFLAGS environment variable:

export CGO_CFLAGS_ALLOW="-D__BLST_PORTABLE__"
export CGO_CFLAGS="-D__BLST_PORTABLE__"

c. By default, a multicore-sdr option is used in the proofs library. This feature is also used in FFI unless explicitly disabled. To disable building with the multicore-sdr dependency, set FFI_USE_MULTICORE_SDR to 0:

 export FFI_USE_MULTICORE_SDR=0

d.An experimental gpu option using CUDA can be used in the proofs library. This feature is disabled by default (opencl is the default, when FFI_USE_GPU=1 is set.). To enable building with the gpu CUDA dependency, set FFI_USE_CUDA=1 when building from source.

export FFI_USE_CUDA=1

3. Build and install

# Sector recovery does not distinguish between mainnet or calibnet
make clean all

sudo make install

Install sealer-recover to /usr/local/bin

4. Once the installation is complete, use the following command to ensure that the installation is successful for the correct network.

sealer-recover --version

Usage

1. Specify miner number and sector list, export sector information file:

export FULLNODE_API_INFO=...:/ip4/10.10.10.1/tcp/1234/http
sealer-recover export --miner=f01000 1 2 3 4 5 6

Generate the sector recovery metadata file sectors-recovery-f01000.json in the current directory.

Note: To read the sector information, the lotus daemon needs to have a state tree that is old enough to store it, and the full node is the best.

2. Start the recovery program and execute the list of sectors that the machine performs recovery:

export FIL_PROOFS_USE_MULTICORE_SDR=1
export FIL_PROOFS_MAXIMIZE_CACHING=1
export FIL_PROOFS_USE_GPU_COLUMN_BUILDER=1
export FIL_PROOFS_USE_GPU_TREE_BUILDER=1

sealer-recover recover \
    --sectors-recovery-metadata=./sectors-recovery-f01000.json \
    --parallel=6 \ 
    --sealing-result=/sector \ 
    --sealing-temp=/temp \
    2 4 5

Tools and Parameter Description

Export sector metadata tool:

sealer-recover export [command options] [sectorNum1 sectorNum2 ...]

Parameters	Meaning	Remarks
miner	The miner number of the sectors to be repaired	Required

Sector recovery tool:

sealer-recover recover [command options] [sectorNum1 sectorNum2 ...]

Parameters	Meaning	Remarks
sectors-recovery-metadata	specify the metadata file for the sectors recovery	Required
parallel	Number of parallel sectors to be repaired for sector p1, refer to core count for setting	default: 1
sealing-result	path to the repaired sector product	default: ~/sector
sealing-temp	Intermediate product path for repair process, large space required, NVMe disk recommended	Default: ~/temp Minimum space: 32GiB # > 512GiB! 64GiB # > 1024GiB!

Optimization

Use your own modified lotus for packing, which optimizes the repair speed

TODO

• Support for sectors with deals, retrieving the deals by the deals id of the chain and regenerating the pieces.

Contributing

PRs, bug reports and issue suggestions are welcome! For major changes, please ask questions in issues first so that compatibility and benefits can be discussed.

Other

• FILFrog
• Lotus CloudC2

Contact us

• Slack @Plus-FrogHub

• Email： [email protected]

• Wechat

License

Licensed under Apache 2.0