Initial RFP: "A website and API service that can list all CIDs along with their current status in the latest state tree. The page could also support queries by CID or miner. One option would be to build 1 long table that shows each miner x sectors they are storing x state as a colored indicator: green - good | grey - capacity | red - failing."
The benefits of the CID checker and Storage Oracle:
-
for a User storing own files: the CID checker is the only service that allows you to quickly check information about your file by its CID or the Deal ID. In addition, you can find information about other CIDs stored by a chosen Miner.
-
for a Developer: you can use this service as a Storage Oracle providing the extended range of data related to a chosen CID, a Deal, or a Miner from the latest state tree.
What it brings to you as a Developer in addition to the existing Lotus API:
-
A simple REST API service. The CID Checker as a service provides two REST endpoints, where you need to specify a single search criteria to get the full CID summary (when the Lotus JSON RPC requires complicated JSON requests).
-
It combines responses of several JSON RPC methods (StateMarketDeals, StateMinerSectors, StateGetActor, ChainReadObj) into a single convenient summary (database).
-
The back-end automates the process of fetching the new data, which requires thousands of JSON RCP calls for every loop.
-
Effective and reliable Mongo DB database with indexes provides fast paginated search by Deal ID, Piece CID, Miner ID.
-
UI provides rich details on deals. For the end user it doesn't require any developer skills to quickly get information on deals.
Software requirements specification (the SRS) is here
Project Roadmap is here
Project management board is here
If you are a total beginner to this, start here!
Use remote CID checker service:
- Navigate to the website: www.filecoin.tools
- See the list of all piece CIDs and related information:
- Deal ID
- Miner ID
- Sector number
- Status
- Search a record by a Piece CID, Deal ID, or Miner ID
- Click on a chosen record and see other related details
If you are running an own Filecoin (Lotus) node:
Install the CID checker to your Filecoin (Lotus) node: see deployment instructions below
If you are an application developer:
Two API endpoints available to be used as a Storage Oracle (see the API section below)
The simplest way to deploy the CID checker is doing it with docker-compose.
The CID checker is supposed to query the data from a running Lotus node. To connect the CID checker to a Lutus node we need to specify the Lotus node's address as an environment variable in the docker-compose.yaml:
- CID_LOTUS_RPCURL - URL available through the network and fully synced Lotus node.
Instructions on how to run Lotus node - https://lotu.sh/en+getting-started
Build backend and frontend containers with:
make docker_build
make docker_build_frontend
Run docker-compose:
docker-compose up -d
Backend is a Golang application consist of two parts:
- API server - Gin framework & API handler
- Data harvester - group of lotus processors, each fetches data from Lotus node and store to MongoDB
Two API endpoints available:
Get deals information about all deals from the database.
Get deals by selector: Piece CID, Miner ID and Deal ID.
Both endpoints support pagination.
More detailed information on the API with examples can be found here: https://documenter.getpostman.com/view/6638692/T17AiA6S?version=latest
BlockWatcher periodically checks the network for new blocks. Every time a new block is added, the watcher runs processors.
The Processor calls Lotus StateMarketDeals() method and saves all deals into the "deals" collection.
The goal is to fetch sector information and discover the connection between deals and sectors. Thr Processor calls StateMinerSectors() for each miner and saves it to "sectors" collection.
The goal is to discover sector states. The combination of StateGetActor() and ChainReadObj() is used to fetch miner information form the miner.State struct. The Miner state has two fields - Fault and Recovery - of type abi.BitField with sectors IDs. Sectors in the "sectors" collection are modified based on these values.
- Deals - stores the deals information
- Sectors - stores the sectors information
There is a data abstraction layer in /internals/repos intended for working with the collections.
Application could be configured by config.yaml file or by environment variables.
Lotus configuration:
- CID_LOTUS_RPCURL - URL of fully synced Lotus node
Database configuration:
- CID_DB_CONNECTIONSTRING - MongoDB connection string
- CID_DB_NAME - Database name