Skip to content

Latest commit

 

History

History

blockchain-indexer

README.md

Klayr Service Blockchain Indexer

The Blockchain Indexer service, in the indexing mode, is primarily responsible to index all the blockchain information, based on the scheduled jobs by the Blockchain Coordinator. In the data service mode, it serves user request queries made via the RESTful API or WebSocket-based RPC calls. It can run both the indexer and data service modes simultaneously, and is enabled by default.

This microservice encapsulates most of the business logic for the Klayr Service API. By default, it only implements the business logic for all the available commands from the Klayr SDK. The applyTransaction and revertTransaction hooks implement the indexing logic and are specific to each available command. The applyTransaction is triggered when processing an included transaction within a new block while (usually) indexing the chain_newBlock event. The revertTransaction hook is triggered when processing an included transaction within a deleted block while processing the chain_deleteBlock event. All the implemented hooks are grouped here. Command specific hooks are always implemented within a single file and are grouped by the module. When interested in extending Klayr Service and implementing hooks for your custom modules, please check the Extending Indexer section below.

Note that this installation instruction is required only for development activities. For a regular Klayr Service user the official documentation is sufficient to run an instance. The global readme file present in the root directory describes running all the microservices at once.

Installation

Prerequisites

Please refer to the README in the project root directory.

Installation

Clone the Klayr Service Repository:

git clone https://github.com/KlayrHQ/klayr-service.git # clone repository
cd klayr-service/services/blockchain-indexer # move into blockchain-indexer microservice directory
yarn install --frozen-lockfile # install required Node.js dependencies

Configuration

To configure the different microservices, there are several environment variables the user can define to customize the configurations.

A list of the most commonly used environment variables is presented below:

  • SERVICE_BROKER: URL of the microservice message broker (Redis).
  • SERVICE_INDEXER_MYSQL: Connection string of the (read/write) primary MySQL instance that the microservice connects to.
  • SERVICE_INDEXER_MYSQL_READ_REPLICA: Connection string of the (read only) replicated MySQL instance that the microservice connects to.
  • SERVICE_MESSAGE_QUEUE_REDIS: URL of the job queue to process the scheduled indexing jobs by the Blockchain Coordinator (Redis).
  • SERVICE_INDEXER_REDIS_VOLATILE: URL of the volatile cache storage (Redis).
  • ENABLE_DATA_RETRIEVAL_MODE: Boolean flag to enable the Data Service mode.
  • ENABLE_INDEXING_MODE: Boolean flag to enable the Data Indexing mode.
  • ENABLE_PERSIST_EVENTS: Boolean flag to permanently maintain the events in the MySQL database.
  • ENABLE_APPLY_SNAPSHOT: Boolean flag to enable initialization of the index with the Klayr Service DB snapshot.
  • DURABILITY_VERIFY_FREQUENCY: Frequency in milliseconds to verify if a block is indexed or rolled-back successfully. By default, it is set to 1.
  • INDEX_SNAPSHOT_URL: URL from where the Klayr Service DB snapshot will be downloaded.
  • ENABLE_SNAPSHOT_ALLOW_INSECURE_HTTP: Boolean flag to enable downloading snapshot from an (unsecured) HTTP URL.
  • KLAYR_STATIC: URL of Klayr static assets.
  • SERVICE_INDEXER_CACHE_REDIS: URL of the cache storage (Redis).
  • ACCOUNT_BALANCE_UPDATE_BATCH_SIZE: Number of accounts for which the balance index is updated at a time. By default, it is set to 1000.
  • INDEX_BLOCKS_QUEUE_SCHEDULED_JOB_MAX_COUNT: Maximum number of jobs (in active and waiting state) allowed in the block indexing queue. By default, it is set to 100000.
  • JOB_INTERVAL_DELETE_SERIALIZED_EVENTS: Job run interval to delete serialized events. By default, it is set to 0.
  • JOB_SCHEDULE_DELETE_SERIALIZED_EVENTS: Job run cron schedule to delete serialized events. By default, it is set to run every 5th minute (*/5 * * * *).
  • JOB_INTERVAL_REFRESH_VALIDATORS: Job run interval to refresh validators cache. By default, it is set to 0.
  • JOB_SCHEDULE_REFRESH_VALIDATORS: Job run cron schedule to refresh validators cache. By default, it is set to run every 5th minute (*/5 * * * *).
  • JOB_INTERVAL_VALIDATE_VALIDATORS_RANK: Job run interval to validate the rank for all the validators. By default, it is set to 0.
  • JOB_SCHEDULE_VALIDATE_VALIDATORS_RANK: Job run cron schedule to validate the rank for all the validators. By default, it is set to run every 15 minutes, and starts at 4 minutes past the hour (4-59/15 * * * *).
  • JOB_INTERVAL_REFRESH_INDEX_STATUS: Job run interval to refresh indexing status. By default, it is set to run every 10 seconds.
  • JOB_SCHEDULE_REFRESH_INDEX_STATUS: Job run cron schedule to refresh indexing status. By default, it is set to ''.
  • JOB_INTERVAL_REFRESH_BLOCKCHAIN_APPS_STATS: Job run interval to refresh blockchain application statistics. By default, it is set to 0.
  • JOB_SCHEDULE_REFRESH_BLOCKCHAIN_APPS_STATS: Job run cron schedule to refresh blockchain application statistics. By default, it is set to run every 15th minute (*/15 * * * *).
  • JOB_INTERVAL_REFRESH_ACCOUNT_KNOWLEDGE: Job run interval to refresh account knowledge. By default, it is set to 0.
  • JOB_SCHEDULE_REFRESH_ACCOUNT_KNOWLEDGE: Job run cron schedule to refresh account knowledge. By default, it is set to run every 15th minute (*/15 * * * *).
  • JOB_INTERVAL_DELETE_FINALIZED_CCU_METADATA: Job run interval to delete finalized CCU metadata. By default, it is set to 0.
  • JOB_SCHEDULE_DELETE_FINALIZED_CCU_METADATA: Job run cron schedule to delete finalized CCU metadata. By default, it is set to run once a day at 02:00 am (0 2 * * *).
  • JOB_INTERVAL_TRIGGER_ACCOUNT_UPDATES: Job run interval to trigger account updates. By default, it is set to 0.
  • JOB_SCHEDULE_TRIGGER_ACCOUNT_UPDATES: Job run cron schedule to trigger account updates. By default, it is set to run every 15th minute (*/15 * * * *).
  • ESTIMATES_BUFFER_BYTES_LENGTH: Transaction buffer bytes to consider when estimating the transaction fees. By default, it is set to 0.
  • MAINCHAIN_SERVICE_URL: Mainchain service URL for custom deployments.
  • INVOKE_ALLOWED_METHODS: List of allowed methods that can be invoked via the /invoke API endpoint. The list can be expressed as a CSV. To allow invocation of all endpoints, set it to *. To allow invocation of all the registered methods under the specified namespaces, set it similar to legacy,chain. To allow invocation of specific methods, set it similar to chain_getBlocksByIDs,chain_getBlocksByHeight.

Note: interval takes priority over schedule and must be greater than 0 to be valid for all the moleculer job configurations.

Management

Start

cd klayr-service/services/blockchain-indexer # move into the root directory of the blockchain-indexer microservice
yarn start # start the microservice with running nodes locally

Use the framework/bin/moleculer_client.js and framework/bin/moleculer_subscribe.js clients to test particular service endpoints.

If you want to run a production variant of the service, use Docker or PM2. This will automatically recover the process when it fails.

Stop

Press Ctrl+C in the terminal to stop the process.

Extending Indexer

The applyTransaction and revertTransaction hooks are arranged per command in a file and are grouped by the module that they belong to.
Existing hooks are located in the shared/indexer/transactionProcessor directory.

When implementing the custom hooks please adhere to the following:

  • Create a sub-directory with the module name. For example: token.
  • Add index.js under the above directory.
    • Export a MODULE_NAME variable. The value must match the module name as registered within the application.
  • Create a file specific to the command for which you need to implement the custom hooks. For example: transfer.
    • Export a COMMAND_NAME variable. The value must match the command name as registered within the application.
    • Implement the custom logic for the applyTransaction and revertTransaction hooks.
    • Export the hooks.
  • To aid your development, please use the sample templates here.

Contributors

https://github.com/KlayrHQ/klayr-service/graphs/contributors

License

Copyright 2016-2023 Lisk Foundation

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.