Skip to content

Commit

Permalink
Merge pull request #283 from panyu4/adapter-filetxDocs
Browse files Browse the repository at this point in the history
Add documentation about adapters and test mode
  • Loading branch information
feihujiang authored Feb 1, 2019
2 parents e15169e + b91d4f6 commit c3f7366
Show file tree
Hide file tree
Showing 2 changed files with 277 additions and 50 deletions.
208 changes: 158 additions & 50 deletions docs/2_Architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,57 +37,60 @@ A default benchmark engine is implemented to help developers to understand the f
Two kinds of configuration files are used. One is the benchmark configuration file, which defines the arguments of the benchmark like workload. Another is the blockchain configuration file, which specify necessary information to help interacting with the SUT.

Below is a benchmark configuration file example:
```json
{
"blockchain": {
"type": "fabric",
"config": "./fabric.json"
},
"command" : {
"start": "docker-compose -f ../../network/fabric/simplenetwork/docker-compose.yaml up -d",
"end" : "docker-compose -f ../../network/fabric/simplenetwork/docker-compose.yaml down;docker rm $(docker ps -aq)"
},
"test": {
"name": "simple",
"description" : "This is an example benchmark for caliper",
"clients": {
"type": "local",
"number": 5
},
"rounds": [{
"label" : "open",
"txNumber" : [5000, 5000, 5000],
"rateControl" : [{"type": "fixed-rate", "opts": {"tps" : 100}}, {"type": "fixed-rate", "opts": {"tps" : 200}}, {"type": "fixed-rate", "opts": {"tps" : 300}}],
"arguments": { "money": 10000 },
"callback" : "benchmark/simple/open.js"
},
{
"label" : "query",
"txNumber" : [5000, 5000],
"rateControl" : [{"type": "fixed-rate", "opts": {"tps" : 300}}, {"type": "fixed-rate", "opts": {"tps" : 400}}],
"callback" : "benchmark/simple/query.js"
}]
},
"monitor": {
"type": ["docker", "process"],
"docker":{
"name": ["peer0.org1.example.com", "http://192.168.1.100:2375/orderer.example.com"]
},
"process": [
{
"command" : "node",
"arguments" : "local-client.js",
"multiOutput" : "avg"
}
],
"interval": 1
}
}
```yaml
test:
name: simple
description: This is an example benchmark for caliper
clients:
type: local
number: 5
rounds:
- label: open
txMode:
type: real-time
txNumber:
- 5000
- 5000
- 5000
rateControl:
- type: fixed-rate
opts:
tps: 100
- type: fixed-rate
opts:
tps: 200
- type: fixed-rate
opts:
tps: 300
arguments:
money: 10000
callback: benchmark/simple/open.js
- label: query
txNumber:
- 5000
- 5000
rateControl:
- type: fixed-rate
opts:
tps: 300
- type: fixed-rate
opts:
tps: 400
callback" : benchmark/simple/query.js
monitor:
type:
- docker
- process
docker:
name:
- peer0.org1.example.com
- http://192.168.1.100:2375/orderer.example.com
process:
- command: node
arguments: local-client.js
multiOutput: avg
interval: 1
```
* **blockchain** - defines the type of backend blockchain system and the configuration file for the adaptor to recognize the backend blockchain network with which to interact. See [*Fabric Config*]({{ site.baseurl }}{% link docs/Fabric_Configuration.md %}) to learn more details.
* **command** - defines commands which will be called at particular phases of the test
* **start** : be called at the beginning of the test
* **end** : be called when finishing all tests
* **test** - defines the metadata of the test, as well as multiple test rounds with specified workload:
* **name&description** : human readable name and description of the benchmark, the value is used by the report generator to show in the testing report.
* **clients** : defines the client type as well as relevant arguments, the 'type' property must be 'local' or 'zookeeper'
Expand All @@ -101,6 +104,7 @@ Below is a benchmark configuration file example:
}
```
* **label** : hint for the test. For example, you can use the transaction name as the label name to tell which transaction is mainly used to test the performance. The value is also used as the context name for *blockchain.getContext()*. For example, developers may want to test performance of different Fabric channels, in that case, tests with different label can be bound to different fabric channels.
* **txMode** : specifies the transactions' generation mode. Two kinds of test transactions configurations are supported. One is named as `real-time` which means the transactions will be generated in real time and sent to to the backend systems immediately. This is Caliper's default test mode, so if txMode is not set, Caliper will run under the `real-time` mode. The other mode is file related test mode, which means the transactions generated by the clients will be buffered, and saved into files before they are sent to the backend systems. The file mode has two subclasses, named as `file-read` and `file-write`. Under the `file-read` mode, Caliper will read transactions files to do the test. Under the `file-write` mode, all of the transactions will be generated according to the configurations settings like txNumber, tps, duration and so on, and then these transactions will be sent to the backend systems to complete the test. The generated transactions files are saved in the current directory. If the transaction files under that directory are not matched with current benchmark configuration file's parameters such as txNumber and clients' number, Caliper will generated new transactions files. The transactions files are able to be used repeatedly in the docker mode, not in the self-deployed network. To be noticed, the file related txMode currently only supports the fixed sending rate(that means rate-control's type should be fixed-rate) and fabric blockchain(1.0/1.1/1.2). Since Fabric client does not support the function sendSignedTransaction in the versions 1.0,1.1 and 1.2, it is necessary to rename the file `node_modules/fabric-client/lib/Channel.js/ChannelSignedTransaction.js` to `node_modules/fabric-client/lib/Channel.js` when using this file mode. The file related txMode marks the transanction's create time just before fabric client sends the transaction to the orderer node. There is an obvious difference about create time between file txMode and `real-time` txMode, while the later marks the create time as the time when transaction is handled by the corresponding adapter.
* **txNumber** : defines an array of sub-rounds with different transaction numbers to be run in each round. For example, [5000,400] means totally 5000 transactions will be generated in the first round and 400 will be generated in the second.
* **txDuration** : defines an array of sub-rounds with time based test runs. For example [150,400] means two runs will be made, the first test will run for 150 seconds, and the second will run for 400 seconds. If specified in addition to txNumber, the txDuration option will take precedence.
* **rateControl** : defines an array of custom rate controls to use during the benchmarking test sub-rounds. If not specified will default to 'fixed-rate' that will drive the benchmarking at a set 1 TPS rate. If defined, the rate control mechanism must exist, and may be provided with options to use to control the rate at which messages are sent, or to specify a message rate profile. Each round, specified within **txNumber** or **txDuration** must have a corresponding rate control item within the **rateControl** array. For more information on available rate controllers and how to implement custom rate controllers, refer to the [rate controllers section]({{ site.baseurl }}{% link docs/Rate_Controllers.md %})
Expand All @@ -112,6 +116,110 @@ Below is a benchmark configuration file example:
* process : a process monitor is used to monitor specified local process. For example, users can use this monitor to watch the resource consumption of simulated blockchain clients. The 'command' and 'arguments' properties are used to specify the processes. The 'multiOutput' property is used to define the meaning of the output if multiple processes are found. 'avg' means the output is the average resource consumption of those processes, while 'sum' means the output is the summing consumption.
* others : to be implemented.

Below is a blockchain configuration file example:
```json
{
"caliper": {
"blockchain": "fabric",
"command" : {
"start": "docker-compose -f network/fabric-v1.1/2org1peergoleveldb/docker-compose.yaml up -d;sleep 3s",
"end" : "docker-compose -f network/fabric-v1.1/2org1peergoleveldb/docker-compose.yaml down;docker rm $(docker ps -aq);docker rmi $(docker images dev* -q)"
}
},
"fabric": {
"cryptodir": "network/fabric-v1.1/config/crypto-config",
"network": {
"orderer": {
"url": "grpc://localhost:7050",
"mspid": "OrdererMSP",
"msp": "network/fabric-v1.1/config/crypto-config/ordererOrganizations/example.com/msp/",
"server-hostname": "orderer.example.com",
"tls_cacerts": "network/fabric-v1.1/config/crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/tls/ca.crt"
},
"org1": {
"name": "peerOrg1",
"mspid": "Org1MSP",
"msp": "network/fabric-v1.1/config/crypto-config/peerOrganizations/org1.example.com/msp/",
"ca": {
"url": "http://localhost:7054",
"name": "ca-org1"
},
"peer1": {
"requests": "grpc://localhost:7051",
"events": "grpc://localhost:7053",
"server-hostname": "peer0.org1.example.com",
"tls_cacerts": "network/fabric-v1.1/config/crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt"
}
},
"org2": {
"name": "peerOrg2",
"mspid": "Org2MSP",
"msp": "network/fabric-v1.1/config/crypto-config/peerOrganizations/org2.example.com/msp/",
"ca": {
"url": "http://localhost:8054",
"name": "ca-org2"
},
"peer1": {
"requests": "grpc://localhost:8051",
"events": "grpc://localhost:8053",
"server-hostname": "peer0.org2.example.com",
"tls_cacerts": "network/fabric-v1.1/config/crypto-config/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt"
}
}
},
"channel": [
{
"name": "mychannel",
"deployed": false,
"config": "network/fabric-v1.1/config/mychannel.tx",
"organizations": ["org1", "org2"]
}
],
"chaincodes": [
{"id": "simple", "path": "contract/fabric/simple/go", "language":"golang", "version": "v0", "channel": "mychannel"},
],
"endorsement-policy": {
"identities": [
{
"role": {
"name": "member",
"mspId": "Org1MSP"
}
},
{
"role": {
"name": "member",
"mspId": "Org2MSP"
}
},
{
"role": {
"name": "admin",
"mspId": "Org1MSP"
}
}
],
"policy": { "2-of": [{"signed-by": 0}, {"signed-by": 1}]}
},
"context": {
"open": "mychannel",
"query": "mychannel"
}
},
"info" : {
"Version": "1.1.0",
"Size": "2 Orgs with 1 Peer",
"Orderer": "Solo",
"Distribution": "Single Host"
}
}
```
* **caliper** - defines the parameters used by the caliper:
* **blockchain** - defines the type of backend blockchain system and the configuration file for the adaptor to recognize the backend blockchain network with which to interact. See [*Fabric Config*]({{ site.baseurl }}{% link docs/Fabric_Configuration.md %}) to learn more details.
* **command** - defines commands which will be called at particular phases of the test
* **start** : be called at the beginning of the test
* **end** : be called when finishing all tests
* **fabric** - defines the network related configurations that will be used by the backend blockchain systems. Here the section name is fabric, but it could be changed to any one of the caliper's supported DLT, such as Composer, Sawtooth, Iroha and Burrow.
### Master

The master implements a default test flow which contains three stages:
Expand Down
119 changes: 119 additions & 0 deletions docs/Writing_Adapters.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
---
layout: page
title: "Writing Benchmarks"
categories: reference
---
# About the Adapters
This folder containers kinds of adapters which will interact with the corresponding backend blockchain.

# How to write your own blockchain adapter
Let's look inside first and learn about how the whole framework interacts with the backend blockchain system.
When the benchmark engine is running, the master process of benchmark engine will call the user defined blockchian class to complete the blockchain's and chaicodes' installation. Then, after the master process launches the corresponding clients, each client will do the test. During the test, the client will get current blockchain's context, run test scripts, release the blockchain's context in the end, and return the performance statistics. Hence, if users intend to test the blockchain system which Caliper is unable to support, the bellows are what the users would concern about.

* Use Blockchain NBI to write your own blockchain Class: Below is a Blockchain interface implementation example.
```
/**
* Implements {BlockchainInterface} for a myblockchain backend.
*/
class Myblockchain extends BlockchainInterface{
/**
* Create a new instance of the {Myblockchain} class.
* @param {string} config_path The path of Myblockchain network configuration file.
*/
constructor(config_path) {
}
/**
* Initialize the {Myblockchain} object.
*/
init() {
...
}
/**
* Deploy the chaincode specified in the network configuration file to all peers.
* @return {Myblockchain} resolve
*/
installSmartContract() {
...
}
/**
* Return the Burrow context associated with the given callback module name.
* @param {string} name The name of the callback module as defined in the configuration files.
* @param {object} args Unused.
* @return {object} The assembled Myblockchain context.
* @async
*/
getContext(name, args) {
...
}
/**
* Release the given Myblockchain context.
* @param {object} context The Myblockchain context to release.
* @async
*/
async releaseContext(context) {
...
}
/**
* Invoke a smart contract.
* @param {Object} context context object
* @param {String} contractID identity of the contract
* @param {String} contractVer version of the contract
* @param {Array} args array of JSON formatted arguments for multiple transactions
* @param {Number} timeout request timeout, in seconds
* @return {Promise<object>} the promise for the result of the execution.
*/
async invokeSmartContract(context, contractID, contractVer, args, timeout) {
...
}
/**
* Query the given chaincode according to the specified options.
* @param {object} context The Myblockchain context returned by {getContext}.
* @param {string} contractID The name of the chaincode.
* @param {string} contractVer The version of the chaincode.
* @param {string} key The argument to pass to the chaincode query.
* @param {string} [fcn=query] The chaincode query function name.
* @return {Promise<object>} The promise for the result of the execution.
*/
async queryState(context, contractID, contractVer, key, fcn = 'query') {
...
}
/**
* Get adapter specific transaction statistics.
* @param {JSON} stats txStatistics object
* @param {Array} results array of txStatus objects.
*/
getDefaultTxStats(stats, results) {
...
}
}
```

* Add your own blockchain type into the blockchain's constructor function: In the file `src/comm/blockchain.js`, a new blockchain type should be added into the constructor function.
```
if(config.caliper.blockchain === 'myblockchain') {
let myblockchain = require('../adapters/myblockchain/myblockchain.js');
this.bcType = 'myblockchain';
this.bcObj = new myblockchain(configPath);
}
```
* Add predefined Network files into the dir `network/`: These files will be useful when Caliper is trying to simulate your blockchain system. Please add all of the related files what a new boot blockchain system needs.
* Add your own network configuration file into the corresponding network folder: Make sure the files that are necessary to boot your blockchain are specified here.
* Define your command which will be executed before and after the test
* Define your own smart contracts: As Caliper has several test cases now, it is necessary to realize your own smart contracts according to current test cases, eg. you could provide your smart contracts about opening an account, querying an account, deleting an account and transfering according to the test case `simple`. Your own smart contracts could be put into the directory `src/contract/myblockchain/`.
* Define the installation script: To facilitate other users, an installation script in the file `package.json` is appreciated. Your packages and other dependencies should be added.
```
"scripts": {
"myblockchain-deps": "npm install --no-save myblockchainpackage
}
```

If you would like to define your test module, please use Blockchain NBI to write your own test script which should include 3 functions(init(), run()and end()) as the files in the directory `bechmark/simple/open.js` and `bechmark/simple/query.js`, and change the callback property in the test configuration file into current test script's path. The whole flow of Benchmarks is referred to [Writing Benchmarks]({{ site.baseurl }}{% link docs/Writing_Benchmarks.md %}).

0 comments on commit c3f7366

Please sign in to comment.