-
Notifications
You must be signed in to change notification settings - Fork 0
Running FeBFT's microbenchmarks
In order to run the microbenchmarks you must obtain testing grounds.
At the moment, there are two tests that are available to run (the rest of the tests are old and not yet adapted to the new changes of FeBFT) microbenchmarks and microbenchmarks-async.
These tests are built to test the performance of the system, not to perform any type of fault emulation or to test the fault tolerance of the program. That can of course manually done by crashing one of the replicas.
At the moment (5/11/22), the timers that trigger the leader election are not active because we had some issues with the asynchronous runtime for performance and the timers ran on this asynchronous runtime, so we are still thinking of a way to properly handle this task as efficiently as possible. This means if the leader crashes, the other replicas will not attempt a view change unless they are programmed to do so.
After obtaining the testing grounds, choose what test you want to run and in what configuration.
There are two possible ways of running these test:
- Locally, on a single machine (which does not represent a real world use case of the system)
- On various systems (at least 5, with 4 of those for replicas and 1 for clients (a case of f=1)
This will be chosen depending on the environment variables chosen.
Install capnproto.
In debian based systems this can be done with:
sudo apt install capnproto
It is also available in several other distros with the same name.
The env file should be placed at the root of each of the tests (in microbenchmarks it would be /testing-grounds/microbenchmarks/env) and should contain the following:
#The local variable indicates the type of functioning. If you want to run this test locally, keep it at 1
#If you want to run this test in multiple system scenario set this to 0
export LOCAL=1
export COMPILE=1
#This is where you should choose the BFT SMR library to test. This test supports running both on FeBFT and on BFT-SMaRt.
#If you want to test FeBFT, this should be set to rust
#If you want to test BFT-SMaRt, this should be set to java
export BACKEND=rust
#Whether you want the test to be verbose
export VERBOSE=false
#Should we update the file descriptor limit of the OS? This should stay at 0 and be set manually by the user to avoid issues
export UPDATE_MAX_FDS=0
#The amount of clients the test should use
export NUM_CLIENTS=999
#The amount of replicas the test should use
export NUM_REPLICAS=4
#The amount of clients that should be placed in each client pool on FeBFT.
#This is not used by BFT-SMaRt
export CLIENTS_PER_POOL=100
#The target batch size of the test.
#In FeBFT this is merelly a suggestion and will not actually limit the amount of requests in the batch (It is also the batch size used at the client pool level, if you want to change the batch size of proposer batch size of FeBFT, use GLOBAL_BATCH_SIZE)
#In BFT-SMaRt this is the target batch size and no batch will be made with more requests than this.
export BATCH_SIZE=256
#FeBFT Proposer target batch size. BFT-SMaRt does not use this variable.
#The proposer will wait until a batch of this size before proposing it, even if it is ready. This does have a timeout, in case the batch size is not reached, which is the GLOBAL_BATCH_SLEEP_MICROS
export GLOBAL_BATCH_SIZE=40000
#How long should the proposer wait before proposing the batch even if it has not reached the target size. In microseconds.
export GLOBAL_BATCH_SLEEP_MICROS=50000
#The timeout for the batch size for the client pool request collectors. Similarly to the proposer, the client pools also wait until a given batch size before sending the requests to the proposer.
# This value is how long the client pool collector will wait before sending the batch even if it is not the required size.
export BATCH_TIMEOUT_MICROS=1000
# How long should the client pool collectors sleep between collections. This was done in order to allow the network layer to access the request buffers without concurrence slowing them down (even if the time taken to collect the requests is so low, as described in the thesis)
export BATCH_SLEEP_MICROS=1000
# This variable controls the above behaviour. If it is disabled it will actively and constantly try to collect requests and pass them to the proposer. This will cause a lot of CPU usage and will be very bad. Should be left at true.
export FILL_BATCH=true
#How many threads should the threadpool used for various computation work like serialization, signatures, digests, etc.
#This value does not really indicate scale. It is most effective at around 4/6 threads, as the rest of the middleware still needs computation power.
export THREADPOOL_THREADS=4
#How many requests should each of the clients perform before stopping.
export OPS_NUMBER=10000
#How many requests should be processed in order to provide a measurement of operations per second, latencies, etc..
export MEASUREMENT_INTERVAL=2000
#What size should the state of the mock service be in bytes.
export STATE_SIZE=0
#How large should the reply content of the service be in bytes.
export REPLY_SIZE=1
#How large should the request content of the service be in byte.
export REQUEST_SIZE=1
#Should the clients sleep for an amount of time between each of the requests.
export REQUEST_SLEEP_MILLIS=0
#This is a variable to indicate the path of a script to provide operating system statistics. It does not need to be set if you are testing FeBFT.
export PATH_TO_OS_STATS="/"
Each of the variables' meaning is described in the env file.
You will also need another file, env_id that should contain the following:
#This variable is only used in the various systems mode. In this mode, this variable should indicate the ID of the replica in this system
#In the client machine this is also not used.
export ID=0
After adding these two files, you can go to the Cargo.toml file in the microbenchmarks/rust/Cargo.toml folder to choose which version of febft you want to use. By default it searches for the febft version at the same level of the testing-grounds folder. You can alter this to your preference within the Rust possibilities.
Next you should enter into the microbenchmarks/rust folder and run:
./ca
This will generate all of the necessary certificates for the clients and the replicas. If you want to heavily alter the amount of clients (adding many clients) you might have to edit this script in order to change the amount of client certificates generated.
You will then need to update generate_conf_cop to reflect the IP addresses of your replica and client machines.
This should generate a replicas.config file that looks like this:
0 srv0 10.11.12.1 10000 12000
1 srv1 10.11.12.2 10001 12001
2 srv2 10.11.12.3 10002 12002
3 srv3 10.11.12.4 10003 12003
Where the first column is the ID of the replica, then the name of the replica, then the IP and then the replica facing port (port used for replica-replica communication) and client facing port (port used for replica-client communication)
It should also generate a clients.config file that looks like this:
1000 cli1000 10.11.12.5 11000
1001 cli1001 10.11.12.5 11001
1002 cli1002 10.11.12.5 11002
1003 cli1003 10.11.12.5 11003
1004 cli1004 10.11.12.5 11004
1005 cli1005 10.11.12.5 11005
It is similar to the replica config file, except without the port for replica-client communication, since it will only communicate with replicas, never other clients.
After all this is done, you should go back to the base microbenchmarks folder and run:
./run servers on the terminal. If you are using the local mode this should start all replicas, if you are using the multi device mode this will only start the corresponding replica. You will need to do this for all devices.
You can then run:
./run clients to start all of the clients. This should be ran on a different terminal if you are using the local mode.
Running microbenchmarks-async is very similar to the previously mentioned protocol so you should repeat it, except in the microbenchmarks-async directory. The only change is that when you are writing the env file, you should add the following:
#This indicates how many concurrent requests should be executed by each individual client.
export CONCURRENT_RQS=1