Skip to content

Latest commit

 

History

History
148 lines (127 loc) · 5.53 KB

README.md

File metadata and controls

148 lines (127 loc) · 5.53 KB

Statefun NDB Benchmark

Benchmark for my Thesis Highly Available Stateful Serverless Functions in Apache Flink.

This benchmark was developed to compare Apache Flink StateFul functions using the current standard RocksDB backend and a prototype of a decoupled state backend called FlinkNDB. It could be also be used to benchmark other state backends or other Stateful Serverless platform. To customize the benchmark, look at the file run.sh and change out the state backend or the deployment of the used serverless platform.

Notes/Todos

The following is a list of notes and todos that you need to know before using the tool.

  • To reduce the amount of data transferred between the local computer and gcp, the flink/data-utils build is first uploaded to a gcp bucket and then transferred from there to the VMs. The address of the gcp bucket is currently hard-coded into the startup-scripts, so the proper address has to be edited into these files. TODO: Make general solution to bucket address in startup scripts.

Deployment

Deployment scripts can be found in deployment.

Deployment Prerequisites

The following prerequisites are necessary for both local and cloud deployment.

Before running the benchmark the input data must be generated. You can generate the data yourself by running:

python data-utils/generate_data.py

The benchmark requires a packaged Flink build to run. Copy your own Flink build into the folder deployment/flink, and name the folder build. Instructions of how to build Flink can be found here.

cp -r <path-to-flink-build> /deployment/flink/build/

Finally, different configuration parameters are set in the deployment scripts, open the script you are about to run and edit the configuration parameters at the top of the file.

Local Deployment

The local benchmark assumes that a RonDB cluster is running at localhost:3306.

Set the state backend to ndb or rocksdb by either of the following commands

./deployment/local/set-ndb-backend.sh
./deployment/local/set-rocksdb-backend.sh

Run the benchmark

./deployment/local/run.sh

GCP Deployment

Before running the benchmark in GCP install the GCP CLI and initialize your environment using:

gcloud init

Then, run the benchmarks in GCP using:

./deployment/gcp/run.sh

Full Benchmark Setup

The following is a description of the complete benchmark. This workflow is automated by the run.sh scripts above.

Prerequisites

For this benchmark pipeline, you will need:

  • Java
  • Maven
  • Jupyter Notebook (for evaluation)
  • Docker

The following is all the procedures necessary to run the StateFun NDB Benchmark:

  1. Set up a RonDB cluster, make sure it is listening to the default port of 3306.
  2. Initialize the database and the tables using ndb-utils/init_db.sql.
  3. Download the NDB lib files and make sure that the environment variable LD_LIBRARY_PATH points to the directory containing the files.
curl https://repo.hops.works/master/lib-ndb-6.1.0.tgz > lib-ndb.tgz
tar -xvzf lib-ndb.tgz
cd lib-ndb-6.1.0
export LD_LIBRARY_PATH=`pwd`
  1. Set up Kafka to listen to port 9092. The topics that need to be created are:
    • "add-to-cart"
    • "add-confirm"
    • "checkout"
    • "restock"
    • "receipts"
  2. Build Flink by cloning: senorcarbone/flink-rondb and running the following command in the project (this might take a while):
mvn clean install -DskipTests -Dscala-2.12
  1. Copy the build folder flink-rondb/flink-dist/target/flink-1.14.3-SNAPSHOT-bin/flink-1.14.3-SNAPSHOT into a the folder deployment/flink and rename it to build
  2. Configure the Flink cluster by making sure that the following fields are set in deployment/flink/build/conf/flink-conf.yaml:
# For NDB:
state.backend: ndb
state.backend.ndb.dbname: flinkndb
state.backend.ndb.truncatetableonstart: false
state.backend.ndb.connectionstring: <address to rondb cluster>

# For RocksDB:
state.backend: rocksdb
state.backend.incremental: true

# For both:
execution.checkpointing.interval: 10sec
execution.checkpointing.mode: EXACTLY_ONCE
state.savepoints.dir: <path in filesystem or gcp bucket>
state.checkpoints.dir: <path in filesystem or gcp bucket>
classloader.parent-first-patterns.additional: org.apache.flink.statefun;org.apache.kafka;com.google.protobuf
jobmanager.rpc.address: <address to Flink JobManager>
jobmanager.rpc.port: <port to Flink JobManager>
  1. Run the JobManager
./deployment/builds/flink-build/bin/jobmanager.sh start
  1. Run one or more TaskManagers
./deployment/builds/flink-build/bin/taskmanager.sh start
  1. Build StateFun job
./deployment/local/build.sh
  1. Run the StateFun Runtime + Job;
./deployment/flink/build/bin/flink run -c org.apache.flink.statefun.flink.core.StatefulFunctionsJob shoppingcart-embedded/target/shoppingcart-embedded-1.0-SNAPSHOT-jar-with-dependencies.jar
  1. Generate input events:
python data-utils/generate_data.py
  1. Run Data Stream Generator
python data-utils/produce_events.py
  1. When job has finished, run output-consumer:
python data-utils/output_consumer.py
  1. Produce output plots using the Jupyter Notebook in the evaluator folder, setting the correct filepaths at the top of the notebook.