Testground is a platform for testing, benchmarking, and simulating distributed and p2p systems at scale. It's designed to be multi-lingual and runtime-agnostic, scaling gracefully from 2 to 10k instances, only when needed.
- Getting started
- Documentation
- How does it work?
- Features
- Where to find test plans?
- Contributing
- Team
- License
NOTE: currently, we don't distribute binaries, so you will have to build from source.
Prerequisites: Go 1.14+, Docker daemon running.
$ git clone https://github.com/testground/testground.git
$ cd testground
$ make install # builds testground and the Docker image, used by the local:docker runner.
$ testground daemon # will start the daemon listening on localhost:8042 by default.
# => open a different console (client-side), in the same directory (testground/testground repo checkout)
# import the network test plan from this repo into $TESTGROUND_HOME/plans
# read more about this path at https://docs.testground.ai/getting-started#running-testground
$ testground plan import --from ./plans/network
# run two instances of the `ping-pong` test case from the `network` plan,
# building with docker:go, running with local:docker
$ testground run single --plan=network --testcase=ping-pong \
--builder=docker:go --runner=local:docker \
--instances=2See Getting started and the rest of the docs on our docs website for more info! 🚀
This README is just the tip of the iceberg! Check out our full documentation site at docs.testground.ai.
There you will find a conceptual system walkthrough, tips on writing test plans, instructions on running test plans, configuring runners and builders, deploying Kubernetes clusters, and a lot more.
Please report any problems or inaccuracies by opening a docs issue on this repo.
-
You develop distributed test plans as if you were writing unit tests against local APIs.
- No puppeteering necessary.
- No need to package and ship the system or component under test as a separate daemon.
- No need to expose every internal setting over an external API, just for the sake of testing.
-
Your test plan calls out to the coordination API to:
- communicate out-of-band information (such as endpoint addresses, peer ids, etc.)
- leverage synchronization and ordering primitives such as signals and barriers to model a distributed state machine.
- programmatically apply network traffic shaping policies, which you can alter during the execution of a test to simulate various network conditions.
-
There is no special "conductor" node telling instances what to do when. The choreography and sequencing emerges from within the test plan itself.
-
You decide what versions of the upstream software you want to exercise your test against.
- Benchmark, simulate, experiment, run attacks, etc. against versions v1.1 and v1.2 of the components under test in order to compare results, or test compatibility.
- Assemble hybrid test runs mixing various versions of the dependency graph.
-
Inside your test plan:
- You record observations, metrics, success/failure statuses.
- You emit structured or unstructured assets you want collected, such as event logs, dumps, snapshots, binary files, etc.
-
Via a TOML-based composition file, you instruct Testground to:
- Assemble a test run comprising groups of 2, 200, or 10000 instances, each with different test parameters, or built against different depencency sets.
- Schedule them for run locally (executable or Docker), or in a cluster (Kubernetes).
-
You collect the outputs of the test plan with a single command, and use data processing scripts and platforms (such as the upcoming Jupyter notebooks integration) to draw conclusions.
(🌕 = fully supported // 🌑 = planned)
- Experimental/iterative development 🌖 (The team at Protocol Labs has used Testground extensively to evaluate protocol changes in large networks, simulate attacks, measure algorithmic improvements across network boundaries, etc.)
- Debugging 🌗
- Comparative testing 🌖
- Backwards/forward-compatibility testing 🌖
- Interoperability testing 🌑
- Continuous integration 🌑
- Stakeholder/acceptance testing 🌑
A test plan is a blackbox with a formal contract. Testground promises to inject a set of env variables, and the test plan promises to emit events on stdout, and assets on the output directory.
- As such, a test plan can be any kind of program, written in Go, JavaScript, C, or shell.
- At present, we offer builders for Go, with TypeScript (node and browser) being in the works.
For running test plans written in different languages, targeted for different runtimes, and levels of scale:
exec:goanddocker:gobuilders: compile test plans written in Go into executables or containers.local:exec,local:docker,cluster:k8srunners: run executables or containers locally (suitable for 2-300 instances), or in a Kubernetes cloud environment (300-10k instances).
Got some spare cycles and would like to add support for writing test plans Rust, Python or X? It's easy! Open an issue, and the community will guide you!
Redis-backed lightweight API offering synchronisation primitives to coordinate and choreograph distributed test workloads across a fleet of nodes.
Test instances are able to set connectedness, latency, jitter, bandwidth, duplication, packet corruption, etc. to simulate a variety of network conditions.
Create a k8s cluster ready to run Testground jobs on AWS by following the instructions at
testground/infra.
Compiling test plans against specific versions of upstream dependencies (e.g. moduleX v0.3, or commit 1a2b3c).
So that a single test plan can work with a range of versions of the components under test, as these evolve over time.
Diagnostics: Automatic diagnostics via pprof (for Go test plans), with metrics emitted to InfluxDB in real-time. Metrics can be raw data points or aggregated measurements, such as histograms, counters, gauges, moving averages, etc.
Results: When the test plan concludes, all results are pushed in batch to InfluxDB for later exploration, analysis, and visualization.
Create tailored test runs by composing scenarios declaratively, with different groups, cohorts, upstream deps, test params, etc.
Emit and collect/export/download test outputs (logs, assets, event trails, run events, etc.) from all participants in a run.
There are some basic, project-agnostic Testground test plans in the plans directory.
We use these plans to validate the functionality of Testground itself.
To link them under $TESTGROUND_HOME/plans, if you're using default paths, these commands should do the trick
(assuming you have already run the Testground daemon once, and therefore the $TESTGROUND_HOME layout has been created
for you):
$ # from the root of this repo, run the following; it will symlink all test plans under $TESTGROUND_HOME/plans
$ ln -s $PWD/plans/* $HOME/testground/plans
$ testground run single --plan network --testcase ping-pong --builder=docker:go --runner=local:docker --instances=2For project-specific test plans, check out these repos:
To use them, import them into $TESTGROUND_HOME/plans using the following testground commands:
$ testground plan import --git --from https://github.com/libp2p/test-plans.git --name libp2p
$ testground plan import --git --from https://github.com/ipfs/test-plans.git --name ipfs
$ # to run the find-peers test case from the libp2p/dht test plan (this is not a complete command!)
$ testground run single --plan libp2p/dht --testcase find-peers --builder docker:go --runner local:docker <options>Please read our CONTRIBUTING Guidelines before making a contribution.
- @raulk 🎈 (founder, project lead, technical product owner, architect)
- @nonsense ⛷ (tech lead + core engineer)
- @coryschwartz 🦉 (core engineer)
- @hacdias 😂 (core engineer)
@daviddias, @stebalien, @jimpick, @aschmahmann, @dirkmc, @yusefnapora.
You can find notes from the Testground team meetings at github.com/testground/pm
Dual-licensed: MIT, Apache Software License v2, by way of the Permissive License Stack.