Fault injection

[derekchiang]

Most mature distributed system projects have some sort of fault injection frameworks for testing purposes. Some examples:

• Hadoop: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/FaultInjectFramework.html
• Spark: https://github.com/databricks/spark-integration-tests/blob/9f515bf9aa8163ee7e7936f6e4a36cc771cfff04/src/test/scala/org/apache/spark/integrationtests/docker/network/NetworkFaultInjector.scala

And there are more general-purpose tools such as Jepsen and ChaosMonkey that can be used to inject network faults.

Using tools such as these will help us identify bugs that would otherwise be found in production.

[derekchiang]

Made some progress for 1.1 by adding tests for dynamic membership. Removing the 1.1 label as we will be revisiting this issue post 1.1.

[jdoliner]

We should aim to do more of this in 1.2.
In particular I'd like to see some faults that affect running jobs and see how the system handles it.

[sjezewski]

We should also add coverage for the places that we support exponential backoff. Specifically in PutBlock and DeleteBlock

[derekchiang]

The need for this tool is getting stronger, as we are seeing more people pumping a serious amount of data into Pachyderm and bugs are being caught in production. Most of those bugs only manifest when certain requests fail due to transient network failures, which can be simulated by such a tool.

[derekchiang]

Here's how Kubernetes does it: https://github.com/kubernetes/kubernetes/tree/master/pkg/client/chaosclient

I think this makes sense for us as the first step as well. Basically the faults are injected at client side: in our case it will be code under src/client/...

[sjezewski]

We have a recent need for testing of this nature:

#2675

This will be a critical path for a customer's deployment.