Migrate leader terminate to zbchaos

[ChrisKujawa]

blocked by #271

related to #237

---

Similar to the other migrated experiments I followed same approach as described here #268

The experiment was executed and verified via the integration test against a self-managed cluster.

I moved the experiment into the chaos-experiments/camunda-cloud/test/ folder and migrated it, with that approach I was able to execute the experiment with eze and running against my self-managed zell-chaos zeebe cluster.

Log output:

Create ChaosToolkit instance
Open workers: [zbchaos, readExperiments].
Handle read experiments job [key: 2251799813685265]
Read experiments successful, complete job with: {"experiments":[{"contributions":{"availability":"high","reliability":"high"},"description":"This fake experiment is just to test the integration with Zeebe and zbchaos workers","method":[{"name":"Show again the version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"Show version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"This is a fake experiment","version":"0.1.0"},{"contributions":{"availability":"high","reliability":"high"},"description":"Zeebe should be fault-tolerant. We expect that Zeebe can handle non-graceful leader restarts.","method":[{"name":"Terminate leader of partition 1 non-gracefully","provider":{"arguments":["terminate","broker","--role","LEADER","--partitionId","1"],"path":"zbchaos","type":"process"},"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"All pods should be ready","provider":{"arguments":["verify","readiness"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Can deploy process model","provider":{"arguments":["deploy","process"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Should be able to create process instances on partition 1","provider":{"arguments":["verify","instance-creation","--partitionId","1"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"Zeebe Leader restart non-graceful experiment","version":"0.1.0"}]}.
Handle zbchaos job [key: 2251799813685328]
Running command with args: [version] 
zbchaos development (commit: HEAD)
Handle zbchaos job [key: 2251799813685374]
Running command with args: [version] 
zbchaos development (commit: HEAD)
Handle zbchaos job [key: 2251799813685417]
Running command with args: [version] 
zbchaos development (commit: HEAD)
Handle zbchaos job [key: 2251799813685508]
Running command with args: [verify readiness] 
Connecting to zell-chaos
Running experiment in self-managed environment.
All Zeebe nodes are running.
Handle zbchaos job [key: 2251799813685551]
Running command with args: [deploy process] 
Connecting to zell-chaos
Running experiment in self-managed environment.
Successfully created port forwarding tunnel
Deploy file bpmn/one_task.bpmn (size: 2526 bytes).
Deployed process model bpmn/one_task.bpmn successful with key 2251799813685431.
Deployed given process model , under key 2251799813685431!
Handle zbchaos job [key: 2251799813685597]
Running command with args: [verify instance-creation --partitionId 1] 
Connecting to zell-chaos
Running experiment in self-managed environment.
Successfully created port forwarding tunnel
Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false]
Created process instance with key 4503599627380556 on partition 2, required partition 1.
Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false]
Created process instance with key 6755399441065816 on partition 3, required partition 1.
Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false]
Created process instance with key 2251799813695425 on partition 1, required partition 1.
The steady-state was successfully verified!
Handle zbchaos job [key: 2251799813685645]
Running command with args: [terminate broker --role LEADER --partitionId 1] 
Connecting to zell-chaos
Running experiment in self-managed environment.
Successfully created port forwarding tunnel
Found Broker zell-chaos-zeebe-1 as LEADER for partition 1.
Terminated zell-chaos-zeebe-1
Handle zbchaos job [key: 2251799813685691]
Running command with args: [verify readiness] 
Connecting to zell-chaos
Running experiment in self-managed environment.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds.
All Zeebe nodes are running.
Handle zbchaos job [key: 2251799813686206]
Running command with args: [deploy process] 
Connecting to zell-chaos
Running experiment in self-managed environment.
Successfully created port forwarding tunnel
Deploy file bpmn/one_task.bpmn (size: 2526 bytes).
Deployed process model bpmn/one_task.bpmn successful with key 2251799813685431.
Deployed given process model , under key 2251799813685431!
Handle zbchaos job [key: 2251799813686252]
Running command with args: [verify instance-creation --partitionId 1] 
Connecting to zell-chaos
Running experiment in self-managed environment.
Successfully created port forwarding tunnel
Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false]
Created process instance with key 4503599627380585 on partition 2, required partition 1.
Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false]
Created process instance with key 6755399441065847 on partition 3, required partition 1.
Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false]
Created process instance with key 2251799813695455 on partition 1, required partition 1.
The steady-state was successfully verified!
Instance 2251799813685255 [definition 2251799813685253 ] completed
--- PASS: Test_ShouldBeAbleToRunExperiments (56.98s)