Skip to content

Commit

Permalink
Migrate follower terminate (#270)
Browse files Browse the repository at this point in the history
depends #269 

related to #237

-----


Migrates the follower terminate experiment and tested with integration
test

Log output:


```

Deploy file bpmn/chaos/actionRunner.bpmn (size: 8788 bytes).
Deployed process model bpmn/chaos/actionRunner.bpmn successful with key 2251799813685249.
Deploy file bpmn/chaos/chaosExperiment.bpmn (size: 21403 bytes).
Deployed process model bpmn/chaos/chaosExperiment.bpmn successful with key 2251799813685251.
Deploy file bpmn/chaos/chaosToolkit.bpmn (size: 11031 bytes).
Deployed process model bpmn/chaos/chaosToolkit.bpmn successful with key 2251799813685253.
Create ChaosToolkit instance
Open workers: [zbchaos, readExperiments].
Handle read experiments job [key: 2251799813685265]
Read experiments successful, complete job with: {"experiments":[{"contributions":{"availability":"high","reliability":"high"},"description":"Zeebe should be fault-tolerant. Zeebe should be able to handle followers terminations.","method":[{"name":"Terminate follower of partition 1","provider":{"arguments":["terminate","broker","--role","FOLLOWER","--partitionId","1"],"path":"zbchaos","type":"process"},"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"All pods should be ready","provider":{"arguments":["verify","readiness"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Can deploy process model","provider":{"arguments":["deploy","process"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Should be able to create process instances on partition 1","provider":{"arguments":["verify","instance-creation","--partitionId","1"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"Zeebe follower restart non-graceful experiment","version":"0.1.0"},{"contributions":{"availability":"high","reliability":"high"},"description":"This fake experiment is just to test the integration with Zeebe and zbchaos workers","method":[{"name":"Show again the version","pauses":{"after":5},"provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"Show version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"This is a fake experiment","version":"0.1.0"}]}.
Handle zbchaos job [key: 2251799813685328]
Running command with args: [verify readiness] 
Connecting to zell-chaos
Running experiment in self-managed environment.
All Zeebe nodes are running.
Handle zbchaos job [key: 2251799813685374]
Running command with args: [deploy process] 
Connecting to zell-chaos
Running experiment in self-managed environment.
Successfully created port forwarding tunnel
Deploy file bpmn/one_task.bpmn (size: 2526 bytes).
Deployed process model bpmn/one_task.bpmn successful with key 2251799813685431.
Deployed given process model , under key 2251799813685431!
Handle zbchaos job [key: 2251799813685418]
Running command with args: [verify instance-creation --partitionId 1] 
Connecting to zell-chaos
Running experiment in self-managed environment.
Successfully created port forwarding tunnel
Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false]
Created process instance with key 4503599627372098 on partition 2, required partition 1.
Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false]
Created process instance with key 6755399441057346 on partition 3, required partition 1.
Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false]
Created process instance with key 2251799813686915 on partition 1, required partition 1.
The steady-state was successfully verified!
Handle zbchaos job [key: 2251799813685467]
Running command with args: [terminate broker --role FOLLOWER --partitionId 1] 
Connecting to zell-chaos
Running experiment in self-managed environment.
Successfully created port forwarding tunnel
Found Broker zell-chaos-zeebe-2 as FOLLOWER for partition 1.
Terminated zell-chaos-zeebe-2
Handle zbchaos job [key: 2251799813685513]
Running command with args: [verify readiness] 
Connecting to zell-chaos
Running experiment in self-managed environment.
Pod zell-chaos-zeebe-2 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Pending, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds.
All Zeebe nodes are running.
Handle zbchaos job [key: 2251799813685852]
Running command with args: [deploy process] 
Connecting to zell-chaos
Running experiment in self-managed environment.
Successfully created port forwarding tunnel
Deploy file bpmn/one_task.bpmn (size: 2526 bytes).
Deployed process model bpmn/one_task.bpmn successful with key 2251799813685431.
Deployed given process model , under key 2251799813685431!
Handle zbchaos job [key: 2251799813685896]
Running command with args: [verify instance-creation --partitionId 1] 
Connecting to zell-chaos
Running experiment in self-managed environment.
Successfully created port forwarding tunnel
Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false]
Created process instance with key 4503599627372125 on partition 2, required partition 1.
Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false]
Created process instance with key 6755399441057377 on partition 3, required partition 1.
Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false]
Created process instance with key 2251799813686946 on partition 1, required partition 1.
The steady-state was successfully verified!
Handle zbchaos job [key: 2251799813685990]
Running command with args: [version] 
zbchaos development (commit: HEAD)
Handle zbchaos job [key: 2251799813686033]
Running command with args: [version] 
zbchaos development (commit: HEAD)
Handle zbchaos job [key: 2251799813686179]
Running command with args: [version] 
zbchaos development (commit: HEAD)
Instance 2251799813685255 [definition 2251799813685253 ] completed
--- PASS: Test_ShouldBeAbleToRunExperiments (41.89s)

```
  • Loading branch information
ChrisKujawa authored Dec 7, 2022
2 parents 67c6f5e + f21a34a commit 2fec132
Showing 1 changed file with 17 additions and 5 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,19 @@
"tolerance": 0,
"provider": {
"type": "process",
"path": "verify-readiness.sh",
"path": "zbchaos",
"arguments": ["verify", "readiness"],
"timeout": 900
}
},
{
"name": "Can deploy process model",
"type": "probe",
"tolerance": 0,
"provider": {
"type": "process",
"path": "zbchaos",
"arguments": ["deploy", "process"],
"timeout": 900
}
},
Expand All @@ -25,8 +37,8 @@
"tolerance": 0,
"provider": {
"type": "process",
"path": "verify-steady-state.sh",
"arguments": "1",
"path": "zbchaos",
"arguments": ["verify", "instance-creation", "--partitionId", "1"],
"timeout": 900
}
}
Expand All @@ -38,8 +50,8 @@
"name": "Terminate follower of partition 1",
"provider": {
"type": "process",
"path": "terminate-partition.sh",
"arguments": [ "Follower", "1"]
"path": "zbchaos",
"arguments": ["terminate", "broker", "--role", "FOLLOWER", "--partitionId", "1"]
}
}
],
Expand Down

0 comments on commit 2fec132

Please sign in to comment.