Adjust all chaos experiments to use the zbchaos tool #237
Assignees
Comments
29 tasks
This was referenced Dec 6, 2022
ChrisKujawa
added a commit
that referenced
this issue
Dec 6, 2022
Based on #267 (blocked by) related to #237 ------- **Fixed some smaller issues, like:** * rm one-direction from mutex * make possible to run workers against self-managed clusters * return correct errors on connect **Migrate the Deployment distribution experiment, as the first experiment, to zbchaos.** The experiment was executed and verified via the integration test against a self-managed cluster. I moved the experiment into the `chaos-experiments/camunda-cloud/test/` folder and migrated it, with that approach I was able to execute the experiment with `eze` and running against my self-managed `zell-chaos` zeebe cluster. Log output: ``` Deploy file bpmn/chaos/actionRunner.bpmn (size: 8788 bytes). Deployed process model bpmn/chaos/actionRunner.bpmn successful with key 2251799813685249. Deploy file bpmn/chaos/chaosExperiment.bpmn (size: 21403 bytes). Deployed process model bpmn/chaos/chaosExperiment.bpmn successful with key 2251799813685251. Deploy file bpmn/chaos/chaosToolkit.bpmn (size: 11031 bytes). Deployed process model bpmn/chaos/chaosToolkit.bpmn successful with key 2251799813685253. Create ChaosToolkit instance Open workers: [zbchaos, readExperiments]. Handle read experiments job [key: 2251799813685265] Read experiments successful, complete job with: {"experiments":[{"contributions":{"availability":"high","reliability":"high"},"description":"Zeebe deployment distribution should be fault-tolerant. Zeebe should be able to handle network outages and fail-overs and distribute the deployments after partitions are available again.","method":[{"name":"Create network partition between leaders","provider":{"arguments":["disconnect","brokers","--broker1PartitionId","1","--broker2PartitionId","3","--one-direction"],"path":"zbchaos","timeout":900,"type":"process"},"type":"action"},{"name":"Deploy different deployment versions.","provider":{"arguments":["deploy","process","--multipleVersions","10"],"path":"zbchaos","timeout":900,"type":"process"},"type":"action"},{"name":"Connect leaders again","provider":{"arguments":["connect","brokers"],"path":"zbchaos","timeout":900,"type":"process"},"type":"action"},{"name":"Create process instance of latest version on partition one","provider":{"arguments":["verify","instance-creation","--bpmnProcessId","multiVersion","--version","10","--partitionId","1"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Create process instance of latest version on partition two","provider":{"arguments":["verify","instance-creation","--bpmnProcessId","multiVersion","--version","10","--partitionId","2"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Create process instance of latest version on partition three","provider":{"arguments":["verify","instance-creation","--bpmnProcessId","multiVersion","--version","10","--partitionId","3"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"All pods should be ready","provider":{"arguments":["verify","readiness"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"Zeebe deployment distribution","version":"0.1.0"},{"contributions":{"availability":"high","reliability":"high"},"description":"This fake experiment is just to test the integration with Zeebe and zbchaos workers","method":[{"name":"Show again the version","pauses":{"after":5},"provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"Show version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"This is a fake experiment","version":"0.1.0"}]}. Handle zbchaos job [key: 2251799813685328] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813685376] Running command with args: [disconnect brokers --broker1PartitionId 1 --broker2PartitionId 3 --one-direction] Connecting to zell-chaos Running experiment in self-managed environment. Did not find zeebe cluster to pause reconciliation, ignoring. Patched statefulset Successfully created port forwarding tunnel Found Broker zell-chaos-zeebe-1 as LEADER for partition 1. Found Broker zell-chaos-zeebe-2 as LEADER for partition 3. Execute ["apt" "-qq" "update"] on pod zell-chaos-zeebe-1 Execute ["apt" "-qq" "install" "-y" "iproute2"] on pod zell-chaos-zeebe-1 Execute ["ip" "route" "replace" "unreachable" "10.0.4.223"] on pod zell-chaos-zeebe-1 Disconnect zell-chaos-zeebe-1 from zell-chaos-zeebe-2 Handle zbchaos job [key: 2251799813685585] Running command with args: [deploy process --multipleVersions 10] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Deploy 10 versions of different type of models. Deployed [2/10] versions. Deployed [4/10] versions. Deployed [6/10] versions. Deployed [8/10] versions. Deployed [10/10] versions. Deployed different process models of different types and versions to zeebe! Handle zbchaos job [key: 2251799813685677] Running command with args: [connect brokers] Connecting to zell-chaos Running experiment in self-managed environment. Execute ["sh" "-c" "command -v ip"] on pod zell-chaos-zeebe-0 Error on connection Broker: zell-chaos-zeebe-0. Error: Execution exited with exit code 127 (Command not found). It is likely that the broker was not disconnected or restarted in between. Execute ["sh" "-c" "command -v ip"] on pod zell-chaos-zeebe-1 Execute ["sh" "-c" "ip route | grep -m 1 unreachable"] on pod zell-chaos-zeebe-1 Execute ["sh" "-c" "ip route del "] on pod zell-chaos-zeebe-1 Error on connection Broker: zell-chaos-zeebe-1. Error: command terminated with exit code 255 Execute ["sh" "-c" "command -v ip"] on pod zell-chaos-zeebe-2 Error on connection Broker: zell-chaos-zeebe-2. Error: Execution exited with exit code 127 (Command not found). It is likely that the broker was not disconnected or restarted in between. Handle zbchaos job [key: 2251799813685732] Running command with args: [verify instance-creation --bpmnProcessId multiVersion --version 10 --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Create process instance with BPMN process ID multiVersion and version 10 [variables: '', awaitResult: false] Created process instance with key 2251799815354503 on partition 1, required partition 1. The steady-state was successfully verified! Handle zbchaos job [key: 2251799813685779] Running command with args: [verify instance-creation --bpmnProcessId multiVersion --version 10 --partitionId 2] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Create process instance with BPMN process ID multiVersion and version 10 [variables: '', awaitResult: false] Created process instance with key 4503599628043172 on partition 2, required partition 2. The steady-state was successfully verified! Handle zbchaos job [key: 2251799813685825] Running command with args: [verify instance-creation --bpmnProcessId multiVersion --version 10 --partitionId 3] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Create process instance with BPMN process ID multiVersion and version 10 [variables: '', awaitResult: false] Created process instance with key 2251799815355181 on partition 1, required partition 3. Created process instance with key 2251799815355311 on partition 1, required partition 3. Created process instance with key 6755399441722396 on partition 3, required partition 3. The steady-state was successfully verified! Handle zbchaos job [key: 2251799813685877] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813685969] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813686012] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813686157] Running command with args: [version] zbchaos development (commit: HEAD) Instance 2251799813685255 [definition 2251799813685253 ] completed --- PASS: Test_ShouldBeAbleToRunExperiments (36.97s) PASS Process finished with the exit code 0 ```
This was referenced Dec 6, 2022
Merged
ChrisKujawa
added a commit
that referenced
this issue
Dec 7, 2022
✅ Depends on #268 related to #237 ------------ Migrates the follower restart experiment and tested with integration test Log output: ``` Deploy file bpmn/chaos/actionRunner.bpmn (size: 8788 bytes). Deployed process model bpmn/chaos/actionRunner.bpmn successful with key 2251799813685249. Deploy file bpmn/chaos/chaosExperiment.bpmn (size: 21403 bytes). Deployed process model bpmn/chaos/chaosExperiment.bpmn successful with key 2251799813685251. Deploy file bpmn/chaos/chaosToolkit.bpmn (size: 11031 bytes). Deployed process model bpmn/chaos/chaosToolkit.bpmn successful with key 2251799813685253. Create ChaosToolkit instance Open workers: [zbchaos, readExperiments]. Handle read experiments job [key: 2251799813685265] Read experiments successful, complete job with: {"experiments":[{"contributions":{"availability":"high","reliability":"high"},"description":"Zeebe should be fault-tolerant. Zeebe should be able to handle follower restarts.","method":[{"name":"Restart follower of partition 1 gracefully","provider":{"arguments":["restart","broker","--role","FOLLOWER","--partitionId","1"],"path":"zbchaos","type":"process"},"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"All pods should be ready","provider":{"arguments":["verify","readiness"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Can deploy process model","provider":{"arguments":["deploy","process"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Should be able to create process instances on partition 1","provider":{"arguments":["verify","instance-creation","--partitionId","1"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"Zeebe follower graceful restart experiment","version":"0.1.0"},{"contributions":{"availability":"high","reliability":"high"},"description":"This fake experiment is just to test the integration with Zeebe and zbchaos workers","method":[{"name":"Show again the version","pauses":{"after":5},"provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"Show version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"This is a fake experiment","version":"0.1.0"}]}. Handle zbchaos job [key: 2251799813685328] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813685380] Running command with args: [deploy process] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Deploy file bpmn/one_task.bpmn (size: 2526 bytes). Deployed process model bpmn/one_task.bpmn successful with key 2251799813685431. Deployed given process model , under key 2251799813685431! Handle zbchaos job [key: 2251799813685424] Running command with args: [verify instance-creation --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 4503599627371757 on partition 2, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 6755399441057008 on partition 3, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 2251799813686572 on partition 1, required partition 1. The steady-state was successfully verified! Handle zbchaos job [key: 2251799813685472] Running command with args: [restart broker --role FOLLOWER --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Found Broker zell-chaos-zeebe-2 as FOLLOWER for partition 1. Restarted zell-chaos-zeebe-2 Handle zbchaos job [key: 2251799813685518] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813685559] Running command with args: [deploy process] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Deploy file bpmn/one_task.bpmn (size: 2526 bytes). Deployed process model bpmn/one_task.bpmn successful with key 2251799813685431. Deployed given process model , under key 2251799813685431! Handle zbchaos job [key: 2251799813685603] Running command with args: [verify instance-creation --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 4503599627371783 on partition 2, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 6755399441057035 on partition 3, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 2251799813686599 on partition 1, required partition 1. The steady-state was successfully verified! Handle zbchaos job [key: 2251799813685698] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813685740] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813685885] Running command with args: [version] zbchaos development (commit: HEAD) Instance 2251799813685255 [definition 2251799813685253 ] completed --- PASS: Test_ShouldBeAbleToRunExperiments (12.03s) ```
ChrisKujawa
added a commit
that referenced
this issue
Dec 7, 2022
depends #269 related to #237 ----- Migrates the follower terminate experiment and tested with integration test Log output: ``` Deploy file bpmn/chaos/actionRunner.bpmn (size: 8788 bytes). Deployed process model bpmn/chaos/actionRunner.bpmn successful with key 2251799813685249. Deploy file bpmn/chaos/chaosExperiment.bpmn (size: 21403 bytes). Deployed process model bpmn/chaos/chaosExperiment.bpmn successful with key 2251799813685251. Deploy file bpmn/chaos/chaosToolkit.bpmn (size: 11031 bytes). Deployed process model bpmn/chaos/chaosToolkit.bpmn successful with key 2251799813685253. Create ChaosToolkit instance Open workers: [zbchaos, readExperiments]. Handle read experiments job [key: 2251799813685265] Read experiments successful, complete job with: {"experiments":[{"contributions":{"availability":"high","reliability":"high"},"description":"Zeebe should be fault-tolerant. Zeebe should be able to handle followers terminations.","method":[{"name":"Terminate follower of partition 1","provider":{"arguments":["terminate","broker","--role","FOLLOWER","--partitionId","1"],"path":"zbchaos","type":"process"},"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"All pods should be ready","provider":{"arguments":["verify","readiness"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Can deploy process model","provider":{"arguments":["deploy","process"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Should be able to create process instances on partition 1","provider":{"arguments":["verify","instance-creation","--partitionId","1"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"Zeebe follower restart non-graceful experiment","version":"0.1.0"},{"contributions":{"availability":"high","reliability":"high"},"description":"This fake experiment is just to test the integration with Zeebe and zbchaos workers","method":[{"name":"Show again the version","pauses":{"after":5},"provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"Show version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"This is a fake experiment","version":"0.1.0"}]}. Handle zbchaos job [key: 2251799813685328] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813685374] Running command with args: [deploy process] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Deploy file bpmn/one_task.bpmn (size: 2526 bytes). Deployed process model bpmn/one_task.bpmn successful with key 2251799813685431. Deployed given process model , under key 2251799813685431! Handle zbchaos job [key: 2251799813685418] Running command with args: [verify instance-creation --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 4503599627372098 on partition 2, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 6755399441057346 on partition 3, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 2251799813686915 on partition 1, required partition 1. The steady-state was successfully verified! Handle zbchaos job [key: 2251799813685467] Running command with args: [terminate broker --role FOLLOWER --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Found Broker zell-chaos-zeebe-2 as FOLLOWER for partition 1. Terminated zell-chaos-zeebe-2 Handle zbchaos job [key: 2251799813685513] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. Pod zell-chaos-zeebe-2 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813685852] Running command with args: [deploy process] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Deploy file bpmn/one_task.bpmn (size: 2526 bytes). Deployed process model bpmn/one_task.bpmn successful with key 2251799813685431. Deployed given process model , under key 2251799813685431! Handle zbchaos job [key: 2251799813685896] Running command with args: [verify instance-creation --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 4503599627372125 on partition 2, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 6755399441057377 on partition 3, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 2251799813686946 on partition 1, required partition 1. The steady-state was successfully verified! Handle zbchaos job [key: 2251799813685990] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813686033] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813686179] Running command with args: [version] zbchaos development (commit: HEAD) Instance 2251799813685255 [definition 2251799813685253 ] completed --- PASS: Test_ShouldBeAbleToRunExperiments (41.89s) ```
ChrisKujawa
added a commit
that referenced
this issue
Dec 7, 2022
blocked by #269 blocked by #270 related to #237 ------ Migrated the leader restart experiment to the zbchaos toolkit Log output: ``` Deployed process model bpmn/chaos/actionRunner.bpmn successful with key 2251799813685249. Deploy file bpmn/chaos/chaosExperiment.bpmn (size: 21403 bytes). Deployed process model bpmn/chaos/chaosExperiment.bpmn successful with key 2251799813685251. Deploy file bpmn/chaos/chaosToolkit.bpmn (size: 11031 bytes). Deployed process model bpmn/chaos/chaosToolkit.bpmn successful with key 2251799813685253. Create ChaosToolkit instance Open workers: [zbchaos, readExperiments]. Handle read experiments job [key: 2251799813685265] Read experiments successful, complete job with: {"experiments":[{"contributions":{"availability":"high","reliability":"high"},"description":"This fake experiment is just to test the integration with Zeebe and zbchaos workers","method":[{"name":"Show again the version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"Show version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"This is a fake experiment","version":"0.1.0"},{"contributions":{"availability":"high","reliability":"high"},"description":"Zeebe should be fault-tolerant. Zeebe should recover after a partition leader was restarted gracefully.","method":[{"name":"Restart leader of partition 1","provider":{"arguments":["restart","broker","--role","LEADER","--partitionId","1"],"path":"zbchaos","type":"process"},"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"All pods should be ready","provider":{"arguments":["verify","readiness"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Can deploy process model","provider":{"arguments":["deploy","process"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Should be able to create process instances on partition 1","provider":{"arguments":["verify","instance-creation","--partitionId","1"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"Zeebe Leader restart gracefully experiment","version":"0.1.0"}]}. Handle zbchaos job [key: 2251799813685328] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813685374] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813685417] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813685508] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813685551] Running command with args: [deploy process] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Deploy file bpmn/one_task.bpmn (size: 2526 bytes). Deployed process model bpmn/one_task.bpmn successful with key 2251799813685431. Deployed given process model , under key 2251799813685431! Handle zbchaos job [key: 2251799813685598] Running command with args: [verify instance-creation --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 2251799813689038 on partition 1, required partition 1. The steady-state was successfully verified! Handle zbchaos job [key: 2251799813685644] Running command with args: [restart broker --role LEADER --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Found Broker zell-chaos-zeebe-2 as LEADER for partition 1. Restarted zell-chaos-zeebe-2 Handle zbchaos job [key: 2251799813685690] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813685732] Running command with args: [deploy process] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Deploy file bpmn/one_task.bpmn (size: 2526 bytes). Deployed process model bpmn/one_task.bpmn successful with key 2251799813685431. Deployed given process model , under key 2251799813685431! Handle zbchaos job [key: 2251799813685803] Running command with args: [verify instance-creation --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 4503599627374207 on partition 2, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 2251799813689056 on partition 1, required partition 1. The steady-state was successfully verified! Instance 2251799813685255 [definition 2251799813685253 ] completed --- PASS: Test_ShouldBeAbleToRunExperiments (11.98s) PASS ```
ChrisKujawa
added a commit
that referenced
this issue
Dec 7, 2022
blocked by #271 related to #237 ----- Similar to the other migrated experiments I followed same approach as described here #268 > The experiment was executed and verified via the integration test against a self-managed cluster. > > I moved the experiment into the chaos-experiments/camunda-cloud/test/ folder and migrated it, with that approach I was able to execute the experiment with eze and running against my self-managed zell-chaos zeebe cluster. Log output: ``` Create ChaosToolkit instance Open workers: [zbchaos, readExperiments]. Handle read experiments job [key: 2251799813685265] Read experiments successful, complete job with: {"experiments":[{"contributions":{"availability":"high","reliability":"high"},"description":"This fake experiment is just to test the integration with Zeebe and zbchaos workers","method":[{"name":"Show again the version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"Show version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"This is a fake experiment","version":"0.1.0"},{"contributions":{"availability":"high","reliability":"high"},"description":"Zeebe should be fault-tolerant. We expect that Zeebe can handle non-graceful leader restarts.","method":[{"name":"Terminate leader of partition 1 non-gracefully","provider":{"arguments":["terminate","broker","--role","LEADER","--partitionId","1"],"path":"zbchaos","type":"process"},"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"All pods should be ready","provider":{"arguments":["verify","readiness"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Can deploy process model","provider":{"arguments":["deploy","process"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Should be able to create process instances on partition 1","provider":{"arguments":["verify","instance-creation","--partitionId","1"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"Zeebe Leader restart non-graceful experiment","version":"0.1.0"}]}. Handle zbchaos job [key: 2251799813685328] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813685374] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813685417] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813685508] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813685551] Running command with args: [deploy process] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Deploy file bpmn/one_task.bpmn (size: 2526 bytes). Deployed process model bpmn/one_task.bpmn successful with key 2251799813685431. Deployed given process model , under key 2251799813685431! Handle zbchaos job [key: 2251799813685597] Running command with args: [verify instance-creation --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 4503599627380556 on partition 2, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 6755399441065816 on partition 3, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 2251799813695425 on partition 1, required partition 1. The steady-state was successfully verified! Handle zbchaos job [key: 2251799813685645] Running command with args: [terminate broker --role LEADER --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Found Broker zell-chaos-zeebe-1 as LEADER for partition 1. Terminated zell-chaos-zeebe-1 Handle zbchaos job [key: 2251799813685691] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-1 is in phase Running, but not ready. Wait for some seconds. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813686206] Running command with args: [deploy process] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Deploy file bpmn/one_task.bpmn (size: 2526 bytes). Deployed process model bpmn/one_task.bpmn successful with key 2251799813685431. Deployed given process model , under key 2251799813685431! Handle zbchaos job [key: 2251799813686252] Running command with args: [verify instance-creation --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 4503599627380585 on partition 2, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 6755399441065847 on partition 3, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 2251799813695455 on partition 1, required partition 1. The steady-state was successfully verified! Instance 2251799813685255 [definition 2251799813685253 ] completed --- PASS: Test_ShouldBeAbleToRunExperiments (56.98s) ```
ChrisKujawa
added a commit
that referenced
this issue
Dec 8, 2022
related to #237 Similar to the other migrated experiments I followed same approach as described here #268 > The experiment was executed and verified via the integration test against a self-managed cluster. > > I moved the experiment into the chaos-experiments/camunda-cloud/test/ folder and migrated it, with that approach I was able to execute the experiment with eze and running against my self-managed zell-chaos zeebe cluster. Log output: ``` Create ChaosToolkit instance Open workers: [zbchaos, readExperiments]. Handle read experiments job [key: 2251799813685265] Read experiments successful, complete job with: {"experiments":[{"contributions":{"availability":"high","reliability":"high"},"description":"This fake experiment is just to test the integration with Zeebe and zbchaos workers","method":[{"name":"Show again the version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"Show version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"This is a fake experiment","version":"0.1.0"},{"contributions":{"availability":"high","reliability":"high"},"description":"Zeebe message correlation should work even if the leader was restarted on which the message was published.","method":[{"name":"Deploy process model with catch event","provider":{"arguments":["deploy","process","--processModelPath","bpmn/msg-catch.bpmn"],"path":"zbchaos","type":"process"},"tolerance":0,"type":"action"},{"name":"Publish message to partition one","provider":{"arguments":["publish","--partitionId","1"],"path":"zbchaos","type":"process"},"tolerance":0,"type":"action"},{"name":"Terminate leader of partition 1 non-gracefully","provider":{"arguments":["terminate","broker","--role","LEADER","--partitionId","1"],"path":"zbchaos","type":"process"},"type":"action"},{"name":"All pods should be ready","provider":{"arguments":["verify","readiness"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Should be able to create a process instance and await the message correlation","provider":{"arguments":["verify","instance-creation","--awaitResult","--bpmnProcessId","oneReceiveMsgEvent","--variables","{\"key\": \"0\"}"],"path":"zbchaos","type":"process"},"tolerance":0,"type":"probe"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"All pods should be ready","provider":{"arguments":["verify","readiness"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"Zeebe message correlation experiment","version":"0.1.0"}]}. Handle zbchaos job [key: 2251799813685328] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813685374] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813685417] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813685508] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813685553] Running command with args: [deploy process --processModelPath bpmn/msg-catch.bpmn] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Deploy file bpmn/msg-catch.bpmn (size: 2986 bytes). Deployed process model bpmn/msg-catch.bpmn successful with key 2251799813685249. Deployed given process model bpmn/msg-catch.bpmn, under key 2251799813685249! Handle zbchaos job [key: 2251799813685609] Running command with args: [publish --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Send message 'msg', with correaltion key '0' (ASCII: 48) Message was sent and returned key 2251799813685251, which corresponds to partition: 1 Handle zbchaos job [key: 2251799813685652] Running command with args: [terminate broker --role LEADER --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Found Broker zell-chaos-zeebe-0 as LEADER for partition 1. Terminated zell-chaos-zeebe-0 Handle zbchaos job [key: 2251799813685696] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. Pod zell-chaos-zeebe-0 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-0 is in phase Running, but not ready. Wait for some seconds. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813686232] Running command with args: [verify instance-creation --awaitResult --bpmnProcessId oneReceiveMsgEvent --variables {"key": "0"}] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Send create process instance command, with BPMN process ID 'oneReceiveMsgEvent' and version '-1' (-1 means latest) [variables: '{"key": "0"}', awaitResult: true] Created process instance with key 2251799813685252 on partition 1, required partition 1. The steady-state was successfully verified! Handle zbchaos job [key: 2251799813686305] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. All Zeebe nodes are running. Instance 2251799813685255 [definition 2251799813685253 ] completed --- PASS: Test_ShouldBeAbleToRunExperiments (61.86s) PASS Process finished with the exit code 0 ```
ChrisKujawa
added a commit
that referenced
this issue
Dec 8, 2022
related to #237 Similar to the other migrated experiments I followed same approach as described here #268 > The experiment was executed and verified via the integration test against a self-managed cluster. > > I moved the experiment into the chaos-experiments/camunda-cloud/test/ folder and migrated it, with that approach I was able to execute the experiment with eze and running against my self-managed zell-chaos zeebe cluster.
Merged
ChrisKujawa
added a commit
that referenced
this issue
Dec 8, 2022
related to #237 Similar to the other migrated experiments I followed same approach as described here #268 > The experiment was executed and verified via the integration test against a self-managed cluster. > > I moved the experiment into the chaos-experiments/camunda-cloud/test/ folder and migrated it, with that approach I was able to execute the experiment with eze and running against my self-managed zell-chaos zeebe cluster. ``` Create ChaosToolkit instance Open workers: [zbchaos, readExperiments]. Handle read experiments job [key: 2251799813685265] Read experiments successful, complete job with: {"experiments":[{"contributions":{"availability":"high","reliability":"high"},"description":"This fake experiment is just to test the integration with Zeebe and zbchaos workers","method":[{"name":"Show again the version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"Show version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"This is a fake experiment","version":"0.1.0"},{"contributions":{"availability":"high","reliability":"high"},"description":"The cpu stress on an abritrary node should not cause any failures. We should be able to start and complete instances.","method":[{"name":"Stress CPU on Broker","provider":{"arguments":["stress","broker","--cpu"],"path":"zbchaos","type":"process"},"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"All pods should be ready","provider":{"arguments":["verify","readiness"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Can deploy process model","provider":{"arguments":["deploy","process"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Should be able to create process instances on partition 1","provider":{"arguments":["verify","instance-creation","--partitionId","1"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"CPU stress on an Broker","version":"0.1.0"}]}. Handle zbchaos job [key: 2251799813685328] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813685374] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813685417] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813685508] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813685551] Running command with args: [deploy process] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Deploy file bpmn/one_task.bpmn (size: 2526 bytes). Deployed process model bpmn/one_task.bpmn successful with key 2251799813685263. Deployed given process model , under key 2251799813685263! Handle zbchaos job [key: 2251799813685595] Running command with args: [verify instance-creation --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 4503599627370533 on partition 2, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 6755399441055781 on partition 3, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 2251799813685304 on partition 1, required partition 1. The steady-state was successfully verified! Handle zbchaos job [key: 2251799813685642] Running command with args: [stress broker --cpu] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Found Broker zell-chaos-zeebe-0 as LEADER for partition 1. Put stress on zell-chaos-zeebe-0 Execute ["apt" "-qq" "update"] on pod zell-chaos-zeebe-0 Execute ["apt" "-qq" "install" "-y" "stress" "procps"] on pod zell-chaos-zeebe-0 Execute ["stress" "--timeout" "30" "--cpu" "256"] on pod zell-chaos-zeebe-0 Handle zbchaos job [key: 2251799813685871] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813685913] Running command with args: [deploy process] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Deploy file bpmn/one_task.bpmn (size: 2526 bytes). Deployed process model bpmn/one_task.bpmn successful with key 2251799813685263. Deployed given process model , under key 2251799813685263! Handle zbchaos job [key: 2251799813685977] Running command with args: [verify instance-creation --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 4503599627370539 on partition 2, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 6755399441055787 on partition 3, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 2251799813685311 on partition 1, required partition 1. The steady-state was successfully verified! Instance 2251799813685257 [definition 2251799813685253 ] completed --- PASS: Test_ShouldBeAbleToRunExperiments (32.20s) PASS Process finished with the exit code 0 ```
ChrisKujawa
added a commit
that referenced
this issue
Dec 8, 2022
First fixed an issue with the worker deployments: In self-managed env the zeebe gateway service is called differently, based on the helm release name. The release name is in our setups normally also the namespace name, which we use here. If the worker is already deployed we update the deployment, instead of failing. + Added missing tests for it. Migrated the worker restart experiment related to #237 Similar to the other migrated experiments I followed same approach as described here #268 > The experiment was executed and verified via the integration test against a self-managed cluster. > > I moved the experiment into the chaos-experiments/camunda-cloud/test/ folder and migrated it, with that approach I was able to execute the experiment with eze and running against my self-managed zell-chaos zeebe cluster. ---------- This is btw the LAST production s experiment to migrate 🎉
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We need to migrate all chaos experiments to zbchaos. Ideally incrementally so we can verify whether it works.
part of #139
The text was updated successfully, but these errors were encountered: