Support disconnect gateway within zbchaos

[ChrisKujawa]

blocked by #232

• Allow to disconnect first gateway with broker of specific nodeId or partitionId and role.
• Allow to disconnect first gateway with all brokers.
• Allow to disconnect only one direction (asymetric)
• Allow to connect all again

Features are similar or more than:

• https://github.com/zeebe-io/zeebe-chaos/blob/main/chaos-workers/chaos-experiments/scripts/disconnect-standalone-gateway.sh
• https://github.com/zeebe-io/zeebe-chaos/blob/main/chaos-workers/chaos-experiments/scripts/connect-standalone-gateway.sh

Note: right now we only support disconnecting the first standalone gateway, this was how we did it before as well. We can improve that later.

closes #225

---

Example

Example usage:

$ ./zbchaos disconnect gateway
Disconnect zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 from zell-chaos-zeebe-2
Disconnect zell-chaos-zeebe-2 from zell-chaos-zeebe-gateway-7c4f86f95f-2kp24

In gateway we can see logs like: "Failed to activate jobs for type benchmark-task from partition 2

One direction:

$ ./zbchaos disconnect gateway --one-direction
Disconnect zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 from zell-chaos-zeebe-2

With verbosity:

$ ./zbchaos disconnect gateway --verbose --one-direction
Connecting to zell-chaos
Running experiment in self-managed environment.
Did not find zeebe cluster to pause reconciliation, ignoring. 
Patched statefulset
Patched deployment
Successfully created port forwarding tunnel
Found Broker zell-chaos-zeebe-2 as LEADER for partition 2.
Execute ["apt" "-qq" "update"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
4 packages can be upgraded. Run 'apt list --upgradable' to see them.
Execute ["apt" "-qq" "install" "-y" "iproute2"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
iproute2 is already the newest version (5.5.0-1ubuntu1).
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
Execute ["ip" "route" "replace" "unreachable" "10.0.30.4"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
Disconnect zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 from zell-chaos-zeebe-2

With nodeId

$ ./zbchaos disconnect gateway --verbose --one-direction --nodeId 1
Connecting to zell-chaos
Running experiment in self-managed environment.
Did not find zeebe cluster to pause reconciliation, ignoring. 
Patched statefulset
Patched deployment
Successfully created port forwarding tunnel
Found Broker zell-chaos-zeebe-1 with node id 1.
Execute ["apt" "-qq" "update"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
4 packages can be upgraded. Run 'apt list --upgradable' to see them.
Execute ["apt" "-qq" "install" "-y" "iproute2"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
iproute2 is already the newest version (5.5.0-1ubuntu1).
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
Execute ["ip" "route" "replace" "unreachable" "10.0.17.4"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
Disconnect zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 from zell-chaos-zeebe-1

When specifying node id and partition id

$ ./zbchaos disconnect gateway --verbose --one-direction --nodeId 1 --partitionId 2
Error: if any flags in the group [partitionId nodeId] are set none of the others can be; [nodeId partitionId] were all set

Specify partition

$ ./zbchaos disconnect gateway --verbose --one-direction --partitionId 3
Connecting to zell-chaos
Running experiment in self-managed environment.
Did not find zeebe cluster to pause reconciliation, ignoring. 
Patched statefulset
Patched deployment
Successfully created port forwarding tunnel
Found Broker zell-chaos-zeebe-0 as LEADER for partition 3.
Execute ["apt" "-qq" "update"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
4 packages can be upgraded. Run 'apt list --upgradable' to see them.
Execute ["apt" "-qq" "install" "-y" "iproute2"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
iproute2 is already the newest version (5.5.0-1ubuntu1).
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
Execute ["ip" "route" "replace" "unreachable" "10.0.12.5"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
Disconnect zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 from zell-chaos-zeebe-0

Disconnect all

$ ./zbchaos disconnect gateway --verbose --one-direction --all
Connecting to zell-chaos
Running experiment in self-managed environment.
Did not find zeebe cluster to pause reconciliation, ignoring. 
Patched statefulset
Patched deployment
Successfully created port forwarding tunnel
Execute ["apt" "-qq" "update"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
4 packages can be upgraded. Run 'apt list --upgradable' to see them.
Execute ["apt" "-qq" "install" "-y" "iproute2"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
iproute2 is already the newest version (5.5.0-1ubuntu1).
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
Execute ["ip" "route" "replace" "unreachable" "10.0.12.5"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
Disconnect zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 from zell-chaos-zeebe-0
Execute ["apt" "-qq" "update"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
4 packages can be upgraded. Run 'apt list --upgradable' to see them.
Execute ["apt" "-qq" "install" "-y" "iproute2"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
iproute2 is already the newest version (5.5.0-1ubuntu1).
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
Execute ["ip" "route" "replace" "unreachable" "10.0.17.4"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
Disconnect zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 from zell-chaos-zeebe-1
Execute ["apt" "-qq" "update"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
4 packages can be upgraded. Run 'apt list --upgradable' to see them.
Execute ["apt" "-qq" "install" "-y" "iproute2"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
iproute2 is already the newest version (5.5.0-1ubuntu1).
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
Execute ["ip" "route" "replace" "unreachable" "10.0.30.4"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
Disconnect zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 from zell-chaos-zeebe-2

Connect all again

$ ./zbchaos connect gateway --verbose
Connecting to zell-chaos
Running experiment in self-managed environment.
Execute ["sh" "-c" "command -v ip"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
/usr/sbin/ip
Execute ["sh" "-c" "ip route del unreachable 10.0.12.5"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
Connected zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 again with zell-chaos-zeebe-0, removed unreachable routes.
Execute ["sh" "-c" "command -v ip"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
/usr/sbin/ip
Execute ["sh" "-c" "ip route del unreachable 10.0.17.4"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
Connected zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 again with zell-chaos-zeebe-1, removed unreachable routes.
Execute ["sh" "-c" "command -v ip"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
/usr/sbin/ip
Execute ["sh" "-c" "ip route del unreachable 10.0.30.4"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24
Connected zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 again with zell-chaos-zeebe-2, removed unreachable routes.

[npepinpe]

🚀