I can disconnect the gateway via Zbchaos #225
Assignees
Comments
This was referenced Nov 16, 2022
ChrisKujawa
added a commit
that referenced
this issue
Nov 18, 2022
blocked by #232 - Allow to disconnect first gateway with broker of specific nodeId or partitionId and role. - Allow to disconnect first gateway with all brokers. - Allow to disconnect only one direction (asymetric) - Allow to connect all again Features are similar or more than: * https://github.com/zeebe-io/zeebe-chaos/blob/main/chaos-workers/chaos-experiments/scripts/disconnect-standalone-gateway.sh * https://github.com/zeebe-io/zeebe-chaos/blob/main/chaos-workers/chaos-experiments/scripts/connect-standalone-gateway.sh Note: right now we only support disconnecting the first standalone gateway, this was how we did it before as well. We can improve that later. closes #225 ----- ### Example Example usage: ``` $ ./zbchaos disconnect gateway Disconnect zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 from zell-chaos-zeebe-2 Disconnect zell-chaos-zeebe-2 from zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 ``` In gateway we can see logs like: `"Failed to activate jobs for type benchmark-task from partition 2` One direction: ``` $ ./zbchaos disconnect gateway --one-direction Disconnect zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 from zell-chaos-zeebe-2 ``` With verbosity: ``` $ ./zbchaos disconnect gateway --verbose --one-direction Connecting to zell-chaos Running experiment in self-managed environment. Did not find zeebe cluster to pause reconciliation, ignoring. Patched statefulset Patched deployment Successfully created port forwarding tunnel Found Broker zell-chaos-zeebe-2 as LEADER for partition 2. Execute ["apt" "-qq" "update"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 4 packages can be upgraded. Run 'apt list --upgradable' to see them. Execute ["apt" "-qq" "install" "-y" "iproute2"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 iproute2 is already the newest version (5.5.0-1ubuntu1). 0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded. Execute ["ip" "route" "replace" "unreachable" "10.0.30.4"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 Disconnect zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 from zell-chaos-zeebe-2 ``` With nodeId ``` $ ./zbchaos disconnect gateway --verbose --one-direction --nodeId 1 Connecting to zell-chaos Running experiment in self-managed environment. Did not find zeebe cluster to pause reconciliation, ignoring. Patched statefulset Patched deployment Successfully created port forwarding tunnel Found Broker zell-chaos-zeebe-1 with node id 1. Execute ["apt" "-qq" "update"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 4 packages can be upgraded. Run 'apt list --upgradable' to see them. Execute ["apt" "-qq" "install" "-y" "iproute2"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 iproute2 is already the newest version (5.5.0-1ubuntu1). 0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded. Execute ["ip" "route" "replace" "unreachable" "10.0.17.4"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 Disconnect zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 from zell-chaos-zeebe-1 ``` When specifying node id and partition id ``` $ ./zbchaos disconnect gateway --verbose --one-direction --nodeId 1 --partitionId 2 Error: if any flags in the group [partitionId nodeId] are set none of the others can be; [nodeId partitionId] were all set ``` Specify partition ``` $ ./zbchaos disconnect gateway --verbose --one-direction --partitionId 3 Connecting to zell-chaos Running experiment in self-managed environment. Did not find zeebe cluster to pause reconciliation, ignoring. Patched statefulset Patched deployment Successfully created port forwarding tunnel Found Broker zell-chaos-zeebe-0 as LEADER for partition 3. Execute ["apt" "-qq" "update"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 4 packages can be upgraded. Run 'apt list --upgradable' to see them. Execute ["apt" "-qq" "install" "-y" "iproute2"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 iproute2 is already the newest version (5.5.0-1ubuntu1). 0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded. Execute ["ip" "route" "replace" "unreachable" "10.0.12.5"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 Disconnect zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 from zell-chaos-zeebe-0 ``` Disconnect all ``` $ ./zbchaos disconnect gateway --verbose --one-direction --all Connecting to zell-chaos Running experiment in self-managed environment. Did not find zeebe cluster to pause reconciliation, ignoring. Patched statefulset Patched deployment Successfully created port forwarding tunnel Execute ["apt" "-qq" "update"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 4 packages can be upgraded. Run 'apt list --upgradable' to see them. Execute ["apt" "-qq" "install" "-y" "iproute2"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 iproute2 is already the newest version (5.5.0-1ubuntu1). 0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded. Execute ["ip" "route" "replace" "unreachable" "10.0.12.5"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 Disconnect zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 from zell-chaos-zeebe-0 Execute ["apt" "-qq" "update"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 4 packages can be upgraded. Run 'apt list --upgradable' to see them. Execute ["apt" "-qq" "install" "-y" "iproute2"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 iproute2 is already the newest version (5.5.0-1ubuntu1). 0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded. Execute ["ip" "route" "replace" "unreachable" "10.0.17.4"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 Disconnect zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 from zell-chaos-zeebe-1 Execute ["apt" "-qq" "update"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 4 packages can be upgraded. Run 'apt list --upgradable' to see them. Execute ["apt" "-qq" "install" "-y" "iproute2"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 iproute2 is already the newest version (5.5.0-1ubuntu1). 0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded. Execute ["ip" "route" "replace" "unreachable" "10.0.30.4"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 Disconnect zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 from zell-chaos-zeebe-2 ``` Connect all again ``` $ ./zbchaos connect gateway --verbose Connecting to zell-chaos Running experiment in self-managed environment. Execute ["sh" "-c" "command -v ip"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 /usr/sbin/ip Execute ["sh" "-c" "ip route del unreachable 10.0.12.5"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 Connected zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 again with zell-chaos-zeebe-0, removed unreachable routes. Execute ["sh" "-c" "command -v ip"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 /usr/sbin/ip Execute ["sh" "-c" "ip route del unreachable 10.0.17.4"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 Connected zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 again with zell-chaos-zeebe-1, removed unreachable routes. Execute ["sh" "-c" "command -v ip"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 /usr/sbin/ip Execute ["sh" "-c" "ip route del unreachable 10.0.30.4"] on pod zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 Connected zell-chaos-zeebe-gateway-7c4f86f95f-2kp24 again with zell-chaos-zeebe-2, removed unreachable routes. ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
related to #139
The text was updated successfully, but these errors were encountered: