fix: allow running as root to inject chaos #525
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lenaschoenburg
force-pushed
the
os/fix-stress-test-permissions
branch
from
d0fa86f to
b2a5178
Compare
lenaschoenburg
changed the title
|
I'm still getting errors in testbench. Can't reproduce locally though.
Output: |
|
One issue was that we did not disable reconciliation before enabling root access. Still, even after fixing this, I did not get any output from running the apt commands to install stress and procps. Executing manually in a zeebe pod that was set up for root access and then running I think at this point there are three unresolved issues:
|
|
One idea I have to overcome this is to attach ephemeral debug containers that have the necessary tools installed. |
|
Was also thinking a out this but thought might not work with permissions? Another one I was thinking whether we could leverage ebpf |
|
I think I got the permissions working: |
|
@oleschoenburg was it working? Can we get it merged? |
|
@Zelldon I have something ready locally that seems to work more or less but now the scaling test doesn't work anymore ?! I need to test this manually a bit more. |
lenaschoenburg
force-pushed
the
os/fix-stress-test-permissions
branch
from
ce187fb to
d145b85
Compare
This overwrites the security context on both the container and the deployment/statefulset. Only works because reconciliation is already disabled at that point.
lenaschoenburg
force-pushed
the
os/fix-stress-test-permissions
branch
2 times, most recently
from
a195ce6 to
859486b
Compare
Due to camunda/camunda#17347 we can't rely on a sensible value
lenaschoenburg
force-pushed
the
os/fix-stress-test-permissions
branch
from
859486b to
755f387
Compare
ChrisKujawa
reviewed
Apr 15, 2024
| cmd := []string{"ip", "route", "replace", "unreachable", podIp} | ||
| cmdWithSetup := []string{"sh", "-c", "apt update && apt install -y iproute2 && " + strings.Join(cmd, " ")} | ||
| var containerName string | ||
| if strings.Contains(podName, "gateway") { |
There was a problem hiding this comment.
Is this to work with SaaS/SM?
There was a problem hiding this comment.
We have also functions to get the right pod name.
There was a problem hiding this comment.
This is to attach to the correct container, both on SaaS and self-managed. We already have the right podName here. Do you mean we have helper functions to get the right container name?
There was a problem hiding this comment.
The pod name you use is actually only correct in SaaS in SM it is different but not sure whether you use the pod name here anyhow.
There was a problem hiding this comment.
This code here uses the pod name to figure out the correct container name. If the pod name includes gateway, we assume that the container name is zeebe-gateway, otherwise we expect it to be zeebe. I think this should work in both SaaS and SM, right?
| @@ -302,6 +303,35 @@ func (c K8Client) createPortForwardUrl(names []string) *url.URL { | |||
| return portForwardCreateURL | |||
| } | |||
|
|
|||
| func (c K8Client) ExecuteCommandViaDebugContainer(podName string, containerName string, debugImage string, cmd []string) error { | |||
There was a problem hiding this comment.
Did you verified whether it was for real disrupting the network? Or maybe it was just silently failing? 🤔 So for example did the brokers showed that they can't connect?
There was a problem hiding this comment.
Not yet! That's one of the things I still have to do :-/
There was a problem hiding this comment.
I've tested it now with a SaaS cluster 🎉
$ zbchaos --namespace 8fe648b5-1137-4d6a-af0d-be8b467fd67e-zeebe disconnect brokers --broker2NodeId 2 --broker1NodeId 1 --verbose
Flags: {1 LEADER -1 10 msg false 1 LEADER 1 2 LEADER 2 1 1713797975967 false false true false false 30 false -1 benchmark 30 8fe648b5-1137-4d6a-af0d-be8b467fd67e-zeebe 1 1 benchmark-task 0 0 0 1 -1 true}
Connecting to 8fe648b5-1137-4d6a-af0d-be8b467fd67e-zeebe
Running experiment in SaaS environment.
Patched statefulset
Port forward to zeebe-gateway-5489b4cdcf-ptfl9
Successfully created port forwarding tunnel from 46535 (local) to 26500 (remote)
Found Broker zeebe-1 with node id 1.
Found Broker zeebe-2 with node id 2.
Debug container debug-lvccl4 is running command [sh -c apt update && apt install -y iproute2 && ip route replace unreachable 10.64.73.38]
Disconnect zeebe-1 from zeebe-2
Debug container debug-658dxp is running command [sh -c apt update && apt install -y iproute2 && ip route replace unreachable 10.64.53.29]
Disconnect zeebe-2 from zeebe-1
$ zbchaos --namespace 8fe648b5-1137-4d6a-af0d-be8b467fd67e-zeebe connect brokers --verbose
Flags: {1 LEADER -1 10 msg false 1 LEADER -1 2 LEADER -1 1 1713798188959 false false true false false 30 false -1 benchmark 30 8fe648b5-1137-4d6a-af0d-be8b467fd67e-zeebe 1 1 benchmark-task 0 0 0 1 -1 true}
Connecting to 8fe648b5-1137-4d6a-af0d-be8b467fd67e-zeebe
Running experiment in SaaS environment.
Debug container debug-cv444p is running command [sh -c apt update && apt install -y iproute2 && ip route del $(ip route | grep -m 1 unreachable)]
Connected zeebe-0 again, removed unreachable routes.
Debug container debug-zg45fm is running command [sh -c apt update && apt install -y iproute2 && ip route del $(ip route | grep -m 1 unreachable)]
Connected zeebe-1 again, removed unreachable routes.
Debug container debug-tm9pdj is running command [sh -c apt update && apt install -y iproute2 && ip route del $(ip route | grep -m 1 unreachable)]
Connected zeebe-2 again, removed unreachable routes.
ChrisKujawa
approved these changes
Apr 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #520