diff --git a/docs/development_guides/add_chaos_daemon.md b/docs/development_guides/add_chaos_daemon.md new file mode 100644 index 0000000000000..dc9752ce3d7a7 --- /dev/null +++ b/docs/development_guides/add_chaos_daemon.md @@ -0,0 +1,302 @@ +--- +id: add_facilities_to_chaos_daemon +title: Add Facilities to Chaos Daemon +sidebar_label: Add Facilities to Chaos Daemon +--- + +In [Develop a new chaos](dev_hello_world.md), we have added a new chaos type named `HelloWorldChaos`, which will print `hello world` in `chaos-controller-manager`. To actually run the chaos, we need to configure some facilities for Chaos Daemon - so that `controller-manager` can select the specified Pods according to the chaos configuration and sends the chaos request to the `chaos-daemon` corresponding to these Pods. Once these are done, the `chaos-daemon` could run the chaos at last. + +This guide covers the following steps: + +- [Add selector for HelloWorldChaos](#add-selector-for-helloworldChaos) +- [Implement the gRPC interface](#implement-grpc-interface-for-chaos-request) +- [Verify your chaos](#verify-your-chaos) + +## Add selector for HelloWorldChaos + +In Chaos Mesh, we have defined the `spec.selector` field to specify the scope of the chaos by namespace, labels, annotation, etc. You can refer to [Define the Scope of Chaos Experiment](../user_guides/experiment_scope.md) for more information. To specify the Pods for `HelloWorld` chaos: + +1. Add the `Spec` field in `HelloWorldChaos`: + + ```golang + // HelloWorldChaos is the Schema for the helloworldchaos API + type HelloWorldChaos struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // Spec defines the behavior of a pod chaos experiment + Spec HelloWorldSpec `json:"spec"` + } + + type HelloWorldSpec struct { + Selector SelectorSpec `json:"selector"` + } + + // GetSelector is a getter for Selector (for implementing SelectSpec) + func (in *HelloWorldSpec) GetSelector() SelectorSpec { + return in.Selector + } + ``` + +2. Generate boilerplate functions for the `spec` field. This is required to integrate the resource in Chaos Mesh. + + ```bash + make generate + ``` + +## Implement the gRPC interface + +In order for `chaos-daemon` to accept requests from `chaos-controller-manager`, a new gRPC interface is required for `chaos-controller-manager` and `chaos-daemon`. Take the steps below to add the gRPC interface: + +1. Add the RPC in [chaosdaemon.proto](https://github.com/chaos-mesh/chaos-mesh/blob/master/pkg/chaosdaemon/pb/chaosdaemon.proto). + + ```proto + service chaosDaemon { + ... + + rpc ExecHelloWorldChaos(ExecHelloWorldRequest) returns (google.protobuf.Empty) {} + } + + message ExecHelloWorldRequest { + string container_id = 1; + } + ``` + + You will need to update golang code generated by this proto file: + + ```bash + make proto + ``` + +2. Implement the gRPC service in `chaos-daemon`. + + Add a new file named `helloworld_server.go` under [chaosdaemon](https://github.com/chaos-mesh/chaos-mesh/tree/master/pkg/chaosdaemon), with the content as below: + + ```golang + package chaosdaemon + + import ( + "context" + "fmt" + + "github.com/golang/protobuf/ptypes/empty" + + pb "github.com/chaos-mesh/chaos-mesh/pkg/chaosdaemon/pb" + ) + + func (s *daemonServer) ExecHelloWorldChaos(ctx context.Context, req *pb.ExecHelloWorldRequest) (*empty.Empty, error) { + log.Info("ExecHelloWorldChaos", "request", req) + + pid, err := s.crClient.GetPidFromContainerID(ctx, req.ContainerId) + if err != nil { + return nil, err + } + + cmd := defaultProcessBuilder("sh", "-c", fmt.Sprintf("echo 'hello' `hostname`")). + SetUtsNS(GetNsPath(pid, utsNS)). + Build(context.Background()) + out, err := cmd.Output() + if err != nil { + return nil, err + } + if len(out) != 0 { + log.Info("cmd output", "output", string(out)) + } + + return &empty.Empty{}, nil + } + ``` + + After `chaos-daemon` receives the `ExecHelloWorldChaos` request, `chaos-daemon` will print `hello` to this container's hostname. + +3. Send gRPC requests in reconcile. + + When a CRD object is updated (for example: create or delete), we need to compare the state specified in the object against the actual state, and then perform operations to make the actual cluster state reflect the state specified. This process is called `reconcile`. For `HelloworldChaos`, `chaos-controller-manager` needs to send chaos request to `chaos-daemon` in `reconcile`. To do this: + + 1. Under [controllers](https://github.com/chaos-mesh/chaos-mesh/tree/master/controllers), create a directory named `helloworldchaos`, which includes a file named `type.go` with the content as below: + + ```golang + package helloworldchaos + + import ( + "context" + "errors" + "fmt" + + "github.com/go-logr/logr" + "k8s.io/client-go/tools/record" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + + "github.com/chaos-mesh/chaos-mesh/api/v1alpha1" + "github.com/chaos-mesh/chaos-mesh/controllers/common" + pb "github.com/chaos-mesh/chaos-mesh/pkg/chaosdaemon/pb" + "github.com/chaos-mesh/chaos-mesh/pkg/utils" + ) + + type Reconciler struct { + client.Client + Log logr.Logger + } + + // Reconcile reconciles a HelloWorldChaos resource + func (r *Reconciler) Reconcile(req ctrl.Request, chaos *v1alpha1.HelloWorldChaos) (ctrl.Result, error) { + r.Log.Info("Reconciling helloworld chaos") + + err := r.Apply(context.Background(), req, chaos) + return ctrl.Result{}, err + } + + // Apply applies helloworld chaos + func (r *Reconciler) Apply(ctx context.Context, req ctrl.Request, chaos v1alpha1.InnerObject) error { + r.Log.Info("Apply helloworld chaos") + helloworldchaos, ok := chaos.(*v1alpha1.HelloWorldChaos) + if !ok { + return errors.New("chaos is not helloworldchao s") + } + + pods, err := utils.SelectPods(ctx, r.Client, helloworldchaos.Spec.GetSelector()) + if err != nil { + return err + } + + for _, pod := range pods { + daemonClient, err := utils.NewChaosDaemonClient(ctx, r.Client, + &pod, common.ControllerCfg.ChaosDaemonPort) + if err != nil { + r.Log.Error(err, "get chaos daemon client") + return err + } + defer daemonClient.Close() + if len(pod.Status.ContainerStatuses) == 0 { + return fmt.Errorf("%s %s can't get the state of container", pod.Namespace, pod.Name) + } + + containerID := pod.Status.ContainerStatuses[0].ContainerID + + _, err = daemonClient.ExecHelloWorldChaos(ctx, &pb.ExecHelloWorldRequest{ + ContainerId: containerID, + }) + if err != nil { + return err + } + } + + return nil + } + + // Recover means the reconciler recovers the chaos action + func (r *Reconciler) Recover(ctx context.Context, req ctrl.Request, chaos v1alpha1.InnerObject) error { + return nil + } + + // Object would return the instance of chaos + func (r *Reconciler) Object() v1alpha1.InnerObject { + return &v1alpha1.HelloWorldChaos{} + } + ``` + + > **Notes:** + > + > In our case here, the `Recover` function does nothing because `HelloWorldChaos` only prints some log and doesn't change anything. You may need to implement the `Recover` function in your development. + + 2. Modify the `controllers/helloworldchaos_controller.go` file created in [Dev a new chaos](dev_hello_world.md), the `Reconcile` function updated to below: + + ```golang + // +kubebuilder:rbac:groups=chaos-mesh.org,resources=helloworldchaos,verbs=get;list;watch;create;update;patch;delete + // +kubebuilder:rbac:groups=chaos-mesh.org,resources=helloworldchaos/status,verbs=get;update;patch + + func (r *HelloWorldChaosReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) { + logger := r.Log.WithValues("reconciler", "helloworldchaos") + logger.Info("Hello World!") + + reconciler := helloworldchaos.Reconciler{ + Client: r.Client, + Log: logger, + } + chaos := &v1alpha1.HelloWorldChaos{} + if err := r.Get(context.Background(), req.NamespacedName, chaos); err != nil { + return ctrl.Result{}, nil + } + + result, err := reconciler.Reconcile(req, chaos) + if err != nil { + return ctrl.Result{}, nil + } + + return result, nil + } + ``` + +## Verify your chaos + +Now you are all set. It's time to verify the chaos type you just created. Take the steps below: + +1. Make the Docker image. Refer to [Make the Docker image](dev_hello_world.md#make-the-docker-image). + +2. Upgrade Chaos Mesh. Since we have already installed Chaos Mesh in [Develop a New Chaos](dev_hello_world.md#run-chaos), we only need to upgrade it: + + ```bash + helm upgrade chaos-mesh helm/chaos-mesh --namespace=chaos-testing --set controllerManager. imagePullPolicy=Always --set chaosDaemon.imagePullPolicy=Always + ``` + +3. Deploy the Pods for test: + + ```bash + kubectl apply -f https://raw.githubusercontent.com/chaos-mesh/apps/master/ping/busybox-statefulset.yaml + ``` + + This command deploys two Pods in the `busybox` namespace. + +4. Create the chaos YAML file: + + ```yaml + apiVersion: chaos-mesh.org/v1alpha1 + kind: HelloWorldChaos + metadata: + name: busybox-helloworld-chaos + spec: + selector: + namespaces: + - busybox + ``` + +5. Apply the chaos: + + ```bash + kubectl apply -f /path/to/helloworld.yaml + ``` +6. Verify your chaos. There are different logs to check to see whether your chaos works as expected: + + + - Check the log of `chaos-controller-manager`: + + ```bash + kubectl logs chaos-controller-manager-{pod-post-fix} -n chaos-testing + ``` + + The log is as follows: + + ```log + 2020-09-09T09:13:36.018Z INFO controllers.HelloWorldChaos Reconciling helloworld chaos {"reconciler": "helloworldchaos"} + 2020-09-09T09:13:36.018Z INFO controllers.HelloWorldChaos Apply helloworld chaos {"reconciler": "helloworldchaos"} + ``` + + - Check the log of `chaos-daemon`: + + ```bash + kubectl logs chaos-daemon-{pod-post-fix} -n chaos-testing + ``` + + The log is as follows: + + ```log + 2020-09-09T09:13:36.036Z INFO chaos-daemon-server exec hello world chaos {"request": "container_id:\"docker://8f2918ee05ed587f7074a923cede3bbe5886277faca95d989e513f7b7e831da5\" "} + 2020-09-09T09:13:36.044Z INFO chaos-daemon-server build command {"command": "nsenter -u/proc/45664/ns/uts -- sh -c echo 'hello' `hostname`"} + 2020-09-09T09:13:36.058Z INFO chaos-daemon-server cmd output {"output": "hello busybox-1\n"} + 2020-09-09T09:13:36.064Z INFO chaos-daemon-server exec hello world chaos {"request": "container_id:\"docker://53e982ba5593fa87648edba665ba0f7da3f58df67f8b70a1354ca00447c00524\" "} + 2020-09-09T09:13:36.066Z INFO chaos-daemon-server build command {"command": "nsenter -u/proc/45620/ns/uts -- sh -c echo 'hello' `hostname`"} + 2020-09-09T09:13:36.070Z INFO chaos-daemon-server cmd output {"output": "hello busybox-0\n"} + ``` + + We can see the `chaos-daemon` prints `hello` to these two Pods. diff --git a/docs/development_guides/dev_hello_world.md b/docs/development_guides/dev_hello_world.md index b271768880933..bb188d219db8e 100644 --- a/docs/development_guides/dev_hello_world.md +++ b/docs/development_guides/dev_hello_world.md @@ -136,24 +136,25 @@ We should also import `github.com/chaos-mesh/chaos-mesh/controllers/helloworldch ## Make the Docker image -Having the object successfully added, you can make a Docker image and push it to your registry: +Having the object successfully added, you can make a Docker image: ```bash make -make docker-push ``` -> **Note:** -> -> The default `DOCKER_REGISTRY` is `localhost:5000`, which is preset in `hack/kind-cluster-build.sh`. You can overwrite it to any registry to which you have access permission. +Then push it to your registry: + +```bash +make docker-push +``` ## Run chaos You are almost there. In this step, you will pull the image and apply it for testing. -Before you pull any image for Chaos Mesh (using `helm install` or `helm upgrade`), modify [values.yaml](https://github.com/chaos-mesh/chaos-mesh/blob/master/helm/chaos-mesh/values.yaml) of helm template to replace the default image with what you just pushed to your local registry. +Before you pull any image for Chaos Mesh (using `helm install` or `helm upgrade`), modify [values.yaml](https://github.com/chaos-mesh/chaos-mesh/blob/master/helm/chaos-mesh/values.yaml) of the helm template to replace the default image with what you just pushed to your local registry. -In this case, the template uses `pingcap/chaos-mesh:latest` as the default target registry, so you need to replace it with `localhost:5000`, as shown below: +In this case, the template uses `pingcap/chaos-mesh:latest` as the default target registry, so you need to replace it with the environment variable `DOCKER_REGISTRY`'s value(default `localhost:5000`), as shown below: ```yaml clusterScoped: true @@ -177,53 +178,94 @@ dashboard: Now take the following steps to run chaos: -1. Get the related custom resource type for Chaos Mesh: +1. Create namespace `chaos-testing` + + ```bash + kubectl create namespace chaos-testing + ``` + +2. Get the related custom resource type for Chaos Mesh: + + ```bash + kubectl apply -f manifests/ + ``` + + You can see CRD `helloworldchaos` is created: + + ```log + customresourcedefinition.apiextensions.k8s.io/helloworldchaos.chaos-mesh.org created + ``` + + Now you can get the CRD using the command below: + + ```bash + kubectl get crd helloworldchaos.chaos-mesh.org + ``` + +3. Install Chaos Mesh: + + - For helm 3.X + + ```bash + helm install chaos-mesh helm/chaos-mesh --namespace=chaos-testing --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock + ``` + + - For helm 2.X + + ```bash + helm install helm/chaos-mesh --name=chaos-mesh --namespace=chaos-testing --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock + ``` + + To verify your installation, get pods from the `chaos-testing` namespace: + + ```bash + kubectl get pods --namespace chaos-testing -l app.kubernetes.io/instance=chaos-mesh + ``` + + > **Note:** + > + > Arguments `--set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock` are used to to support network chaos on kind. - ```bash - kubectl apply -f manifests/ - kubectl get crd podchaos.chaos-mesh.org - ``` +4. Create `chaos.yaml` in any location with the lines below: -2. Install Chaos Mesh: + ```yaml + apiVersion: chaos-mesh.org/v1alpha1 + kind: HelloWorldChaos + metadata: + name: hello-world + namespace: chaos-testing + spec: {} + ``` - ```bash - # for helm v2 - helm install helm/chaos-mesh --name=chaos-mesh --namespace=chaos-testing --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock - # for helm v3 - helm install chaos-mesh helm/chaos-mesh --namespace=chaos-testing --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock - kubectl get pods --namespace chaos-testing -l app.kubernetes.io/instance=chaos-mesh - ``` +5. Apply the chaos: - The arguments `--set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock` is used to to support network chaos on kind. + ```bash + kubectl apply -f /path/to/chaos.yaml + ``` -3. Create `chaos.yaml` in any location with the lines below: + ```bash + kubectl get HelloWorldChaos -n chaos-testing + ``` - ```yaml - apiVersion: chaos-mesh.org/v1alpha1 - kind: HelloWorldChaos - metadata: - name: hello-world - namespace: chaos-testing - spec: {} - ``` + Now you should be able to check the `Hello World!` result in the log: -4. Apply the chaos: + Now you can check the log of `chaos-controller-manager`: - ```bash - kubectl apply -f /path/to/chaos.yaml - kubectl get HelloWorldChaos -n chaos-testing - ``` + ```bash + kubectl logs chaos-controller-manager-{pod-post-fix} -n chaos-testing + ``` - Now you should be able to check the `Hello World!` result in the log: + The log is as follows: - ```bash - kubectl logs chaos-controller-manager-{pod-post-fix} -n chaos-testing - ``` + ```log + 2020-09-07T09:21:29.301Z INFO controllers.HelloWorldChaos Hello World! {"reconciler": "helloworldchaos"} + 2020-09-07T09:21:29.308Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "helloworldchaos", "request": "chaos-testing/hello-world"} + ``` - > **Note:** - > - > `{pod-post-fix}` is a random string generated by Kubernetes, you can check it by executing `kubectl get po -n chaos-testing`. + > **Note:** + > + > `{pod-post-fix}` is a random string generated by Kubernetes. You can check it by executing `kubectl get pod -n chaos-testing`. ## Next steps -Congratulations! You have just added a chaos type for Chaos Mesh successfully. Let us know if you run into any issues during the process. If you feel like doing other types of contributions, refer to Add facilities to chaos daemon (WIP). +Congratulations! You have just added a chaos type for Chaos Mesh successfully. Let us know if you run into any issues during the process. If you feel like doing other types of contributions, refer to [Add facilities to chaos daemon](add_chaos_daemon.md). diff --git a/docs/development_guides/develop.md b/docs/development_guides/develop.md index 06c9be73e13ca..b92c2f593a0f1 100644 --- a/docs/development_guides/develop.md +++ b/docs/development_guides/develop.md @@ -11,4 +11,4 @@ This guide prepares you for the development of Chaos Mesh from scratch. Before y The development flow starts from [Set up your development environment](setup_env.md). After this, you can choose any of the following procedures to contribute: - [Develop a New Chaos Type](dev_hello_world.md) -- Add facilities to chaos daemon +- [Add facilities to chaos daemon](add_chaos_daemon.md) diff --git a/docs/development_guides/setup_env.md b/docs/development_guides/setup_env.md index aaf583be73694..edb0f1f991e4e 100644 --- a/docs/development_guides/setup_env.md +++ b/docs/development_guides/setup_env.md @@ -34,7 +34,14 @@ Make sure you have the above prerequisites met. Now follow the steps below to pr 3. Make sure [Docker](https://docs.docker.com/install/) is installed and running on your local machine. -4. Make sure `${GOPATH}/bin` is in your `PATH`. +4. Make sure [Docker Regitry](https://docs.docker.com/registry/) is running. Set the environment variable `DOCKER_REGISTRY` with the registry address: + + ```bash + echo 'export DOCKER_REGISTRY=localhost:5000' >> ~/.bash_profile + source ~/.bash_profile + ``` + +5. Make sure `${GOPATH}/bin` is in your `PATH`. ```bash echo 'export PATH=$(go env GOPATH)/bin:${PATH}' >> ~/.bash_profile @@ -64,7 +71,7 @@ With the toolchain ready, you still need a local Kubernetes cluster as the deplo hack/kind-cluster-build.sh ``` -The above script will create a Kubernetes cluster by kind. When you don't need this cluster, you can run the following command to delete it: +The above script creates a Kubernetes cluster via Kind. When you are done with it, run the following command to delete it: ```bash kind delete cluster --name=kind diff --git a/versioned_docs/version-0.9.1/user_guides/io_chaos.md b/versioned_docs/version-0.9.1/user_guides/io_chaos.md index c4b7a54fa3ff9..29986432dde4f 100644 --- a/versioned_docs/version-0.9.1/user_guides/io_chaos.md +++ b/versioned_docs/version-0.9.1/user_guides/io_chaos.md @@ -153,7 +153,7 @@ Then, you can start your application and define YAML file to start your chaos ex > **Note:** > -> The value of the annotation in the above examples, `chaos-tikv` is the name filed in your injection config. +> The value of the annotation in the above examples, `chaos-tikv` is the name field in your injection config. ### Start a chaos experiment