diff --git a/docs/configuration.md b/docs/configuration.md index 5d5d8ae5c..941bc62cb 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -21,7 +21,7 @@ Once the configs are loaded (in either case), Sonobuoy parses them and gathers d | | Overview|Path on Cluster Node|[STANDALONE]
JSON example(s)|[CONTAINERIZED]
YAML manifest example(s) |---|---|---|---|---| |*Data configuration*| What Sonobuoy records, how, and where. |*ANY of the following*:
(1) `config.json` in the directory where `sonobuoy` is executed
(2) `/etc/sonobuoy/config.json`
(3) `$SONOBUOY_CONFIG`

|[`config.json`][10]|
[`examples/ksonnet/components/10-configmaps.jsonnet`][11]

*The jsonnet file is basically a wrapper for the `config.json` file, which allows it to be properly mounted onto the cluster's Sonobuoy pod.*

-|*Plugin configuration*|Settings for each plugin integration.|*ANY of the following*:
(1) `/etc/sonobuoy/plugins.d`
(2) `$HOME/.sonobuoy/plugins.d`
(3) `./plugins.d`
(4) `PluginSearchPath` (override from the data configuration)

| There is a YAML config for each plugin:
(1) [`plugins.d/e2e.yaml`][16]
(2)[`plugins.d/systemdlogs.yaml`][17]|
[`examples/ksonnet/components/10-configmaps.jsonnet`][11]

*Same comment about the jsonnet file as above.* +|*Plugin configuration*|Settings for each plugin integration.|*ANY of the following*:
(1) `/etc/sonobuoy/plugins.d`
(2) `$HOME/.sonobuoy/plugins.d`
(3) `./plugins.d`
(4) `PluginSearchPath` (override from the data configuration)

| There is a templatized YAML config for each plugin:
(1) [`plugins.d/e2e.tmpl`][16]
(2) [`plugins.d/systemd_logs.tmpl`][17]
(3) [`plugins.d/heptio-e2e.tmpl`][20]|
[`examples/ksonnet/components/10-configmaps.jsonnet`][11]

*Same comment about the jsonnet file as above.* |*Kubernetes component definitions*|The various K8s objects that need to be defined for Sonobuoy to run as a containerized app.|N/A (manifest only)|N/A|The example splits this into two manifests:
(1) [`examples/ksonnet/components/00-rbac.jsonnet`][12]
(2) [`examples/ksonnet/components/20-pod.jsonnet`][13]| @@ -188,7 +188,8 @@ However, ensure that your pod declaration has addressed the following aspects, w [13]: /examples/ksonnet/components/20-pod.jsonnet [14]: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors [15]: https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/ -[16]: /plugins.d/e2e.yaml -[17]: plugins.d/systemdlogs.yaml +[16]: /plugins.d/e2e.tmpl +[17]: /plugins.d/systemd_logs.tmpl [18]: /examples/quickstart.yaml [19]: /examples/ksonnet/components +[20]: /plugins.d/heptio-e2e.tmpl diff --git a/docs/plugins.md b/docs/plugins.md index beb4bea49..53ae819db 100644 --- a/docs/plugins.md +++ b/docs/plugins.md @@ -19,7 +19,7 @@ There are two main components that specify plugin behavior: *These configs are defined by the **end user**.* -1. **Plugin Definition**: A structured YAML definition that describes a plugin's features, method of launch, and other configurations. +1. **Plugin Definition**: Templatized YAML that defines a Kubernetes API resource like a Pod. Plugin metadata--such as a plugin's features, method of launch, and other configurations--are specified with annotations. *This YAML is defined by the plugin **developer**, and can be taken as a given by the end user.* @@ -27,14 +27,23 @@ The remainder of this document focuses on **Plugin Definition**. ## Plugin Definition -The *plugin definition* is a YAML file (raw or wrapped in a ConfigMap) that describes the core parts of your custom plugin. It should contain the following fields: +The *plugin definition* is a YAML file (raw or wrapped in a ConfigMap) that describes the core parts of your custom plugin. This YAML defines a Kubernetes API resource like a Pod. -| Field | Description | Example Values | +Some of the values in this YAML file are template variables that will be filled in by Sonobuoy. The template will have access to several variables: + +| Variable | Source | Description | | --- | --- | --- | -| `name` | A name that is used to identify the plugin (e.g. in the Plugin Selection described above). | "e2e" | -| `driver` | Sonobuoy implements *plugin drivers* that define different modes of operation.

(1) **"Job" driver**: The plugin will run on a single node (e.g. master).
(2) **"DaemonSet" driver**: The plugin runs on each cluster node.

You can find the implementations [here][7]. | "Job|DaemonSet" | -| `resultType` | The name of the subdirectory that this plugin's results are saved in. With a `resultType` of "e2e", results are written into `plugins/e2e/...` (within the tarball output).

This value is typically the same as the plugin `name`. | "e2e" | -| `spec` | The Pod specification (e.g. network settings, container settings, volume definitions, etc.) | See [the parameter spec][4] below for reference. | +| `{{.SessionID}}` | Autogenerated by Sonobuoy | A generated token used to be able to uniquely identify all pieces of a single sonobuoy run. | +| `{{.Namespace}}` | Populated from the PluginNamespace value in config.json | The Kubernetes namespace the plugins will run in. This value is populated by the `PluginNamespace` value in `config.json`. | +| `{{.MasterAddress}}` | Autogenerated by Sonobuoy | The address the sonobuoy-workers use to communicate with the sonobuoy master process. It is dynamically generated. | + +The plugin system relies on some annotations being present in a plugin definition: + +| Annotation | Description | Example Values | +| --- | --- | --- | +| `sonobuoy-driver` | Sonobuoy implements *plugin drivers* that define different modes of operation.

(1) **"Job" driver**: The plugin runs on a single node (e.g. master).
(2) **"DaemonSet" driver**: The plugin runs on each cluster node.

You can find the implementations [here][7]. | "Job|DaemonSet" | +| `sonobuoy-plugin` | A name that is used to identify the plugin (e.g. in the [Plugin Selection][18]). | "e2e", "systemd_logs" | +| `sonobuoy-result-type` | The name of the subdirectory that this plugin's results are saved in. With a `resultType` of "e2e", results are written into `plugins/e2e/...` (within the tarball output).

This value is typically the same as the `sonobuoy-plugin` annotation. | "e2e" | Sonobuoy searches for these definitions in three locations by default: @@ -76,38 +85,38 @@ To allow your two containers to interact (i.e. for the consumer to read and publ ### Example -Below is the Pod `spec` component from the [`e2e` plugin definition][9]. It follows the two-container-paradigm. One container runs the e2e tests and writes to a temporary folder, and the other container (a Sonobuoy worker) reads those results and sends them back to the master Sonobuoy instance. +Below is the Pod template from the [`e2e` plugin definition][9]. It follows the two-container-paradigm. One container runs the e2e tests and writes to a temporary folder, and the other container (a Sonobuoy worker) reads those results and sends them back to the master Sonobuoy instance. See [Parameter Reference][4] for descriptions of each of the important config keys. ``` +--- +apiVersion: v1 +kind: Pod +metadata: + annotations: + sonobuoy-driver: Job + sonobuoy-plugin: e2e + sonobuoy-result-type: e2e + labels: + component: sonobuoy + sonobuoy-run: '{{.SessionID}}' + tier: analysis + name: sonobuoy-e2e-job-{{.SessionID}} + namespace: '{{.Namespace}}' spec: - serviceAccountName: sonobuoy-serviceaccount - tolerations: - - key: node-role.kubernetes.io/master - operator: Exists - effect: NoSchedule - - key: CriticalAddonsOnly - operator: Exists - restartPolicy: Never containers: - # (1) This container runs the actual end-to-end tests. - - name: e2e + - env: + - name: E2E_FOCUS + value: Pods should be submitted and removed image: gcr.io/heptio-images/kube-conformance:latest imagePullPolicy: Always - # NOTE: Full conformance can take a while depending on your cluster size. - # As a result, only a single test is set atm to verify correctness. - # Operators that want the complete test results can comment out the - # env section. - env: - - name: E2E_FOCUS - value: "Pods should be submitted and removed" + name: e2e volumeMounts: - - name: results - mountPath: /tmp/results - # (2) This container is the Sonobuoy worker that handles uploads. - - name: sonobuoy-worker - command: + - mountPath: /tmp/results + name: results + readOnly: false + - command: - sh - -c - /sonobuoy worker global -v 5 --logtostderr @@ -115,24 +124,30 @@ spec: - name: NODE_NAME valueFrom: fieldRef: - apiVersion: v1 fieldPath: spec.nodeName - name: RESULTS_DIR value: /tmp/results - image: gcr.io/heptio-images/sonobuoy:latest + - name: MASTER_URL + value: '{{.MasterAddress}}' + - name: RESULT_TYPE + value: e2e + image: gcr.io/heptio-images/sonobuoy:master imagePullPolicy: Always + name: sonobuoy-worker volumeMounts: - - name: config - mountPath: /etc/sonobuoy - - name: results - mountPath: /tmp/results + - mountPath: /tmp/results + name: results + readOnly: false + restartPolicy: Never + serviceAccountName: sonobuoy-serviceaccount + tolerations: + - effect: NoSchedule + key: node-role.kubernetes.io/master + - key: CriticalAddonsOnly + operator: Exists volumes: - - name: results - emptyDir: {} - - name: config - configMap: - # This will be rewritten when the JobPlugin driver goes to launch the pod. - name: __SONOBUOY_CONFIGMAP__ + - emptyDir: {} + name: results ``` ### Parameter Reference @@ -143,7 +158,7 @@ spec: | `restartPolicy` | Specifies whether your plugin should retry on failure. | `container.image`, `container.imagePullPolicy` | What and how often a new image should be pulled. | | `container.env` | Set environmental variables here. These variables can be used to configure plugin behavior.

For DaemonSet plugins (e.g. `systemdlogs`), the `sonobuoy worker` consumer container needs a `NODE_NAME` variable to know which node the results should be uploaded for.| -| `container.volumeMounts`, `volumes` |
It is important to set up volumes and mount them properly, so that the container(s) can:

(1) **Get necessary configs** from locations like `/etc/sonobuoy`. While the Sonobuoy master creates the ConfigMap, the inbuilt Sonobuoy drivers actually substitute it into the `__SONOBUOY_CONFIGMAP__` pseudo-template. The ConfigMap's name isn't predictable because it only lasts for one run.

(2) **Write results locally**, such that the Sonobuoy worker can find the files it needs to upload to the Sonobuoy master. Typically the same `emptyDir` `results` directory is shared by both the "plugin" container and the Sonobuoy worker container.

| +| `container.volumeMounts`, `volumes` | **Write results locally**, such that the Sonobuoy worker can find the files it needs to upload to the Sonobuoy master. Typically the same `emptyDir` `results` directory is shared by both the "plugin" container and the Sonobuoy worker container.

| ## Available Plugins @@ -165,7 +180,7 @@ See the [`/build`][14] directory for the source code used to build these plugins [6]: img/sonobuoy-plugins.png [7]: /pkg/plugin/driver [8]: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ -[9]: /plugins.d/e2e.yaml +[9]: /plugins.d/e2e.tmpl [10]: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature [11]: /plugins.d/systemdlogs.yaml [12]: /examples/quickstart.yaml @@ -174,3 +189,4 @@ See the [`/build`][14] directory for the source code used to build these plugins [15]: conformance-testing.md#integration-with-sonobuoy [16]: https://github.com/heptio/sonobuoy-plugin-systemd-logs [17]: https://github.com/heptio/kube-conformance +[18]: /docs/configuration.md#plugin-configuration