Documentation for Custom Test in Skaffold

docs/content/en/docs/design/config.md

@@ -18,7 +18,7 @@ read the configuration file from the current directory.
 | `kind`  |  The Skaffold configuration file has the kind `Config`.  |
 | `metadata`  |  Holds additional properties like the `name` of this configuration.  |
 | `build`  |  Specifies how Skaffold builds artifacts. You have control over what tool Skaffold can use, how Skaffold tags artifacts and how Skaffold pushes artifacts. Skaffold supports using local Docker daemon, Google Cloud Build, Kaniko, or Bazel to build artifacts. See [Builders](/docs/pipeline-stages/builders) and [Taggers]({{< relref "/docs/pipeline-stages/taggers" >}}) for more information. |
-| `test` |  Specifies how Skaffold tests artifacts. Skaffold supports [container-structure-tests](https://github.com/GoogleContainerTools/container-structure-test) to test built artifacts. See [Testers]({{< relref "/docs/pipeline-stages/testers" >}}) for more information. |
+| `test` |  Specifies how Skaffold tests artifacts. Skaffold supports [container-structure-tests](https://github.com/GoogleContainerTools/container-structure-test) to test built artifacts and custom tests to run custom commands as part of the development pipeline. See [Testers]({{< relref "/docs/pipeline-stages/testers" >}}) for more information. |
 | `deploy` |  Specifies how Skaffold deploys artifacts. Skaffold supports using `kubectl`, `helm`, or `kustomize` to deploy artifacts. See [Deployers]({{< relref "/docs/pipeline-stages/deployers" >}}) for more information. |
 | `profiles`|  Profile is a set of settings that, when activated, overrides the current configuration. You can use Profile to override the `build`, `test` and `deploy` sections. |
 | `requires`|  Specifies a list of other skaffold configurations to import into the current config |

docs/content/en/docs/pipeline-stages/_index.md

@@ -26,7 +26,7 @@ will not push artifacts to a remote repository.
 | [Init]({{< relref "/docs/pipeline-stages/init" >}}) | generate a starting point for Skaffold configuration | 
 | [Build]({{< relref "/docs/pipeline-stages/builders" >}}) | build images with different builders | 
 | [Tag]({{< relref "/docs/pipeline-stages/taggers" >}}) | tag images based on different policies |
-| [Test]({{< relref "/docs/pipeline-stages/testers" >}}) |  test images with structure tests |
+| [Test]({{< relref "/docs/pipeline-stages/testers" >}}) | run tests with testers |
 | [Deploy]({{< relref "/docs/pipeline-stages/deployers" >}}) |  deploy with kubectl, kustomize or helm |
 | [File Sync]({{< relref "/docs/pipeline-stages/filesync" >}}) |  sync changed files directly to containers |
 | [Log Tailing]({{< relref "/docs/pipeline-stages/log-tailing" >}}) |  tail logs from workloads |

docs/content/en/docs/pipeline-stages/testers/_index.md

@@ -0,0 +1,15 @@
+---
+title: "Test"
+linkTitle: "Test"
+weight: 10
+featureId: test
+aliases: [/docs/how-tos/testers]
+no_list: true
+---
+
+Skaffold has an integrated testing phase between the build and deploy phases of the pipeline. Skafforl supports the below types of tests.
+
+| Skaffold testers|Description| 
+|----------|-------|
+| [Custom Test]({{< relref "/docs/pipeline-stages/testers/custom.md" >}}) | Enables users to run custom commands in the testing phase of the Skaffold pipeline | 
+| [Container Structure Test]({{< relref "/docs/pipeline-stages/testers/structure.md" >}}) | Enables users to validate built container images before deploying them to our cluster | 

docs/content/en/docs/pipeline-stages/testers/custom.md

@@ -0,0 +1,109 @@
+---
+title: "Custom Test"
+linkTitle: "Custom Test"
+weight: 20
+featureId: test.custom
+---
+
+
+Custom Test would allow developers to run custom commands as part of the development pipeline. The command will be executed in the testing phase of the Skaffold pipeline. It will run on the local machine where Skaffold is being executed and works with all supported Skaffold platforms. Users can opt out of running custom tests by using the `-skipTests` flag.
+
+Custom Test enables the users to:
+- Run validation tests on their code (e.g., unit tests)
+- Run validation and security tests on the image before deploying the image to a cluster (e.g., Developers can shell out GCP Container Analysis or Anchore Grype in a custom test script and use that for validation )
+
+Custom tests are defined per image in the Skaffold config. Every time an artifact is rebuilt, Skaffold runs the associated custom tests as part of the Skaffold dev loop.
+Multiple custom testers can be defined per test. The Skaffold pipeline will be blocked on the custom test to complete or fail. Skaffold will exit the loop when the first test fails. For ongoing test failures, Skaffold will stop the loop (not continue with the deploy) but will not exit the loop. Skaffold would surface the errors to the user and will keep the dev loop running. Skaffold will continue watching user specified test dependencies and re-trigger the loop whenever it detects another change. 
+
+CustomTester has a configurable timeout option to wait for the command to return. If no timeout is specified, Skaffold will wait until the test command has completed execution. 
+
+### Contract between Skaffold and Custom command
+
+Skaffold will pass in the environment variable `$IMAGE` to the custom command to access the image.
+
+This variable can be set as a flag value input to the custom command `--flag=$IMAGE`.
+
+
+### Configuration
+
+To use a custom command, add a `custom` field to the corresponding test in the `test` section of the `skaffold.yaml`.
+Supported schema for `CustomTest` includes:
+
+{{< schema root="CustomTest" >}}
+
+
+
+### Dependencies for a Custom Test
+
+Users can specify `dependencies` for custom tests so that skaffold knows when to retest during a dev loop. Dependencies can be specified per command. Users could list out directories and/or files to watch per command. If no dependencies are specified, only the script file (if the command is a script file) will be watched as a dependency. Test dependencies cannot trigger rebuild of an image.
+
+Supported schema for `dependencies` includes:
+
+{{< schema root="CustomTestDependencies" >}}
+
+
+#### Paths and Ignore
+
+`Paths` and `Ignore` are arrays used to list dependencies. This can be a glob.
+Any paths in `Ignore` will be ignored by the skaffold file watcher, even if they are also specified in `Paths`.
+`Ignore` will only work in conjunction with `Paths`.
+
+```yaml
+    custom:
+      - command: ./test.sh
+        timeoutSeconds: 60
+        dependencies:
+          paths:
+          -  "*_test.go"
+          -  "test.sh"
+```
+
+#### Command for dependencies
+
+Sometimes users might have a command or a script that can provide the dependencies for a given test. Custom tester can ask Skaffold to execute a custom command, which Skaffold can use to get the dependencies for the test for file watching.
+
+The command *must* return dependencies as a JSON array, otherwise skaffold will error out.
+
+```yaml
+    custom:
+      - command: echo Hello world!!
+        dependencies:
+          command: echo [\"main_test.go\"] 
+```
+
+
+>*Note: Adding a file pattern to a test dependency doesn't automatically enable file sync on it.  Refer to the [`file sync`](https://skaffold.dev/docs/pipeline-stages/filesync/) documentation, on how to set that up separately.*
+
+
+### Logging
+
+`STDOUT` and `STDERR` from the custom command script will be redirected and displayed within skaffold logs.
+
+
+## Usage
+
+Custom tests will be automatically invoked as part of the run and dev commands, but can also be run independently by using the test subcommand.
+
+- To execute the custom command as an independent test command run:
+```skaffold test```
+- To execute custom command as part of the run command run:
+```skaffold run```
+- To execute custom command as part of the dev loop run:
+```skaffold dev```
+### Example
+This following example shows the `customTest` section from a `skaffold.yaml`.
+It instructs Skaffold to run unit tests (main_test.go) located in the local folder when the main application changes:
+{{% readfile file="samples/testers/custom/customTest.yaml" %}}
+A sample `test.sh` file, which runs unit tests when the test changes.
+```
+#!/bin/bash
+
+set -e
+
+echo "go custom test $@"
+
+go test .
+```
+
+
+

docs/content/en/docs/pipeline-stages/testers.md → docs/content/en/docs/pipeline-stages/testers/structure.md

@@ -1,9 +1,8 @@
 ---
-title: "Test"
-linkTitle: "Test"
+title: "Container Structure Test"
+linkTitle: "Container Structure Test"
 weight: 20
-featureId: test
-aliases: [/docs/how-tos/testers]
+featureId: test.structure
 ---
 
 It's common practice to validate built container images before deploying them to our cluster.
@@ -17,21 +16,22 @@ Structure tests are defined per image in the Skaffold config.
 Every time an artifact is rebuilt, Skaffold runs the associated structure tests on that image.
 If the tests fail, Skaffold will not continue on to the deploy stage.
 If frequent tests are prohibitive, long-running tests should be moved to a dedicated Skaffold profile.
+Users can opt out of running container structure tests by using the `-skipTests` flag.
 
 ### Example
 This following example shows the `test` section from a `skaffold.yaml`.
 It instructs Skaffold to run all container structure tests in the `structure-test` folder relative to the Skaffold root directory:
 
-{{% readfile file="samples/testers/test.yaml" %}}
+{{% readfile file="samples/testers/structure/test.yaml" %}}
 
 The files matched by the `structureTests` key are `container-structure-test` test configurations, such as:
 
-{{% readfile file="samples/testers/structureTest.yaml" %}}
+{{% readfile file="samples/testers/structure/structureTest.yaml" %}}
 
 For a reference how to write container structure tests, see its [documentation](https://github.com/GoogleContainerTools/container-structure-test#command-tests).
 
 In order to restrict the executed structure tests, a `profile` section can override the file pattern:
 
-{{% readfile file="samples/testers/testProfile.yaml" %}}
+{{% readfile file="samples/testers/structure/testProfile.yaml" %}}
 
 To execute the tests once, run `skaffold build --profile quickcheck`.

docs/content/en/samples/testers/custom/customTest.yaml

@@ -0,0 +1,12 @@
+test:
+  - image: custom-test-example
+    custom:
+      - command: ./test.sh
+        timeoutSeconds: 60
+        dependencies:
+          paths:
+          -  "*_test.go"
+          -  "test.sh"
+      - command: echo Hello world!!
+        dependencies:
+          command: echo [\"main_test.go\"]

docs/data/maturity.json

@@ -310,11 +310,31 @@
   "test": {
     "dev": "x",
     "run": "x",
+    "test": "x",
     "area": "Test",
     "maturity": "GA",
     "description": "Run tests as part of your pipeline",
     "url": "/docs/pipeline-stages/testers"
   },
+  "test.structure": {
+    "dev": "x",
+    "run": "x",
+    "test": "x",
+    "area": "Test",
+    "maturity": "GA",
+    "description": "Run structure tests as part of your pipeline",
+    "url": "/docs/pipeline-stages/testers/structure"
+  },
+  "test.custom": {
+    "dev": "x",
+    "run": "x",
+    "test": "x",
+    "area": "Test",
+    "feature": "custom tester",
+    "maturity": "alpha",
+    "description": "Run custom test command locally",
+    "url": "/docs/pipeline-stages/testers/custom"
+  },
   "triggers": {
     "dev": "x",
     "debug": "x",

docs/layouts/partials/info-panel.html

@@ -1,48 +1,51 @@
 {{ if .Params.featureId }}
-    {{ $data := (index .Site.Data.maturity .Params.featureId)}}
-    {{ $maturity := $data.maturity }}
-    {{ $dev := $data.dev }}
-    {{ $debug := $data.debug }}
-    {{ $run := $data.run }}
-    {{ $build := $data.build }}
-    {{ $deploy := $data.deploy }}
-    {{ $render := $data.render }}
-    {{ $all := and $dev $debug $run $build $deploy $render }}
+{{ $data := (index .Site.Data.maturity .Params.featureId)}}
+{{ $maturity := $data.maturity }}
+{{ $dev := $data.dev }}
+{{ $debug := $data.debug }}
+{{ $run := $data.run }}
+{{ $build := $data.build }}
+{{ $test := $data.test }}
+{{ $deploy := $data.deploy }}
+{{ $render := $data.render }}
+{{ $all := and $dev $debug $run $build $deploy $render }}
 
-    <div class="alert infopanel" role="contentinfo">
-        <table>
-            {{ if and $maturity (not (eq $maturity "GA")) }}
-                <tr class="maturity maturity-{{$maturity}}">
-                    <td><b>Maturity</b></td>
-                    <td><a href="/docs/references/deprecation"><code>{{$maturity}}</code></a></td>
-                </tr>
-            {{ end }}
+<div class="alert infopanel" role="contentinfo">
+    <table>
+        {{ if and $maturity (not (eq $maturity "GA")) }}
+        <tr class="maturity maturity-{{$maturity}}">
+            <td><b>Maturity</b></td>
+            <td><a href="/docs/references/deprecation"><code>{{$maturity}}</code></a></td>
+        </tr>
+        {{ end }}
 
-            <tr>
-                <td><b>Applicable</b></td>
-                <td>
-                    {{if or $dev $all}}<a href="/docs/references/cli#skaffold-dev"><code> dev</code></a>{{end}}
-                    {{if or $debug $all}}<a href="/docs/references/cli#skaffold-debug"><code> debug</code></a>{{end}}
-                    {{if or $run $all}} <a href="/docs/references/cli#skaffold-run"><code> run</code></a>{{end}}
-                    {{if or $build $all}}<a href="/docs/references/cli#skaffold-build"><code> build</code></a>{{end}}
-                    {{if or $deploy $all}}<a href="/docs/references/cli#skaffold-deploy"><code> deploy</code></a>{{end}}
-                    {{if or $render $all}}<a href="/docs/references/cli#skaffold-render"><code> render</code></a>{{end}}
-                </td>
-            </tr>
-            {{ if not $all}}
-                <tr>
-                    <td><b>Not applicable</b></td>
-                    <td>
-                        {{if not $dev}}<a href="/docs/references/cli#skaffold-dev"><code> dev</code></a>{{end}}
-                        {{if not $debug}}<a href="/docs/references/cli#skaffold-debug"><code> debug</code></a>{{end}}
-                        {{if not $run}} <a href="/docs/references/cli#skaffold-run"><code> run</code></a>{{end}}
-                        {{if not $build}}<a href="/docs/references/cli#skaffold-build"><code> build</code></a>{{end}}
-                        {{if not $deploy}}<a href="/docs/references/cli#skaffold-deploy"><code> deploy</code></a>{{end}}
-                        {{if not $render}}<a href="/docs/references/cli#skaffold-render"><code> render</code></a>{{end}}
-                    </td>
-                </tr>
-            {{end}}
-        </table>
-    </div>
+        <tr>
+            <td><b>Applicable</b></td>
+            <td>
+                {{if or $dev $all}}<a href="/docs/references/cli#skaffold-dev"><code> dev</code></a>{{end}}
+                {{if or $debug $all}}<a href="/docs/references/cli#skaffold-debug"><code> debug</code></a>{{end}}
+                {{if or $run $all}} <a href="/docs/references/cli#skaffold-run"><code> run</code></a>{{end}}
+                {{if or $build $all}}<a href="/docs/references/cli#skaffold-build"><code> build</code></a>{{end}}
+                {{if or $test $all}}<a href="/docs/references/cli#skaffold-test"><code> test</code></a>{{end}}
+                {{if or $deploy $all}}<a href="/docs/references/cli#skaffold-deploy"><code> deploy</code></a>{{end}}
+                {{if or $render $all}}<a href="/docs/references/cli#skaffold-render"><code> render</code></a>{{end}}
+            </td>
+        </tr>
+        {{ if not $all}}
+        <tr>
+            <td><b>Not applicable</b></td>
+            <td>
+                {{if not $dev}}<a href="/docs/references/cli#skaffold-dev"><code> dev</code></a>{{end}}
+                {{if not $debug}}<a href="/docs/references/cli#skaffold-debug"><code> debug</code></a>{{end}}
+                {{if not $run}} <a href="/docs/references/cli#skaffold-run"><code> run</code></a>{{end}}
+                {{if not $build}}<a href="/docs/references/cli#skaffold-build"><code> build</code></a>{{end}}
+                {{if not $test}}<a href="/docs/references/cli#skaffold-test"><code> test</code></a>{{end}}
+                {{if not $deploy}}<a href="/docs/references/cli#skaffold-deploy"><code> deploy</code></a>{{end}}
+                {{if not $render}}<a href="/docs/references/cli#skaffold-render"><code> render</code></a>{{end}}
+            </td>
+        </tr>
+        {{end}}
+    </table>
+</div>
 
 {{end}}