v0.5.0

What's Changed

• klient/decoder: add filename context to decode error by @sttts in #419
• Fix value propagation in context for test actions by @snorwin in #432
• third_party/ko: Add ko as third party tools by @heylongdacoder in #415
• execute test suite only if the setup was successful by @snorwin in #424
• chore: fix comment typo by @googs1025 in #435
• Do not cache config if not explicitly set by @phisco in #413
• [env] Do not share config between tests by @Fricounet in #396
• Mark E2E plumbing as test helpers by @negz in #421
• envconf: use crypto/rand for random name generator by @harshanarayana in #439
• add teardown when TestExecPod finished by @googs1025 in #436
• fix: flux kyverno tests & return stderr while reading kubeconfig via kwokctl by @prit342 in #438
• feat: allow accessing the test environments env config by @crandles in #451

New Contributors

• @sttts made their first contribution in #419
• @snorwin made their first contribution in #432
• @googs1025 made their first contribution in #435
• @negz made their first contribution in #421
• @prit342 made their first contribution in #438

Full Changelog: v0.4.0...v0.5.0

v0.4.0

Changes by Kind

Note: Requires minimum Go1.22.0

Feature

• Added conditions.DaemonSetReady, a wait helper for determining a DaemonSet is ready. (#344, @cartermckinnon) [SIG Testing]
• Added a feature-gate argument to the supported flags that will enable setting ReverseTestFinishExecutionOrder so that Finish action can mimic t.Cleanup (#389, @harshanarayana) [SIG Testing]
• Added a way to create namespaces with custom labels and annotations (#382, @Fricounet) [SIG Testing]
• Dependencies update / Go upgrade to 1.22 and cleanup (#408, @cpanato) [SIG Testing]

Bug or Regression

• Fix a bug which both kwok and kind providers don't work expectedly when a cluster already exists. (#406, @sanposhiho) [SIG Testing]
• Fixed an issue with MutateAnnotations when the object had no existing annotations (#322, @Smeb) [SIG Testing]
• Fixes envconfg.RandomName() returning a string beginning with -` when an empty prefix is provided (#327, @pmalek) [SIG Testing]
• Gracefully exit when setup fails (#362, @codegold79) [SIG Testing]
• Stopped running the following assess if FailNow() was called during a test (#391, @Fricounet) [SIG Testing]

Dependencies

Added

• cloud.google.com/go/compute: v1.20.1
• github.com/Azure/go-ansiterm: d185dfc
• github.com/alecthomas/kingpin/v2: v2.3.2
• github.com/alecthomas/units: b94a6e3
• github.com/antlr/antlr4/runtime/Go/antlr/v4: 8188dc5
• github.com/blang/semver/v4: v4.0.0
• github.com/cenkalti/backoff/v4: v4.2.1
• github.com/felixge/httpsnoop: v1.0.3
• github.com/fxamacker/cbor/v2: v2.6.0
• github.com/go-kit/log: v0.2.1
• github.com/go-logfmt/logfmt: v0.5.1
• github.com/go-logr/stdr: v1.2.2
• github.com/google/cel-go: v0.17.8
• github.com/google/gnostic-models: v0.6.8
• github.com/grpc-ecosystem/grpc-gateway/v2: v2.16.0
• github.com/inconshreveable/mousetrap: v1.1.0
• github.com/jpillora/backoff: v1.0.0
• github.com/julienschmidt/httprouter: v1.3.0
• github.com/matttproud/golang_protobuf_extensions/v2: v2.0.0
• github.com/moby/term: 1aeaba8
• github.com/mwitkow/go-conntrack: 2f06839
• github.com/spf13/cobra: v1.7.0
• github.com/x448/float16: v0.8.4
• github.com/xhit/go-str2duration/v2: v2.1.0
• go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp: v0.44.0
• go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc: v1.19.0
• go.opentelemetry.io/otel/exporters/otlp/otlptrace: v1.19.0
• go.opentelemetry.io/otel/metric: v1.19.0
• go.opentelemetry.io/otel/sdk: v1.19.0
• go.opentelemetry.io/otel/trace: v1.19.0
• go.opentelemetry.io/otel: v1.19.0
• google.golang.org/genproto/googleapis/api: 23370e0
• google.golang.org/genproto/googleapis/rpc: b8732ec
• k8s.io/apiserver: v0.30.0
• k8s.io/gengo/v2: 51d4e06
• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.29.0

Changed

• cloud.google.com/go/compute/metadata: v0.2.0 → v0.2.3
• github.com/emicklei/go-restful/v3: v3.10.2 → v3.11.0
• github.com/evanphx/json-patch/v5: v5.6.0 → v5.9.0
• github.com/fsnotify/fsnotify: v1.6.0 → v1.7.0
• github.com/go-logr/logr: v1.2.4 → v1.4.1
• github.com/go-logr/zapr: v1.2.4 → v1.3.0
• github.com/golang/protobuf: v1.5.3 → v1.5.4
• github.com/google/go-cmp: v0.5.9 → v0.6.0
• github.com/gorilla/websocket: v1.4.2 → v1.5.0
• github.com/onsi/ginkgo/v2: v2.9.5 → v2.17.1
• github.com/onsi/gomega: v1.27.7 → v1.32.0
• github.com/prometheus/client_golang: v1.15.1 → v1.18.0
• github.com/prometheus/client_model: v0.4.0 → v0.5.0
• github.com/prometheus/common: v0.42.0 → v0.45.0
• github.com/prometheus/procfs: v0.9.0 → v0.12.0
• github.com/rogpeppe/go-internal: v1.11.0 → v1.10.0
• github.com/stretchr/testify: v1.8.1 → v1.8.4
• go.opentelemetry.io/proto/otlp: v0.7.0 → v1.0.0
• go.uber.org/goleak: v1.2.1 → v1.3.0
• go.uber.org/multierr: v1.6.0 → v1.11.0
• go.uber.org/zap: v1.24.0 → v1.26.0
• golang.org/x/crypto: 75b2880 → v0.21.0
• golang.org/x/exp: 509febe → a9213ee
• golang.org/x/mod: v0.8.0 → v0.15.0
• golang.org/x/net: v0.10.0 → v0.23.0
• golang.org/x/oauth2: v0.7.0 → v0.12.0
• golang.org/x/sync: 67f06af → v0.6.0
• golang.org/x/sys: v0.8.0 → v0.18.0
• golang.org/x/term: v0.8.0 → v0.18.0
• golang.org/x/text: v0.9.0 → v0.14.0
• golang.org/x/tools: v0.9.1 → v0.18.0
• gomodules.xyz/jsonpatch/v2: v2.3.0 → v2.4.0
• google.golang.org/grpc: v1.40.0 → v1.58.3
• google.golang.org/protobuf: v1.30.0 → v1.33.0
• k8s.io/api: v0.27.4 → v0.30.1
• k8s.io/apiextensions-apiserver: v0.27.2 → v0.30.0
• k8s.io/apimachinery: v0.27.4 → v0.30.1
• k8s.io/client-go: v0.27.4 → v0.30.1
• k8s.io/component-base: v0.27.2 → v0.30.1
• k8s.io/klog/v2: v2.100.1 → v2.120.1
• k8s.io/kube-openapi: 8b0f38b → 70dd376
• k8s.io/utils: d93618c → 3b25d92
• sigs.k8s.io/controller-runtime: v0.15.1 → v0.18.2
• sigs.k8s.io/structured-merge-diff/v4: v4.2.3 → v4.4.1
• sigs.k8s.io/yaml: v1.3.0 → v1.4.0

Removed

• cloud.google.com/go: v0.34.0
• github.com/BurntSushi/toml: v0.3.1
• github.com/OneOfOne/xxhash: v1.2.2
• github.com/antihax/optional: v1.0.0
• github.com/buger/jsonparser: v1.1.1
• github.com/census-instrumentation/opencensus-proto: v0.2.1
• github.com/cespare/xxhash: v1.1.0
• github.com/client9/misspell: v0.3.4
• github.com/cncf/udpa/go: 5459f2c
• github.com/cncf/xds/go: fbca930
• github.com/docopt/docopt-go: ee0de3b
• github.com/envoyproxy/go-control-plane: 63b5d3c
• github.com/envoyproxy/protoc-gen-validate: v0.1.0
• github.com/flowstack/go-jsonschema: v0.1.1
• github.com/ghodss/yaml: v1.0.0
• github.com/golang/glog: 23def4e
• github.com/golang/mock: v1.1.1
• github.com/google/gnostic: v0.6.9
• github.com/grpc-ecosystem/grpc-gateway: v1.16.0
• github.com/mitchellh/mapstructure: v1.1.2
• github.com/pkg/diff: 20ebb0f
• github.com/rogpeppe/fastuuid: v1.2.0
• github.com/spaolacci/murmur3: f09979e
• github.com/xeipuuv/gojsonpointer: 4e3ac27
• github.com/xeipuuv/gojsonreference: bd5ef7b
• github.com/xeipuuv/gojsonschema: v1.2.0
• go.uber.org/atomic: v1.7.0
• golang.org/x/lint: d0100b6
• google.golang.org/genproto: 42d7afd
• honnef.co/go/tools: ea95bdf
• k8s.io/gengo: 485abfe

New Contributors

• @Smeb made their first contribution in #322
• @cartermckinnon made their first contribution in #344
• @dmvolod made their first contribution in https...

v0.3.0

This is the third release of the 2023 after nearly four months of active contribution from the community. This release packs a great deal of interesting features, fixes, and documentation updates. Let's highlight some of them below.

Thread Safe Test Context

When Parallel test features where enabled and integrated into the framework, it was enabled with room for running into race conditions. In this release we have included changes that helps us mitigate these issues when the tests are being run in parallel. This has been achieved by doing the following mechanism.

1. Populate the test context via the BeforeEachTest handler
2. Generate a child context from the parent context and provide that for individual tests/assessments
3. Discard the context from step 2 after the tests are done
4. Pass the original parent context to the AfterEachTest handler

As part of the said changes to mitigate the race condition, we also extended the Environment.Test and Environment.TestInParallel function to return a context.Context back after the execution of the tests have been completed. This also allows for better debugging of test context to analyze failures better with the added advantage that this is not breaking any of the existing contracts. However, if you are using a golint infra and have errcheck linter enabled, you will have to make a few changes to your code to account for the newly returned value from the Environment.Test and Environment.TestInParallel functions.

Limitations

1. The Finish phase will only be able to see the context from the Setup phase and not the one from the features themselves.

Related Issues

1. #216
2. #258

Related Pull Requests

1. #292

Continubutors

@phisco @maruina

FluxCD Integration

e2e-framework has had helm workflow integrated under the third_party support package for a while and now, we are adding FluxCD to that arsenal. With this integration, now you can integrate your e2e tests to run against a repository using fluxcd constructs. The supported features include the following.

1. Install FluxCD components onto your cluster
2. Create and Delete GitRepository resource
3. Create and Delete Kustomization resource

Related Pull Requests

1. #255
2. #283

Contributors

@matrus2

kwok Integration as a Cluster Provider

By default, e2e-framework has two ways to run tests. One could bring up a new cluster using kind and run the e2e tests against them using the framework or integrate against a real cluster and run tests against them. In this release,
kwok is being added to that list of supported cluster providers.

Since kwok provided a simulated kubelet instead of running a real kubelet, this can helm create a cluster with large number of nodes with very quick turn around time, reducing the turn around time to run the e2e tests. With this support, end users of the framework can now standup a simple kwok based cluster and run their tests against it. This also supports discovering the kowkctl binary from non standard location in order to simplify the integration.

Releated Issues

1. #214

Related Pull Requests

1. #239

Contributors

@reetasingh

Support for Custom Binary Paths

With this release, e2e-framework provides a mechanism where by, end users can provide a custom path from where the binaries such as kwokctl, helm or kind can be discovered. This helps consumers of the framework who wants to keep their binaries in non standard path outside of the $PATH and consume them for the integration needs.

Related Issues

1. #282

Related Pull Requests

1. #286

Contributors

@harshanarayana

Cluster Provider Interface for easy inclusion of additional Cluster providers

Until now, e2e-framework had a series of custom built hand crafted helper functions defined under envfuncs that enabled end users to perform operation such as instanciating a new cluster, destoring a cluster, collecting logs etc. However, with the interest in adding additional cluster providesr such as kwok, k3d and possibly many others in the future, it became critical that we define a set of common interfaces that can be implemented by the provider so that
we can avoid having duplication in the code provided under envfuncs package. What started as a discussion during the review of kwok provider integration turned into a full blown feature to enable better integration of providers in the
future.

As part of this work, e2e-framework not provides an interface named E2EClusterProvider which can be implemented by any cluster provider that we want to integrate into the framework and the existing envfuncs can be used as is for the new provider without having to add new code to it.

Providers can also implement an additonal optional interface E2EClusterProviderWithImageLoader which extends the E2EClusterProvider and adds two more additional supported feature around being able to load a container image into the cluster. Either as individual images or as a archieve. (kind currenrly supports this workflow)

Deprecation

As part of this implementation, the following envfuncs have been deprecated and should be replaced with the respective alternative in the future.

1. GetKindClusterFromContext can be replaced with GetClusterFromContext
2. CreateKindCluster can be replaced with CreateCluster
3. CreateKindClusterWithConfig can be replaced with CreateClusterWithConfig
4. DestroyKindCluster can be replaced with DestroyCluster
5. ExportKindClusterLogs can be replaced with ExportClusterLogs

Following section has a few example of what this replacement would look like. (Code snippets taken from examples)

Deprecated kind based Cluster setup

func TestMain(m *testing.M) {
    testenv = env.New()
	kindClusterName := envconf.RandomName("decoder", 16)
	testenv.Setup(
		envfuncs.CreateKindCluster(kindClusterName),
	)
	testenv.Finish(
		envfuncs.DestroyKindCluster(kindClusterName),
	)
    os.Exit(testenv.Run(m))
}

Suggested kind based Cluster Setup

func TestMain(m *testing.M) {
	testenv = env.New()
	kindClusterName := envconf.RandomName("decoder", 16)
	testenv.Setup(
		envfuncs.CreateCluster(kind.NewProvider(), kindClusterName),
	)
	testenv.Finish(
		envfuncs.DestroyCluster(kindClusterName),
	)
	os.Exit(testenv.Run(m))
}

Related Issues

1. #245

Related Pull Requests

1. #246

Contributors

@harshanarayana

Subresource Update support via the klient's resources package

klient's resources now supports a few new helper method that are aimed towards updating the sub resources.

1. func (r *Resources) UpdateSubresource(ctx context.Context, obj k8s.Object, subresource string, opts ...UpdateOption) error {}
2. func (r *Resources) UpdateStatus(ctx context.Context, obj k8s.Object, opts ...UpdateOption) error {}
3. func (r *Resources) PatchSubresource(ctx context.Context, obj k8s.Object, subresource string, patch k8s.Patch, opts ...PatchOption) error {}
4. func (r *Resources) PatchStatus(ctx context.Context, objs k8s.Object, patch k8s.Patch, opts ...PatchOption) error {}

These new helper functions make it very easy to interact with subresources as part of your e2e-framework based tests

Related Pull Requests

1. #249

Contributors

@wzshiming

Other notable changes

1. Updated kubernetes component dependency to v1.27.x by @harshanarayana in #244
2. Enabled ability to use --labels where the selectors have the same key (i.e --labels="feature=foo,feature=bar") by @embano1 in #248
3. Added support for DeploymentAvailable condition check helper by @ryankwilliams in #251
4. Enhanced example for using namespaces passed to the test via context by @maruina in #253
5. Improved error reporting for kind Cluster provided by @mmanciop in #256
6. Improved error result for helm command workflows by @bradbeam in #262
7. Improved conversion of resource handler options properly between metav1 and controller-runtime by @harshanarayana in #278
8. Enabled linters on examples along with actual code by @harshanarayana in #281
9. Added documentation about e2e-framework adopters by @vladimirvivien in #285
10. Added support for adding descriptions under table driven test definition model by @harshanarayana in #284
11. Removed unused random source seeding property by @matrus2 in #294
12. Enabled Github Issue and PR templates by @harshanarayana in #298
13. Enabled printing the stacktrace when th...

v0.2.0

This is the second release for 2023 after three months (and few days) of contribution from the community. This release packs a great deal of interesting features, fixes, and documentation updates. Let's highlight some of the below.

Properly filter tests with labels with same key values

This fix/feature, contributed by @embano1, allows e2e-framework tests to parse labels with multiple values, that share the same key, properly. Prior to this fix, tests would not get filtered properly when different features happen to use the same key names. This fix makes it possible to have features with labels with same key names.

Support for Kubernetes cluster context

@maruina contributed code that allows e2e-framework tests to specify the name of a cluster context using the --context flag. With this feature, your tests can now target a specific cluster context during execution.

Exporting kind logs during tests

@alexandrevilain contributed a new feature that allows test authors to easily extract kind logs during tests. The feature is implemented as an envfunc function which means it can be used during any step of the life cycle of a running test. See this feature being used in this example.

Improved documentation

@harshanarayana reorganized the documentation so that readers can easily find feature documentation for the project. You can reach the newly created doc section here.

@maruina contributed documentation that shows how to setup e2e-framework tests using custom CLI flags. See the doc here

@reetasingh created documentation that shows how to use the newly supported -skip when using go test. The doc highlights how to use the new flag along with other skip features already supported in the project. For detail see doc here.

Other notable updates

@cpanato is doing an awesome job supporting the CI/CD pipeline for the project. His work has allowed dependabot to work smoothly allowing the project to keep up with the latest dependencies releases and fixes things when dependabot breaks them. Thanks @cpanato!

@pmalek create an alias for type env.TestFunc. Though a simple change, it will make it easier to repurpose environment functions in many parts of the code and authored tests.

What's Changed

Full changelog of everything that got changed:

• feat: add logging for conditions.ResourcesDeleted by @maximilianbraun in #185
• fix: Parse multiple label values for same key by @embano1 in #196
• Add documentation about passing custom flags by @maruina in #203
• Add --context flag by @maruina in #197
• Document -skip flag in go test by @reetasingh in #204
• feat: add env.TestFunc type alias by @pmalek in #210
• Update go.mod to be min 1.19 and golangci-lint by @cpanato in #226
• docs: Update Document organization structure by @harshanarayana in #206
• update helm test by @cpanato in #229
• Updates for dependabot and ci job by @cpanato in #225
• Add envfunc to export kind cluster logs by @alexandrevilain in #220
• pin helm chart to a specific version by @cpanato in #231
• docs: update main README reference to design doc by @piotrkpc in #238

New Contributors

• @maximilianbraun made their first contribution in #185
• @embano1 made their first contribution in #196
• @maruina made their first contribution in #203
• @reetasingh made their first contribution in #204
• @pmalek made their first contribution in #210
• @alexandrevilain made their first contribution in #220
• @piotrkpc made their first contribution in #238

Full Changelog: v0.1.0...v0.2.0

v0.1.0

The e2e-framework continues to evolve with useful features for code writers looking for tooling to test their components running in Kubernetes. As with previous releases, members of the community contributed the lion share of this release.

Version number change

After nearly 2 years of being in development, this release will adopt the minor version number, starting with v0.1.0, to indicate the relative stability and continued adoption of the project.

Run commands inside pods

New in this release is the ability to programmatically launch commands that get executed inside a pod. This a useful feature that allows e2e-framework test writers to test code from within the pod itself. For instance, the following uses the ExecInPod method call to check connectivity from whithin a running pod.

func TestExecPod(t *testing.T) {
    deploymentName := "test-deployment"
    containerName := "curl"
    feature := features.New("Call external service").
        Assess("check connectivity to wikipedia.org main page", func(ctx context.Context, t *testing.T, c *envconf.Config) context.Context {
            client, _ := c.NewClient()
            pods := &corev1.PodList{}
            err = client.Resources(c.Namespace()).List(context.TODO(), pods)
            if err != nil || pods.Items == nil {
                t.Error("error while getting pods", err)
            }
            var stdout, stderr bytes.Buffer
            podName := pods.Items[0].Name
            command := []string{"curl", "-I", "https://en.wikipedia.org/wiki/Main_Page"}
            
            err := client.Resources().ExecInPod(c.Namespace(), podName, containerName, command, &stdout, &stderr)
            if err != nil {
                t.Log(stderr.String())
                t.Fatal(err)
            }
            
            httpStatus := strings.Split(stdout.String(), "\n")[0]
            if !strings.Contains(httpStatus, "200") {
                t.Fatal("Couldn't connect to en.wikipedia.org")
            }
            return ctx
        }).Feature()
    testEnv.Test(t, feature)
}

For further detail, see the example on ExecInPod.

Support for Kubernetes-SIGs/Kubetest2

Another feature introduced in this release is the support for running e2e-framework tests using the kubetest2. Assuming that your environment has the kubetest2 binary and KinD installed on the OS path, the following example will launch kind, run the e2e-framework tests found in the specified package directory, and shutdown kind when done.

kubetest2 kind --up --down           \
     --test=e2e-framework --         \
     --packages ./cluster            \
     --kubeconfig=$HOME/.kube/config \
     --skip-assessments=pod-count

For additional detail on kubetest2 support, see README in the third_party directory.

Access to Controller-Runtime client

When writing tests, sometimes you may want to have direct access to the controller-runtime client being used in the framework or use an existing client in your code. This release allows test writers to inject an existing client or access the client being used by e2e-framework.

func TestClient() {
func TestExecPod(t *testing.T) {
    feature := features.New("Client").
        Assess("access client", func(ctx context.Context, t *testing.T, c *envconf.Config) context.Context {
            e2eC, _ := c.NewClient()
            ctrlC := e2eC.GetControllerRuntimeClient()
            // use controller-runtime client directly
            ...
            return ctx
        }).Feature()
    ...
}

Other notable updates

• @harshanarayana added as project approver/maintainer for his many contributions
• Setup of GitHub depabot for automatic updates of source dependencies
• Minor documentation and code fix updates

Contributors

Special thanks to all who contributed:

@harshanarayana
@v0lkc
@matrus2
@mitchmckenzie
@sozercan
@jbpratt
@cpanato

What's Changed

• chore: bump vladimirvivien/gexe to v0.2.0 by @v0lkc in #166
• Add ExecInPod method with an example test by @matrus2 in #167
• controller-runtime: enable ability to fetch controller runtime client by @harshanarayana in #156
• Fix decoder delete ignore not found by @mitchmckenzie in #170
• bump kind to 0.17 by @sozercan in #172
• enable dependabot scheduled updates by @jbpratt in #171
• update k8s dependencies by @cpanato in #181
• Bump sigs.k8s.io/controller-runtime from 0.13.1 to 0.14.0 by @dependabot in #182
• Bump sigs.k8s.io/controller-runtime from 0.14.0 to 0.14.1 by @dependabot in #183
• Support for an e2e-framework kubetest2 tester by @vladimirvivien in #168
• Adding harshanarayana as review/approver by @vladimirvivien in #184

New Contributors

• @v0lkc made their first contribution in #166
• @mitchmckenzie made their first contribution in #170
• @sozercan made their first contribution in #172
• @jbpratt made their first contribution in #171

Full Changelog: v0.0.8...v0.1.0

Release v0.0.8

This release include example and doc updates, fixes, and new features.

Handlers to gracefully finish tests

A new feature was introduce to allow tests to gracefully recover, after a runtime panic, and execute the test environment's Finish step. If a test writer would rather have tests abruptly end, that behavior can be turned off with the new flag disable-graceful-teardown which would cause the test execution to stop, skipping any finalization steps in Finish.

Multi-cluster test workflow

Certain tests can require more than one cluster to be available during execution. This release introduces the ability to start one or more test clusters as shown in the following snippet:

func TestMain(m *testing.M) {
	testEnv = env.NewConfig()

	clusterNames = []string{
		envconf.RandomName("cluster-one", 16),
		envconf.RandomName("cluster-two", 16),
	}

	testEnv.Setup(
		func(ctx context.Context, config *envconf.Config) (context.Context, error) {
			var err error
                         // create two clusters
			for _, cluster := range clusterNames {
				ctx, err = envfuncs.CreateKindCluster(cluster)(ctx, config)
				if err != nil {
					return ctx, err
				}
			}
			return ctx, nil
		},
	).Finish(
		func(ctx context.Context, config *envconf.Config) (context.Context, error) {
			var err error
			// shutdown clusters when done
			for _, cluster := range clusterNames {
				ctx, err = envfuncs.DestroyKindCluster(cluster)(ctx, config)
				if err != nil {
					return ctx, err
				}
			}
			return ctx, nil
		},
	)

	os.Exit(testEnv.Run(m))
}

For more information, see multi-cluster example.

Other notable updates

• Support for Helm package uninstall
• Project dependency update to Kubernetes 1.24.1
• Additional tests cases for the Watch functionality
• New example showing how to test components running on a cloud provider (EKS, AKS, GKE, etc)
• And other minor fixes

Contributors

Special thanks to all who contributed to this release including:

• @fracasula
• @0xff-dev
• @harshanarayana
• @ernado
• @ShwethaKumbla
• @matrus2
• @twpayne
• @tklauser
• @mhofstetter

Changelog

• 9d551b5 Fix lint errors
• 5d2b591 Fix log output on errors during test lifecycle actions
• b7b9982 build: fix golangci-lint SA1019 io/ioutil
• e1b7605 GIT-138: add example of multi cluster test workflow
• ce742cf examples/wait_for_resources: fix Go code indentation in README
• 31960c4 Add env to distinguish type of cluster
• daf1681 Add example to readme
• ece3ddb docs: Fix some typos and code examples
• 3a33e3a Add example of how to use e2e framework with real cluster
• ebe68b0 additional test cases to validate watch functionality
• 50d84f2 GIT-141: enable panic handlers for ensuring Finish Steps
• 03e0588 update k8s dependencies to v1.24.1
• f255dbc docs(readme): fix test example
• 2ad0d74 Fix function call errors in README
• 48bbdfb feat: helm uninstall support

Release v0.0.7

This release has some great features that have been contributed by members of the community.

Custom resource testing

This release introduces new features that makes it easy to create integration tests for custom resources. One enhancement to the klient package now allows to registration of arbitrary resource schemes allowing testing of arbitrary resources such as custom resources.

Another improvement is the introduction of helper functions, envfuncs.SetupCRDs and envfuncs.TeardownCRDs, to setup/teardown custom resource definition YAML files when testing of custom resources.

func TestMain(m *testing.M) {
	cfg, _ := envconf.NewFromFlags()
	testEnv = env.NewWithConfig(cfg)
	kindClusterName = envconf.RandomName("crdtest-", 16)
	namespace = envconf.RandomName("my-ns", 10)

	testEnv.Setup(
		envfuncs.CreateKindCluster(kindClusterName),
		envfuncs.CreateNamespace(namespace),
		envfuncs.SetupCRDs("./testdata/crds", "*"),
	)

	testEnv.Finish(
		envfuncs.DeleteNamespace(namespace),
		envfuncs.TeardownCRDs("./testdata/crds", "*"),
		envfuncs.DestroyKindCluster(kindClusterName),
	)

	os.Exit(testEnv.Run(m))
}

This feature, along with other facilities in the framework, should make it easy to test components such as custom resources and their controllers.

Read more about this feature here.

Fail fast mode

This feature allows test writers to short circuit the execution of test, causing the entire test to fail immediately when a fail signal is encountered.

func TestExample(t *testing.T) {
	failFeature := features.New("fail-feature").
		Assess("1==2", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
			if 1 != 2 {
				t.Log("1 != 2")
				t.FailNow() // mark test case as failed here, don't continue execution
			} else {
				t.Log("1 == 2")
			}
			return ctx
		}).
		Assess("print", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
			t.Log("THIS LINE SHOULDN'T BE PRINTED")
			return ctx
		}).
		Teardown(func(ctx context.Context, t *testing.T, c *envconf.Config) context.Context {
			t.Log("This teardown should not be invoked")
			return ctx
		}).
 		Feature()

	nextFeature := features.New("next-feature").
		Assess("print", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
			t.Log("THIS LINE ALSO SHOULDN'T BE PRINTED")
			return ctx
		}).
		Feature()

	testenv.Test(t, failFeature, nextFeature)
}

When the previous is executed with the --fail-fast flag, the execution of the test function will exit when t.FailNow() is encountered.

Read more about this feature here.

Dry run mode

This release introduces a new feature with a new CLI flag, --dry-fun, that lists the expected feature tests without actually running them.

go test . -test.v -args --dry-run

=== RUN   TestPodBringUp
=== RUN   TestPodBringUp/Feature_One
=== RUN   TestPodBringUp/Feature_One/Create_Nginx_Deployment_1
=== RUN   TestPodBringUp/Feature_One/Wait_for_Nginx_Deployment_1_to_be_scaled_up
=== RUN   TestPodBringUp/Feature_Two
=== RUN   TestPodBringUp/Feature_Two/Create_Nginx_Deployment_2
=== RUN   TestPodBringUp/Feature_Two/Wait_for_Nginx_Deployment_2_to_be_scaled_up
--- PASS: TestPodBringUp (0.00s)
    --- PASS: TestPodBringUp/Feature_One (0.00s)
        --- PASS: TestPodBringUp/Feature_One/Create_Nginx_Deployment_1 (0.00s)
        --- PASS: TestPodBringUp/Feature_One/Wait_for_Nginx_Deployment_1_to_be_scaled_up (0.00s)
    --- PASS: TestPodBringUp/Feature_Two (0.00s)
        --- PASS: TestPodBringUp/Feature_Two/Create_Nginx_Deployment_2 (0.00s)
        --- PASS: TestPodBringUp/Feature_Two/Wait_for_Nginx_Deployment_2_to_be_scaled_up (0.00s)
PASS
ok  	sigs.k8s.io/e2e-framework/examples/parallel_features	0.353s

Read more about this feature here.

A new API to watch resources

This release comes with a new API to facilitate resource watching during tests allowing test authors to write reactive test code by getting notifications when resource state changes.

For instance, the following shows how to setup your test to watch a Deployment object and callback functions to handle the changes.

Setup(func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
    cl, err := cfg.NewClient()
    if err != nil {
        t.Fatal(err)
    }

	dep := appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: "watch-dep", Namespace: cfg.Namespace()}}

// Start watching for the deployment and triger action based on the event received.
    cl.Resources().Watch(&appsv1.DeploymentList{}, resources.WithFieldSelector(labels.FormatLabels(map[string]string{"metadata.name": dep.Name}))).
	WithAddFunc(onAdd).WithDeleteFunc(onDelete).Start(ctx)

    return ctx
})

// Call backs
func onAdd(obj interface{}) {
	dep := obj.(*appsv1.Deployment)
	depName := dep.GetName()
	if depName == "watch-dep" || depName == "watchnstop-dep" {
		klog.InfoS("Deployment name matches with actual name!")
	}
}

func onDelete(obj interface{}) {
	dep := obj.(*appsv1.Deployment)
	depName := dep.GetName()
	if depName == "watch-dep" || depName == "watchnstop-dep" {
		klog.InfoS("Deployment deleted successfully!")
	}
}

For more detail on this feature, see the example.

Extended the test Feature API

The feature test type now supports arbitrary name for all steps allowing for better parsing of test results

func TestHello_WithSetup(t *testing.T) {
	e := env.NewWithConfig(envconf.New())
	var name string
	feat := features.New("Hello Feature").
		WithLabel("type", "simple").
		WithSetup("SetupName", func(ctx context.Context, t *testing.T, _ *envconf.Config) context.Context {
			name = "foobar"
			return ctx
		}).
		Assess("test message", func(ctx context.Context, t *testing.T, _ *envconf.Config) context.Context {
			result := Hello(name)
			if result != "Hello foobar" {
				t.Error("unexpected message")
			}
			return ctx
		}).WithTeardown("Teardown", func(ctx context.Context, t *testing.T, _ *envconf.Config) context.Context {
                ...
		}).Feature()

	e.Test(t, feat)
}

Fix: Parallel test scoping

This release introduces a fix for a race condition that was causing data overwrites when running feature tests in parallel.

Other improvements

• Improved Github Actions
• Automation using goreleaser

Changelog

8339ef1 k8s resource watch and triger action based on the events
7b3aabf GIT-109: enable CRD setup helper to ease the testing of operators
b69e158 GIT-112: enable framework specific fail-fast mode
d19222a GIT-62: Enable --dry-run mode
ed7d3ad Add support for registring other types with klient
a22cbf1 Update Owners: add ShwethaKumbla as approver
2b14f5e GIT-119: uptick kind and fix the install mode
e47ad0c Update config.go
3467252 GIT-119: handle scoping for parallel tests
353abeb GIT-94: enable extended Feature API to provide named setup and teardown
1b2232e add github actions/goreleaser to release the lib

v0.0.6

Release v0.0.6

This project is fast moving. As with previous releases, members of the community has contributed a set of great new features. Thank you to everyone involved 😃

YAML/JSON decoder

This release introduces package klient/decoder to help test writers to create Kubernetes API objects from YMAL or JSON snippets. This provides a convenient method to programmatically generate API objects from YAML or JSON string values. The decoder supports the followings:

• Decoding single-document YAML/JSON input
• Decoding a multi-document YAML/JSON input
• Ability to decode from a Go io.Reader

For detail, see the design doc.

Example

Given file testdata/config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-config
data:
  example.yaml: |
    key: value

The following can be used to decode and create API objects from a file:

func main {
    f, err := os.Open("testdata/config.yaml")
	if err != nil {
		t.Fatal(err)
	}
	obj, err := decoder.DecodeAny(f)
	if err != nil {
		log.Fatal(err)
	}
	configMap, ok := obj.(*v1.ConfigMap)
	if !ok {
		log.Fatal("object decoded to unexpected type")
	}
}

Alternatively, the package supports the use of handler functions to provider decoder logic for objects. For instance, assuming directory testdata has several YMAL files, the following snippet would automatically decode each object and insert them (using the CreateHandler handler function) in the API server.

func main() {
    r, err := resources.New(cfg.Client().RESTConfig())
    if err != nil {
        log.Fatal(err)
    }

    if err := decoder.DecodeEachFile(ctx, os.DirFS("testdata"), "*",
        decoder.CreateHandler(r),           // try to CREATE objects after decoding
		decoder.MutateNamespace(namespace), // inject a namespace into decoded objects, before CREATE
	); err != nil {
		log.Fatal(err)
	}
}

See YMAL/JSON decoder example here.

Parallel feature tests

New with this release is the ability to test multiple features in parallel. When a test environment receives multiple features, the test features can be executed in parallel either by using a CLI argument flag:

go test -v . -args --parallel

Or, the parallel test execution can be triggered programmatically with Environment.TestInParallel method:

func TestPodBringUp(t *testing.T) {
    featureOne := features.New("Feature One").
        Assess("Create Nginx Deployment 1", func(ctx context.Context, t *testing.T, config *envconf.Config) context.Context {
            deployment := newDeployment(namespace, "deployment-1", 2)
            if err := config.Client().Resources().Create(ctx, deployment); err != nil
                t.Error("failed to create test pod for deployment-1")
            }
            return ctx
        }).Feature()

    featureTwo := features.New("Feature Two").
        Assess("Create Nginx Deployment 2", func(ctx context.Context, t *testing.T, config *envconf.Config) context.Context {
            deployment := newDeployment(namespace, "deployment-2", 2)
            if err := config.Client().Resources().Create(ctx, deployment); err != nil {
                t.Error("failed to create test pod for deployment-2")
            }
            return ctx
        }).Feature()

    testEnv.TestInParallel(t, featureOne, featureTwo)
}

See parallel feature test example.

Third-party tool support (Helm)

This release introduces the notion of third-party tool support, found in the third_party directory. The first tool supported is Helm. The release introduces the ability to integrate Helm-managed deployments programmatically into the e2e tests as shown below.

func TestHelmChartRepoWorkflow(t *testing.T) {
    feature := features.New("Repo based helm chart workflow").
        Setup(func(ctx context.Context, t *testing.T, config *envconf.Config) context.Context {
            manager := helm.New(config.KubeconfigFile())
            if err := manager.RunRepo(helm.WithArgs("add", "nginx-stable", "https://helm.nginx.com/stable")); err != nil {
                t.Fatal("failed to add nginx helm chart repo")
            }
            if err := manager.RunRepo(helm.WithArgs("update")); err != nil {
                t.Fatal("failed to upgrade helm repo")
            }
            if err := manager.RunInstall(helm.WithName("nginx"), helm.WithNamespace(namespace), helm.WithReleaseName("nginx-stable/nginx-ingress")); err != nil {
                t.Fatal("failed to install nginx Helm chart")
            }
            return ctx
        }).Feature()
   ...
}

See Helm support example here.

Test hooks have access *testing.T

In this release, test environment hooks Environment.BeforeEachTest/Environment.AfterEachTest and Environment.BeforeEachFeature/Environment.AfterEachFeature will now receive *testing.T. This is useful for test features that are interested in receiving pass/fail status for instance.

func TestMain(m *testing.M) {
    testenv = env.New()
    
    testenv.Setup(
        envfuncs.CreateKindCluster(kindClusterName),
    )
    
    testenv.BeforeEachFeature(func(ctx context.Context, _ *envconf.Config, t *testing.T, info features.Feature) (context.Context, error) {
        return envfuncs.CreateNamespace("ns-name")
    })
    testenv.AfterEachFeature(func(ctx context.Context, _ *envconf.Config, _ *testing.T, info features.Feature) (context.Context, error) {
        if t.Failed() {
          return envfuncs.DeleteNamespace("ns-name")
        }
        return ctx, nil
    })

    testenv.Finish(
        envfuncs.DestroyKindCluster(kindClusterName),
    )
    os.Exit(testenv.Run(m))
}

Other updates

• Support for controller-runtime version 0.11.0
• Update to use golangci-lint v1.44.0
• Enhancement to KinD support when tearing down clusters

Changelog

c1aec2e Update controller-runtime to 0.11.0 and their dependencies
b7a8a56 update golangci-lint to v1.44.0
f431b0b update repo infra to release v0.2.5
72e1963 Add *testing.T to before/after feature hooks
0c8ca84 GIT-83: enable helm support for test workflow
301dd09 GIT-49: enable paralle run of test features
0970afe add yaml and json helpers
bd4c3cd match on exact cluster names

Release v0.0.5

This release is packed with new and super useful functionalities to continue to make it easy to create end-to-end tests, including:

The new wait and conditions package

The wait and conditions packages allow test writers to express cluster conditions to wait for before proceeding, during a test execution. The API takes the same simple and familiar approach by providing an expressive way to construct conditional predicates as shown below:

// Test to wait for a deployment condition
func TestDeployment(t *testing.T) {
    res := envconf.New().Client().Resources() // get default resource manger
    deployment := createDeployment("d2") // create a deployment object

    // wait for the deployment to become at least 50% available within 1 minute
    err = wait.For(conditions.New(res).ResourceMatch(&dep, func(object k8s.Object) bool {
        d := object.(*appsv1.Deployment)
	return float64(d.Status.ReadyReplicas)/float64(*d.Spec.Replicas) >= 0.50
    }), wait.WithTimeout(time.Minute*1))
    ...
}

Pre-defined conditions

To make test-writing more convenient, the package comes with a long list of pre-defined conditions that can be used in your tests as shown below:

// Test to wait for a deployment condition
func TestDeploymentConditionMatch(t *testing.T) {
    res := envconf.New().Client().Resources() // get default resource manger
    deployment := createDeployment("d2") // create a deployment object

    // wait for deployment condition (deployment is available)
    err := wait.For(conditions.New(res).DeploymentConditionMatch(deployment, appsv1.DeploymentAvailable, v1.ConditionTrue))
    ...
}

Other pre-defined conditions

• ResourceScaled - resource scaled to a specified replica number
• ResourceMatch - matches a provided condition
• ResourceListN - number of available resource list matches a provided number
• ResourcesFound specified resource(s) are found in the cluster
• ResourcesDeleted specified resource(s) has/have been deleted
• JobConditionMatch - job status matches a specified condition
• JobFailed / JobCompleted - waits for specified job to fail / job to complete
• DeploymentMatch - deployment matches a provided condition/status
• PodConditionMatch - pod matches a status or condition
• PodPhaseMatch - pod matches a specified phase
• PodReady / PodRunning - specified pod is ready / pod is running
• ContainersReady - wait for containers in specified pod to be ready

Table-driven test representation

This release introduces table-driven tests as a convenient way to define tests

var test = env.New()
...
func TestTableDriven(t *testing.T) {
	// feature 1
	table0 := features.Table{
		{
			Name: "less than equal 64",
			Assessment: func(ctx context.Context, t *testing.T, config *envconf.Config) context.Context {
				rnd := ctx.Value("randsrc").(*rand.Rand) 
				lim := ctx.Value("limit").(int32) 
				if rnd.Int31n(lim) > 64 {
					t.Log("limit should be less than 64")
				}
				return ctx
			},
		},
		{
			Name: "more than than equal 128",
			Assessment: func(ctx context.Context, t *testing.T, config *envconf.Config) context.Context {
				rnd := ctx.Value("randsrc").(*rand.Rand)  // in real test, check asserted type
				lim := ctx.Value("limit").(int32) // check type assertion
				if rnd.Int31n(lim) > 128 {
					t.Log("limit should be less than 128")
				}
				return ctx
			},
		},
	}.Build("Random numbers").Feature()
        test.Test(t, table0.Build().Feature())

Other enhancements

• Integration of klog package for leveled logging
• Ability to specify custom configuration for KinD during tests
• Ability to specify docker image and local image when launching KinD clusters during tests

Changelog

c4a197e Support for table-driven tests
67ca656 add condition helper for checking deployment status and methods for waiting on object lists
041d212 support loading images into kind cluster
0001077 GIT-77: enable klog based generic logging infra with verbose handlers
bdb0467 use stdout from kind command when writing kubeconfig
11cb5d1 GIT-81: fix syntax highlighting for klient Design Doc
6d24c50 Fix typos in the doc
40fee41 fetch kubeconfig if cluster is already created
b79a789 GIT-60: enable helper function for conditional waits

v0.0.4

Version 0.0.4 continues to add valuable functionalities to the project and includes new contributors from the community:

• Enhancement to label support - tests can be included and excluded based on labeled assessments.
• Enhancement to kind support - ability to launch KinD using user-specified configuration file
• Update to use the klient package to make it easier to specify in/out of cluster usage
• Rework of the test harness API so that call back functions receive richer info about executing tests
• Additional examples showing how to use these new functionalities
• And much more

Changelog

7635793 Kind Cluster with parameter
ffd0f68 Rework the FeatureInfo wiring
1ada6a6 Add more info for before/after hooks
45a0029 Make it easier to use in-cluster configurations
9c1d186 Ensure kind and PATH are set as needed
3771367 Support skipping tests by labels and feature name
2e30656 Rework some unit test patterns
84baa29 Flags support fix and examples
8e8d9d9 Run only filtered features by their assigned labels using --labels