The k6 Operator is not working

[feliu]

I'm running the following version:
GKE: 1.20.15-gke.300

Logs:

2022-03-14T18:04:11.767Z	INFO	controllers.K6	perf-test-service-1 service is not ready, aborting	{"k6": "atlantis-perftest/perf-test"}
2022-03-14T18:04:16.755Z	INFO	controllers.K6	1/1 pods ready	{"k6": "atlantis-perftest/perf-test"}
2022-03-14T18:04:16.768Z	ERROR	controllers.K6	failed to get status from perf-test-service-1	{"k6": "atlantis-perftest/perf-test", "error": "Get \"http://perf-test-service-1.atlantis-perftest.svc.cluster.local:6565/v1/status\": read tcp 10.132.4.162:52154->10.128.219.182:6565: read: connection reset by peer"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
github.com/grafana/k6-operator/controllers.isServiceReady
	/workspace/controllers/k6_start.go:23
github.com/grafana/k6-operator/controllers.StartJobs.func1
	/workspace/controllers/k6_start.go:74
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:211
k8s.io/apimachinery/pkg/util/wait.WaitFor
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:541
k8s.io/apimachinery/pkg/util/wait.pollInternal
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:427
k8s.io/apimachinery/pkg/util/wait.pollImmediateInternal
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:452
k8s.io/apimachinery/pkg/util/wait.PollImmediate
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:441
github.com/grafana/k6-operator/controllers.StartJobs
	/workspace/controllers/k6_start.go:34
github.com/grafana/k6-operator/controllers.(*K6Reconciler).Reconcile
	/workspace/controllers/k6_controller.go:63
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90

[yorugac]

Hi @feliu! Going by your log, it seems like the runner pod has started successfully:

INFO	controllers.K6	1/1 pods ready

But then controller failed to reach the REST service:

ERROR	controllers.K6	failed to get status from perf-test-service-1	

Controller should try to continue to reach it for some time, up to a minute -- at the moment, I assume it continued to fail. If so, could you please check if perf-test-service-1.atlantis-perftest.svc.cluster.local is a correct and reachable address in your cluster?

[yorugac]

Hi @feliu, Did you manage to resolve the issue? 🙂

[brettstewart]

Chiming in here with a few checks @feliu

1. As @yorugac said, I'd double-check that the service actually exists in K8s and the pod it's pointing to has port 6565 Exposed.
2. I'd verify your NetworkPolicy in the atlantis-perftest namespace is allowing connection from pods in the k6-operator-system namespace

[yorugac]

Closing this as there hasn't been updates awhile.

Also, a related issue to improve networking errors: #186