Using GitOps with Grafana Operator to create Grafana Dashboard with Flux. #164

elamaran11 · 2023-05-25T18:06:37Z

What does this PR do?

🛑 Please open an issue first to discuss any significant work and flesh out details/direction - we would hate for your time to be wasted. Consult the CONTRIBUTING guide for submitting pull-requests.

Motivation

Using GitOps with Grafana Operator to create Grafana Dashboard with Flux.

More

Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
Yes, I ran pre-commit run -a with this PR
Yes, I have added a new example under examples to support my PR (when applicable)
Yes, I have updated the Pages for this feature

Note: Not all the PRs required examples and docs.

For Moderators

E2E Test successfully complete before merge?

Additional Notes

bonclay7 · 2023-05-30T06:45:20Z

This looks good @elamaran11, how do we validate this?

elamaran11 · 2023-05-30T13:12:59Z

This looks good @elamaran11, how do we validate this?

@bonclay7 Just run it as is. It will work and you can validate a new dashboard getting created.

bonclay7 · 2023-06-02T17:48:44Z

@elamaran11 To fully validate this, maybe we should drop this in terraform? so we are sure the operator set it up correctly?

And this too?

bonclay7 · 2023-06-02T17:57:21Z

So I went ahead and did it, I dropped the dashboard provided by Terraform but for some reason the operator can't renew my API key, maybe i'm doing something wrong?

1.6857285045139527e+09	INFO	found matching Grafana instances for datasource	{"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "GrafanaDatasource": {"name":"grafanadatasource-sample-amp","namespace":"grafana-operator"}, "namespace": "grafana-operator", "name": "grafanadatasource-sample-amp", "reconcileID": "72cfd60c-a255-44a1-bfbd-88b0cbc4f90c", "count": 1}
1.6857285045445938e+09	INFO	found matching Grafana instances for dashboard	{"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "GrafanaDashboard": {"name":"external-grafanadashboard-url","namespace":"grafana-operator"}, "namespace": "grafana-operator", "name": "external-grafanadashboard-url", "reconcileID": "5a07e7b0-d200-4223-94bc-a2fe6e6009db", "count": 1}
1.6857285045556655e+09	ERROR	error reconciling datasource	{"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "GrafanaDatasource": {"name":"grafanadatasource-sample-amp","namespace":"grafana-operator"}, "namespace": "grafana-operator", "name": "grafanadatasource-sample-amp", "reconcileID": "72cfd60c-a255-44a1-bfbd-88b0cbc4f90c", "datasource": "grafanadatasource-sample-amp", "grafana": "external-grafana", "error": "status: 401, body: {\"message\":\"Expired API key\"}\n"}
github.com/grafana-operator/grafana-operator/controllers.(*GrafanaDatasourceReconciler).Reconcile
	github.com/grafana-operator/grafana-operator/controllers/datasource_controller.go:209
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234
1.6857285046102674e+09	ERROR	error reconciling dashboard	{"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "GrafanaDashboard": {"name":"external-grafanadashboard-url","namespace":"grafana-operator"}, "namespace": "grafana-operator", "name": "external-grafanadashboard-url", "reconcileID": "5a07e7b0-d200-4223-94bc-a2fe6e6009db", "dashboard": "external-grafanadashboard-url", "grafana": "external-grafana", "error": "status: 401, body: {\"message\":\"Expired API key\"}\n"}
github.com/grafana-operator/grafana-operator/controllers.(*GrafanaDashboardReconciler).Reconcile
	github.com/grafana-operator/grafana-operator/controllers/dashboard_controller.go:216
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234

elamaran11 · 2023-06-02T18:13:35Z

@elamaran11 To fully validate this, maybe we should drop this in terraform? so we are sure the operator set it up correctly?

And this too?

@bonclay7 Dropping is not required both boards can coexist for now.

elamaran11 · 2023-06-02T18:16:59Z

So I went ahead and did it, I dropped the dashboard provided by Terraform but for some reason the operator can't renew my API key, maybe i'm doing something wrong?

1.6857285045139527e+09	INFO	found matching Grafana instances for datasource	{"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "GrafanaDatasource": {"name":"grafanadatasource-sample-amp","namespace":"grafana-operator"}, "namespace": "grafana-operator", "name": "grafanadatasource-sample-amp", "reconcileID": "72cfd60c-a255-44a1-bfbd-88b0cbc4f90c", "count": 1}
1.6857285045445938e+09	INFO	found matching Grafana instances for dashboard	{"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "GrafanaDashboard": {"name":"external-grafanadashboard-url","namespace":"grafana-operator"}, "namespace": "grafana-operator", "name": "external-grafanadashboard-url", "reconcileID": "5a07e7b0-d200-4223-94bc-a2fe6e6009db", "count": 1}
1.6857285045556655e+09	ERROR	error reconciling datasource	{"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "GrafanaDatasource": {"name":"grafanadatasource-sample-amp","namespace":"grafana-operator"}, "namespace": "grafana-operator", "name": "grafanadatasource-sample-amp", "reconcileID": "72cfd60c-a255-44a1-bfbd-88b0cbc4f90c", "datasource": "grafanadatasource-sample-amp", "grafana": "external-grafana", "error": "status: 401, body: {\"message\":\"Expired API key\"}\n"}
github.com/grafana-operator/grafana-operator/controllers.(*GrafanaDatasourceReconciler).Reconcile
	github.com/grafana-operator/grafana-operator/controllers/datasource_controller.go:209
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234
1.6857285046102674e+09	ERROR	error reconciling dashboard	{"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "GrafanaDashboard": {"name":"external-grafanadashboard-url","namespace":"grafana-operator"}, "namespace": "grafana-operator", "name": "external-grafanadashboard-url", "reconcileID": "5a07e7b0-d200-4223-94bc-a2fe6e6009db", "dashboard": "external-grafanadashboard-url", "grafana": "external-grafana", "error": "status: 401, body: {\"message\":\"Expired API key\"}\n"}
github.com/grafana-operator/grafana-operator/controllers.(*GrafanaDashboardReconciler).Reconcile
	github.com/grafana-operator/grafana-operator/controllers/dashboard_controller.go:216
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234

@bonclay7 The Grafana Key is in the Secrets Manager which is expired. Operator wont renew keys with AMG. The external secrets will take care of refreshing secrets manager key with k8s secret for grafana operator but operator has to take care of refreshing grafana keys with below :

export GO_AMG_API_KEY=$(aws grafana create-workspace-api-key \
  --key-name "grafana-operator-key" \
  --key-role "ADMIN" \
  --seconds-to-live 432000 \
  --workspace-id <YOUR_WORKSPACE_ID> \
  --query key \
  --output text)
  
export GO_API_KEY_SECRET_NAME=$(aws secretsmanager list-secrets \
    --query 'SecretList[1].Name' | tr -d '"')

aws secretsmanager update-secret \
    --secret-id $GO_API_KEY_SECRET_NAME \
    --secret-string "${GO_AMG_API_KEY}" \
    --region $GO_AWS_REGION

bonclay7 · 2023-06-02T18:27:22Z

Did I miss these instructions in the docs, or we are waiting for the full move to make it clear?

bonclay7 · 2023-06-02T18:30:41Z

Dropping is not required both boards can coexist for now

Yes, but for customers using latest, they would be quite confused that we have two data sources and duplicates dashboards. I would suggest that we make it more seamless

elamaran11 · 2023-06-02T18:44:44Z

Did I miss these instructions in the docs, or we are waiting for the full move to make it clear?

This is same as how you upgrade the Grafana Keys today. We can add these instructions after full move

elamaran11 · 2023-06-02T21:24:39Z

@bonclay7 Now i have all dashboards in the mix with Grafana Operator and commented out all dashboards in dashboard.tf. We are dependent on the GrafanaFolder feature from the ticket i previously mentioned and also dependent on this PR.

elamaran11 · 2023-06-03T13:30:34Z

@bonclay7 Grafana Folder works now with Grafana Operator. So i have migrated all the 6 dashboards for EKS Monitoring Module to Grafana Operator and Flux with GitOps. I think we can seamlessly move other pattern dashboards and cleanup as subsequent work in future PRs. Please take this forward and let me know if you have questions.

elamaran11 · 2023-06-03T15:56:35Z

Added an operational procedure as well under Troubleshooting on how to replace the expired Grafana API Keys in AWS Secrets Manager.

elamaran11 · 2023-06-03T15:59:00Z

modules/eks-monitoring/variables.tf

bonclay7

Awesome job @elamaran11. There is a few cleanup to do after this one, and docs updates

elamaran11 added 2 commits May 25, 2023 12:55

Grafana With GitOps Feature

73acbd9

Grafana With GitOps Feature

d5b912c

elamaran11 temporarily deployed to Observability Test May 25, 2023 18:06 — with GitHub Actions Inactive

elamaran11 requested review from bonclay7 and lewinkedrs May 25, 2023 18:06

Grafana With GitOps Feature

baff02e

elamaran11 temporarily deployed to Observability Test May 25, 2023 18:29 — with GitHub Actions Inactive

Merge branch 'main' into feature/gitopsWithGrafana

1b1263b

bonclay7 temporarily deployed to Observability Test May 31, 2023 15:37 — with GitHub Actions Inactive

Merge branch 'main' into feature/gitopsWithGrafana

a6e3566

bonclay7 temporarily deployed to Observability Test June 1, 2023 09:39 — with GitHub Actions Inactive

bonclay7 and others added 2 commits June 2, 2023 10:35

Fix setup logs retention policy (#169)

3b1a71b

Fixing GitOps Repo

13835ec

elamaran11 temporarily deployed to Observability Test June 2, 2023 14:35 — with GitHub Actions Inactive

Merge branch 'main' into feature/gitopsWithGrafana

2b86ee9

bonclay7 temporarily deployed to Observability Test June 2, 2023 15:15 — with GitHub Actions Inactive

Merge branch 'main' into feature/gitopsWithGrafana

28cc0ba

bonclay7 temporarily deployed to Observability Test June 2, 2023 17:46 — with GitHub Actions Inactive

elamaran11 added 2 commits June 2, 2023 14:43

Commenting out the NodeExp Dash

0d26b54

Commenting out the NodeExp Dash

1a43621

elamaran11 temporarily deployed to Observability Test June 2, 2023 18:43 — with GitHub Actions Inactive

Adding all Grafana Dashboards

8f78879

elamaran11 temporarily deployed to Observability Test June 2, 2023 21:22 — with GitHub Actions Inactive

Adding all Grafana Dashboards

e1569bf

elamaran11 temporarily deployed to Observability Test June 2, 2023 21:24 — with GitHub Actions Inactive

Fixing Grafana Operator Version and cleaning full boards

8d44533

elamaran11 temporarily deployed to Observability Test June 3, 2023 13:17 — with GitHub Actions Inactive

Fixing Grafana Operator Version and cleaning full boards

db631aa

elamaran11 temporarily deployed to Observability Test June 3, 2023 13:28 — with GitHub Actions Inactive

Fixing Grafana Operator Version and cleaning full boards

9d6ae4d

elamaran11 temporarily deployed to Observability Test June 3, 2023 13:43 — with GitHub Actions Inactive

Fixing Grafana Operator Version and cleaning full boards

e17d80c

elamaran11 temporarily deployed to Observability Test June 3, 2023 13:49 — with GitHub Actions Inactive

Fixing Grafana Operator Version and cleaning full boards and PR Issues

03d3b59

elamaran11 temporarily deployed to Observability Test June 3, 2023 15:53 — with GitHub Actions Inactive

Fixing Grafana Operator Version and cleaning full boards and PR Issues

f00aca7

elamaran11 temporarily deployed to Observability Test June 3, 2023 16:09 — with GitHub Actions Inactive

elamaran11 added 4 commits June 3, 2023 12:36

Fixing Grafana Operator Version and cleaning full boards and PR Issues

6839ecc

Fixing Grafana Operator Version and cleaning full boards and PR Issues

442594b

Fixing Grafana Operator Version and cleaning full boards and PR Issues

606c275

Fixing Grafana Operator Version and cleaning full boards and PR Issues

491cea1

elamaran11 temporarily deployed to Observability Test June 3, 2023 16:41 — with GitHub Actions Inactive

bonclay7 reviewed Jun 5, 2023

View reviewed changes

modules/eks-monitoring/variables.tf Show resolved Hide resolved

bonclay7 approved these changes Jun 5, 2023

View reviewed changes

bonclay7 merged commit 0392f5b into main Jun 5, 2023

bonclay7 deleted the feature/gitopsWithGrafana branch June 5, 2023 09:11

bonclay7 mentioned this pull request Jun 6, 2023

Move all dashboards to GitOps #175

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using GitOps with Grafana Operator to create Grafana Dashboard with Flux. #164

Using GitOps with Grafana Operator to create Grafana Dashboard with Flux. #164

elamaran11 commented May 25, 2023

bonclay7 commented May 30, 2023

elamaran11 commented May 30, 2023

bonclay7 commented Jun 2, 2023 •

edited

Loading

bonclay7 commented Jun 2, 2023

elamaran11 commented Jun 2, 2023

elamaran11 commented Jun 2, 2023 •

edited

Loading

bonclay7 commented Jun 2, 2023

bonclay7 commented Jun 2, 2023

elamaran11 commented Jun 2, 2023

elamaran11 commented Jun 2, 2023

elamaran11 commented Jun 3, 2023

elamaran11 commented Jun 3, 2023

elamaran11 commented Jun 3, 2023

bonclay7 left a comment

Using GitOps with Grafana Operator to create Grafana Dashboard with Flux. #164

Using GitOps with Grafana Operator to create Grafana Dashboard with Flux. #164

Conversation

elamaran11 commented May 25, 2023

What does this PR do?

Motivation

More

For Moderators

Additional Notes

bonclay7 commented May 30, 2023

elamaran11 commented May 30, 2023

bonclay7 commented Jun 2, 2023 • edited Loading

bonclay7 commented Jun 2, 2023

elamaran11 commented Jun 2, 2023

elamaran11 commented Jun 2, 2023 • edited Loading

bonclay7 commented Jun 2, 2023

bonclay7 commented Jun 2, 2023

elamaran11 commented Jun 2, 2023

elamaran11 commented Jun 2, 2023

elamaran11 commented Jun 3, 2023

elamaran11 commented Jun 3, 2023

elamaran11 commented Jun 3, 2023

bonclay7 left a comment

Choose a reason for hiding this comment

bonclay7 commented Jun 2, 2023 •

edited

Loading

elamaran11 commented Jun 2, 2023 •

edited

Loading