Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wip #12

Merged
merged 2 commits into from
Aug 13, 2024
Merged

Wip #12

merged 2 commits into from
Aug 13, 2024

Conversation

rkoosaar
Copy link
Owner

No description provided.

Copy link

--- kubernetes/apps/observability/blackbox-exporter/app Kustomization: flux-system/blackbox-exporter HelmRelease: observability/blackbox-exporter

+++ kubernetes/apps/observability/blackbox-exporter/app Kustomization: flux-system/blackbox-exporter HelmRelease: observability/blackbox-exporter

@@ -1,83 +0,0 @@

----
-apiVersion: helm.toolkit.fluxcd.io/v2
-kind: HelmRelease
-metadata:
-  labels:
-    app.kubernetes.io/name: blackbox-exporter
-    kustomize.toolkit.fluxcd.io/name: blackbox-exporter
-    kustomize.toolkit.fluxcd.io/namespace: flux-system
-  name: blackbox-exporter
-  namespace: observability
-spec:
-  chart:
-    spec:
-      chart: prometheus-blackbox-exporter
-      sourceRef:
-        kind: HelmRepository
-        name: prometheus-community
-        namespace: flux-system
-      version: 9.0.0
-  install:
-    remediation:
-      retries: 3
-  interval: 30m
-  upgrade:
-    cleanupOnFail: true
-    remediation:
-      retries: 3
-      strategy: rollback
-  values:
-    config:
-      modules:
-        http_2xx:
-          http:
-            follow_redirects: true
-            preferred_ip_protocol: ipv4
-            valid_http_versions:
-            - HTTP/1.1
-            - HTTP/2.0
-          prober: http
-          timeout: 5s
-        icmp:
-          icmp:
-            preferred_ip_protocol: ipv4
-          prober: icmp
-          timeout: 5s
-        tcp_connect:
-          prober: tcp
-          tcp:
-            preferred_ip_protocol: ipv4
-          timeout: 5s
-    fullnameOverride: blackbox-exporter
-    ingress:
-      annotations:
-        external-dns.alpha.kubernetes.io/target: internal...PLACEHOLDER..
-      className: internal
-      enabled: true
-      hosts:
-      - host: blackbox-exporter...PLACEHOLDER..
-        paths:
-        - path: /
-          pathType: Prefix
-    prometheusRule:
-      enabled: true
-      rules:
-      - alert: BlackboxProbeFailed
-        annotations:
-          summary: The host {{ $labels.target }} is currently unreachable
-        expr: probe_success == 0
-        for: 15m
-        labels:
-          severity: critical
-    securityContext:
-      allowPrivilegeEscalation: false
-      capabilities:
-        add:
-        - NET_RAW
-      readOnlyRootFilesystem: true
-    serviceMonitor:
-      defaults:
-        interval: 1m
-        scrapeTimeout: 10s
-      enabled: true
-
--- kubernetes/apps Kustomization: flux-system/cluster-apps Kustomization: flux-system/blackbox-exporter

+++ kubernetes/apps Kustomization: flux-system/cluster-apps Kustomization: flux-system/blackbox-exporter

@@ -1,40 +0,0 @@

----
-apiVersion: kustomize.toolkit.fluxcd.io/v1
-kind: Kustomization
-metadata:
-  labels:
-    kustomize.toolkit.fluxcd.io/name: cluster-apps
-    kustomize.toolkit.fluxcd.io/namespace: flux-system
-  name: blackbox-exporter
-  namespace: flux-system
-spec:
-  commonMetadata:
-    labels:
-      app.kubernetes.io/name: blackbox-exporter
-  decryption:
-    provider: sops
-    secretRef:
-      name: sops-age
-  interval: 30m
-  path: ./kubernetes/apps/observability/blackbox-exporter/app
-  postBuild:
-    substituteFrom:
-    - kind: ConfigMap
-      name: cluster-settings
-    - kind: Secret
-      name: cluster-secrets
-    - kind: ConfigMap
-      name: cluster-user-settings
-      optional: true
-    - kind: Secret
-      name: cluster-user-secrets
-      optional: true
-  prune: true
-  retryInterval: 1m
-  sourceRef:
-    kind: GitRepository
-    name: home-kubernetes
-  targetNamespace: observability
-  timeout: 15m
-  wait: false
-
--- kubernetes/apps Kustomization: flux-system/cluster-apps Kustomization: flux-system/kube-prometheus-stack

+++ kubernetes/apps Kustomization: flux-system/cluster-apps Kustomization: flux-system/kube-prometheus-stack

@@ -0,0 +1,42 @@

+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  labels:
+    kustomize.toolkit.fluxcd.io/name: cluster-apps
+    kustomize.toolkit.fluxcd.io/namespace: flux-system
+  name: kube-prometheus-stack
+  namespace: flux-system
+spec:
+  commonMetadata:
+    labels:
+      app.kubernetes.io/name: kube-prometheus-stack
+  decryption:
+    provider: sops
+    secretRef:
+      name: sops-age
+  dependsOn:
+  - name: external-secrets-stores
+  interval: 30m
+  path: ./kubernetes/apps/observability/kube-prometheus-stack/app
+  postBuild:
+    substituteFrom:
+    - kind: ConfigMap
+      name: cluster-settings
+    - kind: Secret
+      name: cluster-secrets
+    - kind: ConfigMap
+      name: cluster-user-settings
+      optional: true
+    - kind: Secret
+      name: cluster-user-secrets
+      optional: true
+  prune: true
+  retryInterval: 1m
+  sourceRef:
+    kind: GitRepository
+    name: home-kubernetes
+  targetNamespace: observability
+  timeout: 15m
+  wait: false
+
--- kubernetes/apps Kustomization: flux-system/cluster-apps Kustomization: flux-system/loki

+++ kubernetes/apps Kustomization: flux-system/cluster-apps Kustomization: flux-system/loki

@@ -0,0 +1,40 @@

+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  labels:
+    kustomize.toolkit.fluxcd.io/name: cluster-apps
+    kustomize.toolkit.fluxcd.io/namespace: flux-system
+  name: loki
+  namespace: flux-system
+spec:
+  commonMetadata:
+    labels:
+      app.kubernetes.io/name: loki
+  decryption:
+    provider: sops
+    secretRef:
+      name: sops-age
+  interval: 30m
+  path: ./kubernetes/apps/observability/loki/app
+  postBuild:
+    substituteFrom:
+    - kind: ConfigMap
+      name: cluster-settings
+    - kind: Secret
+      name: cluster-secrets
+    - kind: ConfigMap
+      name: cluster-user-settings
+      optional: true
+    - kind: Secret
+      name: cluster-user-secrets
+      optional: true
+  prune: true
+  retryInterval: 1m
+  sourceRef:
+    kind: GitRepository
+    name: home-kubernetes
+  targetNamespace: observability
+  timeout: 15m
+  wait: false
+
--- kubernetes/apps Kustomization: flux-system/cluster-apps Kustomization: flux-system/prometheus-operator-crds

+++ kubernetes/apps Kustomization: flux-system/cluster-apps Kustomization: flux-system/prometheus-operator-crds

@@ -0,0 +1,40 @@

+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  labels:
+    kustomize.toolkit.fluxcd.io/name: cluster-apps
+    kustomize.toolkit.fluxcd.io/namespace: flux-system
+  name: prometheus-operator-crds
+  namespace: flux-system
+spec:
+  commonMetadata:
+    labels:
+      app.kubernetes.io/name: prometheus-operator-crds
+  decryption:
+    provider: sops
+    secretRef:
+      name: sops-age
+  interval: 30m
+  path: ./kubernetes/apps/observability/prometheus-operator-crds/app
+  postBuild:
+    substituteFrom:
+    - kind: ConfigMap
+      name: cluster-settings
+    - kind: Secret
+      name: cluster-secrets
+    - kind: ConfigMap
+      name: cluster-user-settings
+      optional: true
+    - kind: Secret
+      name: cluster-user-secrets
+      optional: true
+  prune: false
+  retryInterval: 1m
+  sourceRef:
+    kind: GitRepository
+    name: home-kubernetes
+  targetNamespace: observability
+  timeout: 5m
+  wait: false
+
--- kubernetes/apps Kustomization: flux-system/cluster-apps Kustomization: flux-system/promtail

+++ kubernetes/apps Kustomization: flux-system/cluster-apps Kustomization: flux-system/promtail

@@ -0,0 +1,40 @@

+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  labels:
+    kustomize.toolkit.fluxcd.io/name: cluster-apps
+    kustomize.toolkit.fluxcd.io/namespace: flux-system
+  name: promtail
+  namespace: flux-system
+spec:
+  commonMetadata:
+    labels:
+      app.kubernetes.io/name: promtail
+  decryption:
+    provider: sops
+    secretRef:
+      name: sops-age
+  interval: 30m
+  path: ./kubernetes/apps/observability/promtail/app
+  postBuild:
+    substituteFrom:
+    - kind: ConfigMap
+      name: cluster-settings
+    - kind: Secret
+      name: cluster-secrets
+    - kind: ConfigMap
+      name: cluster-user-settings
+      optional: true
+    - kind: Secret
+      name: cluster-user-secrets
+      optional: true
+  prune: true
+  retryInterval: 1m
+  sourceRef:
+    kind: GitRepository
+    name: home-kubernetes
+  targetNamespace: observability
+  timeout: 5m
+  wait: false
+
--- kubernetes/apps/observability/prometheus-operator-crds/app Kustomization: flux-system/prometheus-operator-crds HelmRelease: observability/prometheus-operator-crds

+++ kubernetes/apps/observability/prometheus-operator-crds/app Kustomization: flux-system/prometheus-operator-crds HelmRelease: observability/prometheus-operator-crds

@@ -0,0 +1,28 @@

+---
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  labels:
+    app.kubernetes.io/name: prometheus-operator-crds
+    kustomize.toolkit.fluxcd.io/name: prometheus-operator-crds
+    kustomize.toolkit.fluxcd.io/namespace: flux-system
+  name: prometheus-operator-crds
+  namespace: observability
+spec:
+  chart:
+    spec:
+      chart: prometheus-operator-crds
+      sourceRef:
+        kind: HelmRepository
+        name: prometheus-community
+        namespace: flux-system
+      version: 13.0.2
+  install:
+    remediation:
+      retries: 3
+  interval: 30m
+  upgrade:
+    cleanupOnFail: true
+    remediation:
+      retries: 3
+
--- kubernetes/apps/observability/promtail/app Kustomization: flux-system/promtail HelmRelease: observability/promtail

+++ kubernetes/apps/observability/promtail/app Kustomization: flux-system/promtail HelmRelease: observability/promtail

@@ -0,0 +1,35 @@

+---
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  labels:
+    app.kubernetes.io/name: promtail
+    kustomize.toolkit.fluxcd.io/name: promtail
+    kustomize.toolkit.fluxcd.io/namespace: flux-system
+  name: promtail
+  namespace: observability
+spec:
+  chart:
+    spec:
+      chart: promtail
+      sourceRef:
+        kind: HelmRepository
+        name: grafana
+        namespace: flux-system
+      version: 6.16.4
+  install:
+    remediation:
+      retries: 3
+  interval: 30m
+  upgrade:
+    cleanupOnFail: true
+    remediation:
+      retries: 3
+  values:
+    config:
+      clients:
+      - url: http://loki-headless.observability.svc.cluster.local:3100/loki/api/v1/push
+    fullnameOverride: promtail
+    serviceMonitor:
+      enabled: true
+
--- kubernetes/apps/observability/loki/app Kustomization: flux-system/loki HelmRelease: observability/loki

+++ kubernetes/apps/observability/loki/app Kustomization: flux-system/loki HelmRelease: observability/loki

@@ -0,0 +1,82 @@

+---
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  labels:
+    app.kubernetes.io/name: loki
+    kustomize.toolkit.fluxcd.io/name: loki
+    kustomize.toolkit.fluxcd.io/namespace: flux-system
+  name: loki
+  namespace: observability
+spec:
+  chart:
+    spec:
+      chart: loki
+      sourceRef:
+        kind: HelmRepository
+        name: grafana
+        namespace: flux-system
+      version: 6.10.0
+  dependsOn:
+  - name: openebs
+    namespace: openebs-system
+  install:
+    crds: Skip
+    remediation:
+      retries: 3
+  interval: 30m
+  timeout: 15m
+  upgrade:
+    cleanupOnFail: true
+    crds: Skip
+    remediation:
+      retries: 3
+      strategy: rollback
+  values:
+    backend:
+      replicas: 0
+    chunksCache:
+      enabled: false
+    deploymentMode: SingleBinary
+    gateway:
+      replicas: 0
+    loki:
+      analytics:
+        reporting_enabled: false
+      auth_enabled: false
+      commonConfig:
+        replication_factor: 1
+      ingester:
+        chunk_encoding: snappy
+      limits_config:
+        retention_period: 14d
+      schemaConfig:
+        configs:
+        - from: '2024-04-01'
+          index:
+            period: 24h
+            prefix: loki_index_
+          object_store: filesystem
+          schema: v13
+          store: tsdb
+      server:
+        log_level: info
+      storage:
+        type: filesystem
+    lokiCanary:
+      enabled: false
+    read:
+      replicas: 0
+    resultsCache:
+      enabled: false
+    singleBinary:
+      persistence:
+        enabled: true
+        size: 50Gi
+        storageClass: openebs-hostpath
+      replicas: 1
+    test:
+      enabled: false
+    write:
+      replicas: 0
+
--- kubernetes/apps/observability/kube-prometheus-stack/app Kustomization: flux-system/kube-prometheus-stack ExternalSecret: observability/alertmanager

+++ kubernetes/apps/observability/kube-prometheus-stack/app Kustomization: flux-system/kube-prometheus-stack ExternalSecret: observability/alertmanager

@@ -0,0 +1,94 @@

+---
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  labels:
+    app.kubernetes.io/name: kube-prometheus-stack
+    kustomize.toolkit.fluxcd.io/name: kube-prometheus-stack
+    kustomize.toolkit.fluxcd.io/namespace: flux-system
+  name: alertmanager
+  namespace: observability
+spec:
+  dataFrom:
+  - extract:
+      key: pushover
+  - extract:
+      key: alertmanager
+  refreshInterval: 5m
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: onepassword-connect
+  target:
+    name: alertmanager-secret
+    template:
+      data:
+        alertmanager.yaml: |
+          global:
+            resolve_timeout: 5m
+          route:
+            group_by: ["alertname", "job"]
+            group_interval: 10m
+            group_wait: 1m
+            receiver: pushover
+            repeat_interval: 12h
+            routes:
+              - receiver: heartbeat
+                group_interval: 5m
+                group_wait: 0s
+                matchers:
+                  - alertname =~ "Watchdog"
+                repeat_interval: 5m
+              - receiver: "null"
+                matchers:
+                  - alertname =~ "InfoInhibitor"
+              - receiver: pushover
+                continue: true
+                matchers:
+                  - severity = "critical"
+          inhibit_rules:
+            - equal: ["alertname", "namespace"]
+              source_matchers:
+                - severity = "critical"
+              target_matchers:
+                - severity = "warning"
+          receivers:
+            - name: heartbeat
+              webhook_configs:
+                - send_resolved: true
+                  url: "{{ .ALERTMANAGER_CLUSTER_MAIN_HEARTBEAT_URL }}"
+            - name: "null"
+            - name: pushover
+              pushover_configs:
+                - html: true
+                  message: |-
+                    {{ "{{-" }} range .Alerts {{ "}}" }}
+                      {{ "{{-" }} if ne .Annotations.description "" {{ "}}" }}
+                        {{ "{{" }} .Annotations.description {{ "}}" }}
+                      {{ "{{-" }} else if ne .Annotations.summary "" {{ "}}" }}
+                        {{ "{{" }} .Annotations.summary {{ "}}" }}
+                      {{ "{{-" }} else if ne .Annotations.message "" {{ "}}" }}
+                        {{ "{{" }} .Annotations.message {{ "}}" }}
+                      {{ "{{-" }} else {{ "}}" }}
+                        Alert description not available
+                      {{ "{{-" }} end {{ "}}" }}
+                      {{ "{{-" }} if gt (len .Labels.SortedPairs) 0 {{ "}}" }}
+                        <small>
+                        {{ "{{-" }} range .Labels.SortedPairs {{ "}}" }}
+                          <b>{{ "{{" }} .Name {{ "}}" }}:</b> {{ "{{" }} .Value {{ "}}" }}
+                        {{ "{{-" }} end {{ "}}" }}
+                        </small>
+                      {{ "{{-" }} end {{ "}}" }}
+                    {{ "{{-" }} end {{ "}}" }}
+                  priority: |-
+                    {{ "{{" }} if eq .Status "firing" {{ "}}" }}1{{ "{{" }} else {{ "}}" }}0{{ "{{" }} end {{ "}}" }}
+                  send_resolved: true
+                  sound: gamelan
+                  # ttl: 1d
+                  title: >-
+                    [{{ "{{" }} .Status | toUpper {{ "}}" }}{{ "{{" }} if eq .Status "firing" {{ "}}" }}:{{ "{{" }} .Alerts.Firing | len {{ "}}" }}{{ "{{" }} end {{ "}}" }}]
+                    {{ "{{" }} .CommonLabels.alertname {{ "}}" }}
+                  token: "{{ .ALERTMANAGER_PUSHOVER_TOKEN }}"
+                  url_title: View in Alertmanager
+                  user_key: "{{ .PUSHOVER_USER_KEY }}"
+      engineVersion: v2
+
--- kubernetes/apps/observability/kube-prometheus-stack/app Kustomization: flux-system/kube-prometheus-stack HelmRelease: observability/kube-prometheus-stack

+++ kubernetes/apps/observability/kube-prometheus-stack/app Kustomization: flux-system/kube-prometheus-stack HelmRelease: observability/kube-prometheus-stack

@@ -0,0 +1,145 @@

+---
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  labels:
+    app.kubernetes.io/name: kube-prometheus-stack
+    kustomize.toolkit.fluxcd.io/name: kube-prometheus-stack
+    kustomize.toolkit.fluxcd.io/namespace: flux-system
+  name: kube-prometheus-stack
+  namespace: observability
+spec:
+  chart:
+    spec:
+      chart: kube-prometheus-stack
+      sourceRef:
+        kind: HelmRepository
+        name: prometheus-community
+        namespace: flux-system
+      version: 61.8.0
+  dependsOn:
+  - name: prometheus-operator-crds
+    namespace: observability
+  - name: openebs
+    namespace: openebs-system
+  install:
+    crds: Skip
+    remediation:
+      retries: 3
+  interval: 30m
+  timeout: 15m
+  upgrade:
+    cleanupOnFail: true
+    crds: Skip
+    remediation:
+      retries: 3
+      strategy: rollback
+  values:
+    alertmanager:
+      alertmanagerSpec:
+        configSecret: alertmanager-secret
+        storage:
+          volumeClaimTemplate:
+            spec:
+              resources:
+                requests:
+                  storage: 1Gi
+              storageClassName: openebs-hostpath
+        useExistingSecret: true
+      ingress:
+        annotations:
+          external-dns.alpha.kubernetes.io/target: internal...PLACEHOLDER..
+        enabled: true
+        hosts:
+        - alertmanager...PLACEHOLDER..
+        ingressClassName: internal
+        pathType: Prefix
+    cleanPrometheusOperatorObjectNames: true
+    crds:
+      enabled: false
+    grafana:
+      enabled: false
+      forceDeployDashboards: true
+    kube-state-metrics:
+      fullnameOverride: kube-state-metrics
+      metricLabelsAllowlist:
+      - pods=[*]
+      - deployments=[*]
+      - persistentvolumeclaims=[*]
+      prometheus:
+        monitor:
+          enabled: true
+          relabelings:
+          - action: replace
+            regex: (.*)
+            replacement: $1
+            sourceLabels:
+            - __meta_kubernetes_pod_node_name
+            targetLabel: kubernetes_node
+    kubeApiServer:
+      serviceMonitor:
+        selector:
+          k8s-app: kube-apiserver
+    kubeControllerManager:
+      service:
+        selector:
+          k8s-app: kube-controller-manager
+    kubeEtcd:
+      service:
+        selector:
+          k8s-app: kube-controller-manager
+    kubeProxy:
+      enabled: false
+    kubeScheduler:
+      service:
+        selector:
+          k8s-app: kube-scheduler
+    prometheus:
+      ingress:
+        annotations:
+          external-dns.alpha.kubernetes.io/target: internal...PLACEHOLDER..
+        enabled: true
+        hosts:
+        - prometheus...PLACEHOLDER..
+        ingressClassName: internal
+        pathType: Prefix
+      prometheusSpec:
+        enableAdminAPI: true
+        enableFeatures:
+        - auto-gomemlimit
+        - memory-snapshot-on-shutdown
+        - new-service-discovery-manager
+        podMonitorSelectorNilUsesHelmValues: false
+        probeSelectorNilUsesHelmValues: false
+        resources:
+          limits:
+            memory: 1500Mi
+          requests:
+            cpu: 100m
+        retention: 14d
+        retentionSize: 70GB
+        ruleSelectorNilUsesHelmValues: false
+        scrapeConfigSelectorNilUsesHelmValues: false
+        scrapeInterval: 1m
+        serviceMonitorSelectorNilUsesHelmValues: false
+        storageSpec:
+          volumeClaimTemplate:
+            spec:
+              resources:
+                requests:
+                  storage: 75Gi
+              storageClassName: openebs-hostpath
+        walCompression: true
+    prometheus-node-exporter:
+      fullnameOverride: node-exporter
+      prometheus:
+        monitor:
+          enabled: true
+          relabelings:
+          - action: replace
+            regex: (.*)
+            replacement: $1
+            sourceLabels:
+            - __meta_kubernetes_pod_node_name
+            targetLabel: kubernetes_node
+
--- kubernetes/apps/observability/kube-prometheus-stack/app Kustomization: flux-system/kube-prometheus-stack PrometheusRule: observability/miscellaneous-rules

+++ kubernetes/apps/observability/kube-prometheus-stack/app Kustomization: flux-system/kube-prometheus-stack PrometheusRule: observability/miscellaneous-rules

@@ -0,0 +1,45 @@

+---
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  labels:
+    app.kubernetes.io/name: kube-prometheus-stack
+    kustomize.toolkit.fluxcd.io/name: kube-prometheus-stack
+    kustomize.toolkit.fluxcd.io/namespace: flux-system
+  name: miscellaneous-rules
+  namespace: observability
+spec:
+  groups:
+  - name: dockerhub
+    rules:
+    - alert: BootstrapRateLimitRisk
+      annotations:
+        summary: Kubernetes cluster at risk of being rate limited by dockerhub on
+          bootstrap
+      expr: count(time() - container_last_seen{image=~"(docker.io).*",container!=""}
+        < 30) > 100
+      for: 15m
+      labels:
+        severity: critical
+  - name: oom
+    rules:
+    - alert: OOMKilled
+      annotations:
+        summary: Container {{ $labels.container }} in pod {{ $labels.namespace }}/{{
+          $labels.pod }} has been OOMKilled {{ $value }} times in the last 10 minutes.
+      expr: (kube_pod_container_status_restarts_total - kube_pod_container_status_restarts_total
+        offset 10m >= 1) and ignoring (reason) min_over_time(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[10m])
+        == 1
+      labels:
+        severity: critical
+  - name: zfs
+    rules:
+    - alert: ZfsUnexpectedPoolState
+      annotations:
+        summary: ZFS pool {{$labels.zpool}} on {{$labels.instance}} is in a unexpected
+          state {{$labels.state}}
+      expr: node_zfs_zpool_state{state!="online"} > 0
+      for: 15m
+      labels:
+        severity: critical
+
--- kubernetes/apps/observability/kube-prometheus-stack/app Kustomization: flux-system/kube-prometheus-stack ScrapeConfig: observability/minio

+++ kubernetes/apps/observability/kube-prometheus-stack/app Kustomization: flux-system/kube-prometheus-stack ScrapeConfig: observability/minio

@@ -0,0 +1,21 @@

+---
+apiVersion: monitoring.coreos.com/v1alpha1
+kind: ScrapeConfig
+metadata:
+  labels:
+    app.kubernetes.io/name: kube-prometheus-stack
+    kustomize.toolkit.fluxcd.io/name: kube-prometheus-stack
+    kustomize.toolkit.fluxcd.io/namespace: flux-system
+  name: minio
+  namespace: observability
+spec:
+  metricsPath: /minio/v2/metrics/cluster
+  relabelings:
+  - action: replace
+    replacement: minio
+    targetLabel: job
+  scheme: HTTPS
+  staticConfigs:
+  - targets:
+    - s3.kvshs.xyz
+
--- kubernetes/apps/observability/kube-prometheus-stack/app Kustomization: flux-system/kube-prometheus-stack ScrapeConfig: observability/node-exporter

+++ kubernetes/apps/observability/kube-prometheus-stack/app Kustomization: flux-system/kube-prometheus-stack ScrapeConfig: observability/node-exporter

@@ -0,0 +1,20 @@

+---
+apiVersion: monitoring.coreos.com/v1alpha1
+kind: ScrapeConfig
+metadata:
+  labels:
+    app.kubernetes.io/name: kube-prometheus-stack
+    kustomize.toolkit.fluxcd.io/name: kube-prometheus-stack
+    kustomize.toolkit.fluxcd.io/namespace: flux-system
+  name: node-exporter
+  namespace: observability
+spec:
+  metricsPath: /metrics
+  relabelings:
+  - action: replace
+    replacement: node-exporter
+    targetLabel: job
+  staticConfigs:
+  - targets:
+    - nas.kvshs.xyz:9100
+

@rkoosaar rkoosaar merged commit 84ff77e into main Aug 13, 2024
4 of 5 checks passed
rkoosaar added a commit that referenced this pull request Aug 22, 2024
rkoosaar added a commit that referenced this pull request Aug 26, 2024
rkoosaar added a commit that referenced this pull request Aug 27, 2024
rkoosaar added a commit that referenced this pull request Aug 28, 2024
rkoosaar added a commit that referenced this pull request Aug 31, 2024
rkoosaar added a commit that referenced this pull request Aug 31, 2024
rkoosaar added a commit that referenced this pull request Sep 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant