kustomize build failed: map[string]interface {}(nil): yaml: unmarshal errors #880

egazzarr · 2023-06-07T12:57:25Z

Hi all,

I have connected flux to a healthy K8s cluster running on v1.22.9 and am trying to bootstrap to a repository containing the necessary files to manage the cluster with flux. I have flux version 2.0.0-rc.5.
The command I run is:

flux bootstrap github --owner=vre-hub --repository=vre --branch=main --path=infrastructure/cluster/flux-v2 --author-name flux-ops

The command returns:

► connecting to github.com
► cloning branch "main" from Git repository "https://github.com/vre-hub/vre.git"
✔ cloned repository
► generating component manifests
# Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
✔ generated component manifests
✔ committed sync manifests to "main" ("f7d788fe232981f2661febb80beccd87ef088464")
► pushing component manifests to "https://github.com/vre-hub/vre.git"
► installing components in "flux-system" namespace
✔ installed components
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
► generating source secret
✔ public key: ..
✔ configured deploy key "flux-system-main-flux-system-./infrastructure/cluster/flux-v2" for "https://github.com/vre-hub/vre"
► applying source secret "flux-system/flux-system"
✔ reconciled source secret
► generating sync manifests
✔ generated sync manifests
✔ committed sync manifests to "main" ("df72cb9620ec39fef595047962844243b3f92761")
► pushing sync manifests to "https://github.com/vre-hub/vre.git"
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for Kustomization "flux-system/flux-system" to be reconciled
✗ client rate limiter Wait returned an error: context deadline exceeded
► confirming components are healthy
✔ helm-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy
✗ bootstrap failed with 1 health check failure(s)

A flux check:

► checking prerequisites
✔ Kubernetes 1.22.9 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.34.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.0.0-rc.4
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.0.0-rc.4
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.0.0-rc.5
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta2
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta2
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

And the kustomize-controller pod returns an error that I can get by running kubectl describe kustomization flux-system -n flux-system

Name:         flux-system
Namespace:    flux-system
Labels:       <none>
Annotations:  <none>
API Version:  kustomize.toolkit.fluxcd.io/v1
Kind:         Kustomization
Metadata:
  Creation Timestamp:  2023-06-07T11:57:18Z
  Finalizers:
    finalizers.fluxcd.io
  Generation:  1
  Managed Fields:
    API Version:  kustomize.toolkit.fluxcd.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:interval:
        f:path:
        f:prune:
        f:sourceRef:
          f:kind:
          f:name:
    Manager:      flux
    Operation:    Apply
    Time:         2023-06-07T11:57:18Z
    API Version:  kustomize.toolkit.fluxcd.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"finalizers.fluxcd.io":
    Manager:      gotk-kustomize-controller
    Operation:    Update
    Time:         2023-06-07T11:57:18Z
    API Version:  kustomize.toolkit.fluxcd.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
        f:lastAttemptedRevision:
    Manager:         gotk-kustomize-controller
    Operation:       Update
    Subresource:     status
    Time:            2023-06-07T11:57:21Z
  Resource Version:  107915037
  Self Link:         /apis/kustomize.toolkit.fluxcd.io/v1/namespaces/flux-system/kustomizations/flux-system
  UID:               7933d591-bdd4-4a7e-9289-007b5358a205
Spec:
  Force:     false
  Interval:  10m0s
  Path:      ./infrastructure/cluster/flux-v2
  Prune:     true
  Source Ref:
    Kind:  GitRepository
    Name:  flux-system
Status:
  Conditions:
    Last Transition Time:  2023-06-07T12:47:21Z
    Message:               Building manifests for revision main@sha1:ebac5c5ee8baf43b64038ce888e6a4567f116805 with a timeout of 9m30s
    Observed Generation:   1
    Reason:                ProgressingWithRetry
    Status:                True
    Type:                  Reconciling
    Last Transition Time:  2023-06-07T12:47:21Z
    Message:               kustomize build failed: map[string]interface {}(nil): yaml: unmarshal errors:
  line 88: mapping key "cpu" already defined at line 85
  line 89: mapping key "memory" already defined at line 86
    Observed Generation:    1
    Reason:                 BuildFailed
    Status:                 False
    Type:                   Ready
  Last Attempted Revision:  main@sha1:ebac5c5ee8baf43b64038ce888e6a4567f116805
  Observed Generation:      -1
Events:
  Type     Reason       Age                From                  Message
  ----     ------       ----               ----                  -------
  Warning  BuildFailed  52s (x7 over 50m)  kustomize-controller  kustomize build failed: map[string]interface {}(nil): yaml: unmarshal errors:
  line 88: mapping key "cpu" already defined at line 85
  line 89: mapping key "memory" already defined at line 86
(rucio-admin) root@egazzarr-rucio:~/clusters/vre-cluster/vre-hub/vre/infrastructure/cluster/flux-v2# kubectl describe kustomization flux-system -n flux-system
Name:         flux-system
Namespace:    flux-system
Labels:       <none>
Annotations:  <none>
API Version:  kustomize.toolkit.fluxcd.io/v1
Kind:         Kustomization
Metadata:
  Creation Timestamp:  2023-06-07T11:57:18Z
  Finalizers:
    finalizers.fluxcd.io
  Generation:  1
  Managed Fields:
    API Version:  kustomize.toolkit.fluxcd.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:interval:
        f:path:
        f:prune:
        f:sourceRef:
          f:kind:
          f:name:
    Manager:      flux
    Operation:    Apply
    Time:         2023-06-07T11:57:18Z
    API Version:  kustomize.toolkit.fluxcd.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"finalizers.fluxcd.io":
    Manager:      gotk-kustomize-controller
    Operation:    Update
    Time:         2023-06-07T11:57:18Z
    API Version:  kustomize.toolkit.fluxcd.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
        f:lastAttemptedRevision:
    Manager:         gotk-kustomize-controller
    Operation:       Update
    Subresource:     status
    Time:            2023-06-07T11:57:21Z
  Resource Version:  107915037
  Self Link:         /apis/kustomize.toolkit.fluxcd.io/v1/namespaces/flux-system/kustomizations/flux-system
  UID:               7933d591-bdd4-4a7e-9289-007b5358a205
Spec:
  Force:     false
  Interval:  10m0s
  Path:      ./infrastructure/cluster/flux-v2
  Prune:     true
  Source Ref:
    Kind:  GitRepository
    Name:  flux-system
Status:
  Conditions:
    Last Transition Time:  2023-06-07T12:47:21Z
    Message:               Building manifests for revision main@sha1:ebac5c5ee8baf43b64038ce888e6a4567f116805 with a timeout of 9m30s
    Observed Generation:   1
    Reason:                ProgressingWithRetry
    Status:                True
    Type:                  Reconciling
    Last Transition Time:  2023-06-07T12:47:21Z
    Message:               kustomize build failed: map[string]interface {}(nil): yaml: unmarshal errors:
  line 88: mapping key "cpu" already defined at line 85
  line 89: mapping key "memory" already defined at line 86
    Observed Generation:    1
    Reason:                 BuildFailed
    Status:                 False
    Type:                   Ready
  Last Attempted Revision:  main@sha1:ebac5c5ee8baf43b64038ce888e6a4567f116805
  Observed Generation:      -1
Events:
  Type     Reason       Age                 From                  Message
  ----     ------       ----                ----                  -------
  Warning  BuildFailed  3m9s (x7 over 53m)  kustomize-controller  kustomize build failed: map[string]interface {}(nil): yaml: unmarshal errors:
  line 88: mapping key "cpu" already defined at line 85
  line 89: mapping key "memory" already defined at line 86

I can not find the file where these values are specified twice... Can someone help on this please?

Thanks in advance!

The text was updated successfully, but these errors were encountered:

makkes · 2023-06-07T12:59:19Z

First thing you could try is running kustomize build . in the infrastructure/cluster/flux-v2 directory.

egazzarr · 2023-06-07T13:15:05Z

Thanks for the quick reply. I run kubectl apply -k flux-v2/flux-system/. and nothing changes in the deployment.
Should I first delete all of the below resources?

namespace/flux-system unchanged
resourcequota/critical-pods-flux-system configured
customresourcedefinition.apiextensions.k8s.io/alerts.notification.toolkit.fluxcd.io unchanged
customresourcedefinition.apiextensions.k8s.io/buckets.source.toolkit.fluxcd.io unchanged
customresourcedefinition.apiextensions.k8s.io/gitrepositories.source.toolkit.fluxcd.io unchanged
customresourcedefinition.apiextensions.k8s.io/helmcharts.source.toolkit.fluxcd.io unchanged
customresourcedefinition.apiextensions.k8s.io/helmreleases.helm.toolkit.fluxcd.io unchanged
customresourcedefinition.apiextensions.k8s.io/helmrepositories.source.toolkit.fluxcd.io unchanged
customresourcedefinition.apiextensions.k8s.io/kustomizations.kustomize.toolkit.fluxcd.io unchanged
customresourcedefinition.apiextensions.k8s.io/ocirepositories.source.toolkit.fluxcd.io unchanged
customresourcedefinition.apiextensions.k8s.io/providers.notification.toolkit.fluxcd.io unchanged
customresourcedefinition.apiextensions.k8s.io/receivers.notification.toolkit.fluxcd.io unchanged
serviceaccount/helm-controller unchanged
serviceaccount/kustomize-controller unchanged
serviceaccount/notification-controller unchanged
serviceaccount/source-controller unchanged
clusterrole.rbac.authorization.k8s.io/crd-controller-flux-system unchanged
clusterrole.rbac.authorization.k8s.io/flux-edit-flux-system unchanged
clusterrole.rbac.authorization.k8s.io/flux-view-flux-system unchanged
clusterrolebinding.rbac.authorization.k8s.io/cluster-reconciler-flux-system unchanged
clusterrolebinding.rbac.authorization.k8s.io/crd-controller-flux-system unchanged
service/notification-controller unchanged
service/source-controller unchanged
service/webhook-receiver unchanged
deployment.apps/helm-controller configured
deployment.apps/kustomize-controller configured
deployment.apps/notification-controller configured
deployment.apps/source-controller configured
kustomization.kustomize.toolkit.fluxcd.io/flux-system configured
networkpolicy.networking.k8s.io/allow-egress unchanged
networkpolicy.networking.k8s.io/allow-scraping unchanged
networkpolicy.networking.k8s.io/allow-webhooks unchanged
gitrepository.source.toolkit.fluxcd.io/flux-system configured

This is how my kustomization look slike.

somtochiama · 2023-06-08T11:12:15Z

@egazzarr have you tried running kustomize build . in the directory containing the manifests as @makkes recommended?

egazzarr · 2023-06-08T14:05:12Z

Yes and nothing changed at all.

I also tried, after the bootstrap to the git repository, to:

delete the flux-system Kustomization and the kustomize-controller deployment manually
took away the lines which were supposedly duplicated (here)
ran kustomize edit fix --vars with kustomize v5.0.3
ran from the directory containing the manifests kubectl apply -k ./ after following this.

Nothing changes, the Kustomization gets regenerated, but the error persists. This should not happen on a normal flux bootstrap command as the whole kustomization procedure should be done behind the scenes, without the user having to debug this so deeply...

somtochiama · 2023-06-08T14:14:20Z

Please take a look at https://github.com/vre-hub/vre/blob/f6dcd1a9fd1095885465d88d8cbbef4902f1455d/infrastructure/cluster/flux-v2/rucio-vre/rucio-servers.yaml#L97

I see duplicate keys in the map.

egazzarr · 2023-06-08T14:30:29Z

Thanks @somtochiama! clearly did not understand enough about the Kustomization, as I was sure the error arose from the kustomize-controller deployment. Any way the error message could say a bit more which file causes the error?

How do I then overwrite the values.yaml of a remote HelmRepository?

somtochiama · 2023-06-08T14:36:48Z

Any way the error message could say a bit more which file causes the error?

The error is from kustomize library that we use, so there's little we can do.

How do I then overwrite the values.yaml of a remote HelmRepository?

You have an indention error, Fixing it is all you need to do. I guess this is what it should actually be:

    resources:
      limits:
        cpu: 500m
        memory: 256Mi
      requests:
        cpu: 100m
        memory: 128Mi

egazzarr · 2023-06-08T15:19:31Z

Thanks for spotting, and for your prompt help, really appreciate it! Indentation it was in the end.. sorry for the noise.

egazzarr mentioned this issue Jun 8, 2023

Breaking changes in Flux due to Kustomize v4 fluxcd/flux2#1522

Closed

somtochiama closed this as completed Jun 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kustomize build failed: map[string]interface {}(nil): yaml: unmarshal errors #880

kustomize build failed: map[string]interface {}(nil): yaml: unmarshal errors #880

egazzarr commented Jun 7, 2023

makkes commented Jun 7, 2023

egazzarr commented Jun 7, 2023 •

edited

Loading

somtochiama commented Jun 8, 2023

egazzarr commented Jun 8, 2023 •

edited

Loading

somtochiama commented Jun 8, 2023 •

edited

Loading

egazzarr commented Jun 8, 2023

somtochiama commented Jun 8, 2023 •

edited

Loading

egazzarr commented Jun 8, 2023

kustomize build failed: map[string]interface {}(nil): yaml: unmarshal errors #880

kustomize build failed: map[string]interface {}(nil): yaml: unmarshal errors #880

Comments

egazzarr commented Jun 7, 2023

makkes commented Jun 7, 2023

egazzarr commented Jun 7, 2023 • edited Loading

somtochiama commented Jun 8, 2023

egazzarr commented Jun 8, 2023 • edited Loading

somtochiama commented Jun 8, 2023 • edited Loading

egazzarr commented Jun 8, 2023

somtochiama commented Jun 8, 2023 • edited Loading

egazzarr commented Jun 8, 2023

egazzarr commented Jun 7, 2023 •

edited

Loading

egazzarr commented Jun 8, 2023 •

edited

Loading

somtochiama commented Jun 8, 2023 •

edited

Loading

somtochiama commented Jun 8, 2023 •

edited

Loading