failed to reconcile Slurm Controllers: slurmdb configs not found #203

george-kuanli-peng · 2024-11-27T05:45:10Z

I'm using tag 1.15.3.

It fails to build a Slurm cluster with the following error msg:

reconciling SlurmCluster: reconciling Slurm Controllers: failed to execute steps: multi-step execution failed with collected errors: Slurm Controller StatefulSet: retrieving dependencies for controller StatefulSet: Secret \"slurm1-slurmdbd-configs\" not found"

Logs:

2024-11-27T05:37:38.4524871Z stderr F {"level":"error","ts":"2024-11-27T05:37:38Z","msg":"Failed to reconcile Slurm Controllers","controller":"slurmcluster","controllerGroup":"slurm.nebius.ai","controllerKind":"SlurmCluster","SlurmCluster":{"name":"slurm1","namespace":"slurm1"},"namespace":"slurm1","name":"slurm1","reconcileID":"f41263e9-9755-4fec-8ebc-cef1757aad80","error":"failed to execute steps: multi-step execution failed with collected errors: Slurm Controller StatefulSet: retrieving dependencies for controller StatefulSet: Secret \"slurm1-slurmdbd-configs\" not found","errorVerbose":"multi-step execution failed with collected errors: Slurm Controller StatefulSet: retrieving dependencies for controller StatefulSet: Secret \"slurm1-slurmdbd-configs\" not found\nfailed to execute steps\nnebius.ai/slurm-operator/internal/utils.ExecuteMultiStep\n\t/operator/internal/utils/mutltistep.go:58\nnebius.ai/slurm-operator/internal/controller/clustercontroller.SlurmClusterReconciler.ReconcileControllers.func1\n\t/operator/internal/controller/clustercontroller/controller.go:35\nnebius.ai/slurm-operator/internal/controller/clustercontroller.SlurmClusterReconciler.ReconcileControllers\n\t/operator/internal/controller/clustercontroller/controller.go:118\nnebius.ai/slurm-operator/internal/controller/clustercontroller.(*SlurmClusterReconciler).reconcile.func2\n\t/operator/internal/controller/clustercontroller/reconcile.go:234\nnebius.ai/slurm-operator/internal/controller/clustercontroller.(*SlurmClusterReconciler).runWithPhase\n\t/operator/internal/controller/clustercontroller/reconcile.go:432\nnebius.ai/slurm-operator/internal/controller/clustercontroller.(*SlurmClusterReconciler).reconcile\n\t/operator/internal/controller/clustercontroller/reconcile.go:219\nnebius.ai/slurm-operator/internal/controller/clustercontroller.(*SlurmClusterReconciler).Reconcile\n\t/operator/internal/controller/clustercontroller/reconcile.go:147\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695","stacktrace":"nebius.ai/slurm-operator/internal/controller/clustercontroller.SlurmClusterReconciler.ReconcileControllers\n\t/operator/internal/controller/clustercontroller/controller.go:119\nnebius.ai/slurm-operator/internal/controller/clustercontroller.(*SlurmClusterReconciler).reconcile.func2\n\t/operator/internal/controller/clustercontroller/reconcile.go:234\nnebius.ai/slurm-operator/internal/controller/clustercontroller.(*SlurmClusterReconciler).runWithPhase\n\t/operator/internal/controller/clustercontroller/reconcile.go:432\nnebius.ai/slurm-operator/internal/controller/clustercontroller.(*SlurmClusterReconciler).reconcile\n\t/operator/internal/controller/clustercontroller/reconcile.go:219\nnebius.ai/slurm-operator/internal/controller/clustercontroller.(*SlurmClusterReconciler).Reconcile\n\t/operator/internal/controller/clustercontroller/reconcile.go:147\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}
2024-11-27T05:37:38.45251448Z stderr F {"level":"info","ts":"2024-11-27T05:37:38Z","msg":"Reconciliation state removed for object","controller":"slurmcluster","controllerGroup":"slurm.nebius.ai","controllerKind":"SlurmCluster","SlurmCluster":{"name":"slurm1","namespace":"slurm1"},"namespace":"slurm1","name":"slurm1","reconcileID":"f41263e9-9755-4fec-8ebc-cef1757aad80","kind":"slurm.nebius.ai/v1, Kind=SlurmCluster","key":"slurm1/slurm1"}
2024-11-27T05:37:38.452659541Z stderr F {"level":"error","ts":"2024-11-27T05:37:38Z","msg":"Failed to reconcile SlurmCluster","controller":"slurmcluster","controllerGroup":"slurm.nebius.ai","controllerKind":"SlurmCluster","SlurmCluster":{"name":"slurm1","namespace":"slurm1"},"namespace":"slurm1","name":"slurm1","reconcileID":"f41263e9-9755-4fec-8ebc-cef1757aad80","slurm.cluster.namespace":"slurm1","slurm.cluster.name":"slurm1","error":"reconciling Slurm Controllers: failed to execute steps: multi-step execution failed with collected errors: Slurm Controller StatefulSet: retrieving dependencies for controller StatefulSet: Secret \"slurm1-slurmdbd-configs\" not found","errorVerbose":"multi-step execution failed with collected errors: Slurm Controller StatefulSet: retrieving dependencies for controller StatefulSet: Secret \"slurm1-slurmdbd-configs\" not found\nfailed to execute steps\nnebius.ai/slurm-operator/internal/utils.ExecuteMultiStep\n\t/operator/internal/utils/mutltistep.go:58\nnebius.ai/slurm-operator/internal/controller/clustercontroller.SlurmClusterReconciler.ReconcileControllers.func1\n\t/operator/internal/controller/clustercontroller/controller.go:35\nnebius.ai/slurm-operator/internal/controller/clustercontroller.SlurmClusterReconciler.ReconcileControllers\n\t/operator/internal/controller/clustercontroller/controller.go:118\nnebius.ai/slurm-operator/internal/controller/clustercontroller.(*SlurmClusterReconciler).reconcile.func2\n\t/operator/internal/controller/clustercontroller/reconcile.go:234\nnebius.ai/slurm-operator/internal/controller/clustercontroller.(*SlurmClusterReconciler).runWithPhase\n\t/operator/internal/controller/clustercontroller/reconcile.go:432\nnebius.ai/slurm-operator/internal/controller/clustercontroller.(*SlurmClusterReconciler).reconcile\n\t/operator/internal/controller/clustercontroller/reconcile.go:219\nnebius.ai/slurm-operator/internal/controller/clustercontroller.(*SlurmClusterReconciler).Reconcile\n\t/operator/internal/controller/clustercontroller/reconcile.go:147\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nreconciling Slurm Controllers\nnebius.ai/slurm-operator/internal/controller/clustercontroller.SlurmClusterReconciler.ReconcileControllers\n\t/operator/internal/controller/clustercontroller/controller.go:120\nnebius.ai/slurm-operator/internal/controller/clustercontroller.(*SlurmClusterReconciler).reconcile.func2\n\t/operator/internal/controller/clustercontroller/reconcile.go:234\nnebius.ai/slurm-operator/internal/controller/clustercontroller.(*SlurmClusterReconciler).runWithPhase\n\t/operator/internal/controller/clustercontroller/reconcile.go:432\nnebius.ai/slurm-operator/internal/controller/clustercontroller.(*SlurmClusterReconciler).reconcile\n\t/operator/internal/controller/clustercontroller/reconcile.go:219\nnebius.ai/slurm-operator/internal/controller/clustercontroller.(*SlurmClusterReconciler).Reconcile\n\t/operator/internal/controller/clustercontroller/reconcile.go:147\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695","stacktrace":"nebius.ai/slurm-operator/internal/controller/clustercontroller.(*SlurmClusterReconciler).Reconcile\n\t/operator/internal/controller/clustercontroller/reconcile.go:149\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}
2024-11-27T05:37:38.487512945Z stderr F {"level":"error","ts":"2024-11-27T05:37:38Z","msg":"Reconciler error","controller":"slurmcluster","controllerGroup":"slurm.nebius.ai","controllerKind":"SlurmCluster","SlurmCluster":{"name":"slurm1","namespace":"slurm1"},"namespace":"slurm1","name":"slurm1","reconcileID":"f41263e9-9755-4fec-8ebc-cef1757aad80","error":"reconciling SlurmCluster: reconciling Slurm Controllers: failed to execute steps: multi-step execution failed with collected errors: Slurm Controller StatefulSet: retrieving dependencies for controller StatefulSet: Secret \"slurm1-slurmdbd-configs\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}

It is noted that the Slurm Cluster CRD object has the following settings:

      slurmdbdConfig:
        archiveEvents: 'yes'
        archiveJobs: 'yes'
        archiveResvs: 'yes'
        archiveSteps: 'no'
        archiveSuspend: 'no'
        archiveTXN: 'no'
        archiveUsage: 'yes'
        debugLevel: info
        purgeEventAfter: 1month
        purgeJobAfter: 12month
        purgeResvAfter: 1month
        purgeStepAfter: 1month
        purgeSuspendAfter: 1month
        purgeTXNAfter: 12month
        purgeUsageAfter: 24month
        tcpTimeout: 2

However, I could not see slurm1-slurmdbd-configs in the slurm namespace.

These are the secrets it generated:

And the workloads:

The text was updated successfully, but these errors were encountered:

Uburro · 2024-11-27T08:43:48Z

Can you show the values.yaml file that you are deploying?

george-kuanli-peng · 2024-11-28T01:32:29Z

Can you show the values.yaml file that you are deploying?

This is the values file I used at that time: slurm-cluster-values-issue.zip

My workaround was to create a slurm1-slurmdbd-configs secret manually, and the operator continued to create other pods.

Uburro · 2024-11-28T08:55:17Z

@george-kuanli-peng can you put values in https://gist.github.com/ please?

My workaround was to create a slurm1-slurmdbd-configs secret manually, and the operator continued to create other pods.

it's not ok, cause operator cannot watch this CM and reconcile it. Because don't have owner reference

george-kuanli-peng · 2024-11-28T09:36:36Z

@george-kuanli-peng can you put values in https://gist.github.com/ please?

https://gist.github.com/george-kuanli-peng/c0e05391b59e2592a9b8f078502b401a

Uburro · 2024-11-28T10:01:02Z

@george-kuanli-peng thanks. I will try to reproduce the issue and get back to you.

Uburro · 2024-11-28T10:04:10Z

@george-kuanli-peng i found problem

    externalDB:
      enabled: false
      # host: ""
      # port: 3306
      # user: ""
      # passwordSecretKeyRef:
      #   name: ""
      #   key: ""
    mariadbOperator:
      enabled: false
      resources:
        cpu: "1000m"
        memory: "1Gi"
        ephemeralStorage: "5Gi"

you didn't use mariadb operator or external database of mariadb but accounting is enabled. The slurmdbd config will not be created if the mariadb instance is not running.

if you do not need accounting, please disable it

slurmNodes:
  accounting:
    enabled: false

Uburro · 2024-11-28T12:27:30Z

wi will fix it next release in helm chat. Thank you for your issue.

* build(deps): bump k8s.io/api in /images/jail/gpubench (#193) Bumps [k8s.io/api](https://github.com/kubernetes/api) from 0.31.2 to 0.31.3. - [Commits](kubernetes/api@v0.31.2...v0.31.3) --- updated-dependencies: - dependency-name: k8s.io/api dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump k8s.io/client-go in /images/jail/gpubench (#194) Bumps [k8s.io/client-go](https://github.com/kubernetes/client-go) from 0.31.1 to 0.31.3. - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](kubernetes/client-go@v0.31.1...v0.31.3) --- updated-dependencies: - dependency-name: k8s.io/client-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump github.com/stretchr/testify in /images/jail/gpubench (#197) Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.9.0 to 1.10.0. - [Release notes](https://github.com/stretchr/testify/releases) - [Commits](stretchr/testify@v1.9.0...v1.10.0) --- updated-dependencies: - dependency-name: github.com/stretchr/testify dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * hotfix issue #204 : fix ssh config (#198) * hotfix: fix ssh config * Create SECURITY.md * excluded SECURETY.MD from ci pepiline * rm auto_assign file * MSP-2349: add memory.max to system.slice (#201) * MSP-2349: add real memory and check resources * fix issue #203 (#205) --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Alexander <[email protected]> Co-authored-by: Pavel Sofrony <[email protected]> Co-authored-by: Pavel Sofronii <[email protected]>

Uburro · 2024-11-28T13:06:02Z

@george-kuanli-peng https://github.com/nebius/soperator/releases/tag/1.15.4 fixed, can you check it with 1.15.4 version of helm chart, please?

george-kuanli-peng · 2024-11-29T07:54:54Z

Thanks for your quick fix.
I'm trying to check it with 1.15.4 but may not finish it soon since I have a slow network connection in pulling the new images, and the host will be temporarily off this weekend. Will give my results next week.

george-kuanli-peng · 2024-12-03T06:57:08Z

Hi @Uburro, I can create a Slurm cluster now based on 1.15.4. Thanks for your help.

…hinery from 0.30.2 to 0.31.2, sigs.k8s.io/controller-runtime from 0.18.4 to 0.19.0, mariadb-operator from v0.0.29 to v0.36.0. Close PR from dependobot #38, #39, #47, #189, #195 (#196) * build(deps): bump k8s.io/api in /images/jail/gpubench (#193) Bumps [k8s.io/api](https://github.com/kubernetes/api) from 0.31.2 to 0.31.3. - [Commits](kubernetes/api@v0.31.2...v0.31.3) --- updated-dependencies: - dependency-name: k8s.io/api dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump k8s.io/client-go in /images/jail/gpubench (#194) Bumps [k8s.io/client-go](https://github.com/kubernetes/client-go) from 0.31.1 to 0.31.3. - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](kubernetes/client-go@v0.31.1...v0.31.3) --- updated-dependencies: - dependency-name: k8s.io/client-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump github.com/stretchr/testify in /images/jail/gpubench (#197) Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.9.0 to 1.10.0. - [Release notes](https://github.com/stretchr/testify/releases) - [Commits](stretchr/testify@v1.9.0...v1.10.0) --- updated-dependencies: - dependency-name: github.com/stretchr/testify dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * hotfix: fix ssh config (#198) * hotfix: fix ssh config * Create SECURITY.md * excluded SECURETY.MD from ci pepiline * rm auto_assign file * MSP-2349: add memory.max to system.slice (#201) * MSP-2349: add real memory and check resources * fix issue #203 and upgrade version (#205) * BUMP: version otel and mariadb with convert ctruct * fix test affinityConfig * bump version of golang * make sync-version-from-scratch * notask: fix space in ln (#208) * NOTASK: real memory convert bytes to mebibytes (#209) * bump 1.15.5 (#210) --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Alexander <[email protected]> Co-authored-by: Pavel Sofrony <[email protected]> Co-authored-by: Pavel Sofronii <[email protected]>

…hinery from 0.30.2 to 0.31.2, sigs.k8s.io/controller-runtime from 0.18.4 to 0.19.0, mariadb-operator from v0.0.29 to v0.36.0. Close PR from dependobot #38, #39, #47, #189, #195 (#196) * build(deps): bump k8s.io/api in /images/jail/gpubench (#193) Bumps [k8s.io/api](https://github.com/kubernetes/api) from 0.31.2 to 0.31.3. - [Commits](kubernetes/api@v0.31.2...v0.31.3) --- updated-dependencies: - dependency-name: k8s.io/api dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump k8s.io/client-go in /images/jail/gpubench (#194) Bumps [k8s.io/client-go](https://github.com/kubernetes/client-go) from 0.31.1 to 0.31.3. - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](kubernetes/client-go@v0.31.1...v0.31.3) --- updated-dependencies: - dependency-name: k8s.io/client-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump github.com/stretchr/testify in /images/jail/gpubench (#197) Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.9.0 to 1.10.0. - [Release notes](https://github.com/stretchr/testify/releases) - [Commits](stretchr/testify@v1.9.0...v1.10.0) --- updated-dependencies: - dependency-name: github.com/stretchr/testify dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * hotfix: fix ssh config (#198) * hotfix: fix ssh config * Create SECURITY.md * excluded SECURETY.MD from ci pepiline * rm auto_assign file * MSP-2349: add memory.max to system.slice (#201) * MSP-2349: add real memory and check resources * fix issue #203 and upgrade version (#205) * BUMP: version otel and mariadb with convert ctruct * fix test affinityConfig * bump version of golang * make sync-version-from-scratch * notask: fix space in ln (#208) * NOTASK: real memory convert bytes to mebibytes (#209) * bump 1.15.5 (#210) --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Alexander <[email protected]> Co-authored-by: Pavel Sofrony <[email protected]> Co-authored-by: Pavel Sofronii <[email protected]> update

* notask: fix space in ln (#208) * NOTASK: real memory convert bytes to mebibytes (#209) * bump 1.15.5 (#210) * MSP-3525: Bump golang from 1.22 to 1.23, k8s.io/api and k8s.io/apimachinery from 0.30.2 to 0.31.2, sigs.k8s.io/controller-runtime from 0.18.4 to 0.19.0, mariadb-operator from v0.0.29 to v0.36.0. Close PR from dependobot #38, #39, #47, #189, #195 (#196) * build(deps): bump k8s.io/api in /images/jail/gpubench (#193) Bumps [k8s.io/api](https://github.com/kubernetes/api) from 0.31.2 to 0.31.3. - [Commits](kubernetes/api@v0.31.2...v0.31.3) --- updated-dependencies: - dependency-name: k8s.io/api dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump k8s.io/client-go in /images/jail/gpubench (#194) Bumps [k8s.io/client-go](https://github.com/kubernetes/client-go) from 0.31.1 to 0.31.3. - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](kubernetes/client-go@v0.31.1...v0.31.3) --- updated-dependencies: - dependency-name: k8s.io/client-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump github.com/stretchr/testify in /images/jail/gpubench (#197) Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.9.0 to 1.10.0. - [Release notes](https://github.com/stretchr/testify/releases) - [Commits](stretchr/testify@v1.9.0...v1.10.0) --- updated-dependencies: - dependency-name: github.com/stretchr/testify dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * hotfix: fix ssh config (#198) * hotfix: fix ssh config * Create SECURITY.md * excluded SECURETY.MD from ci pepiline * rm auto_assign file * MSP-2349: add memory.max to system.slice (#201) * MSP-2349: add real memory and check resources * fix issue #203 and upgrade version (#205) * BUMP: version otel and mariadb with convert ctruct * fix test affinityConfig * bump version of golang * make sync-version-from-scratch * notask: fix space in ln (#208) * NOTASK: real memory convert bytes to mebibytes (#209) * bump 1.15.5 (#210) --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Alexander <[email protected]> Co-authored-by: Pavel Sofrony <[email protected]> Co-authored-by: Pavel Sofronii <[email protected]> * Bump golang from 1.22 to 1.23 (#38) Bumps golang from 1.22 to 1.23. --- updated-dependencies: - dependency-name: golang dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump k8s.io/apimachinery from 0.30.2 to 0.31.3 (#195) Bumps [k8s.io/apimachinery](https://github.com/kubernetes/apimachinery) from 0.30.2 to 0.31.3. - [Commits](kubernetes/apimachinery@v0.30.2...v0.31.3) --- updated-dependencies: - dependency-name: k8s.io/apimachinery dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump k8s.io/api from 0.31.2 to 0.31.3 (#213) Bumps [k8s.io/api](https://github.com/kubernetes/api) from 0.31.2 to 0.31.3. - [Commits](kubernetes/api@v0.31.2...v0.31.3) --- updated-dependencies: - dependency-name: k8s.io/api dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump sigs.k8s.io/controller-runtime from 0.19.1 to 0.19.3 (#214) Bumps [sigs.k8s.io/controller-runtime](https://github.com/kubernetes-sigs/controller-runtime) from 0.19.1 to 0.19.3. - [Release notes](https://github.com/kubernetes-sigs/controller-runtime/releases) - [Changelog](https://github.com/kubernetes-sigs/controller-runtime/blob/main/RELEASE.md) - [Commits](kubernetes-sigs/controller-runtime@v0.19.1...v0.19.3) --- updated-dependencies: - dependency-name: sigs.k8s.io/controller-runtime dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * MSP-3191: add webhook protect delete secret mariadb (#215) MSP-3191: add webhook protect delete secret mariadb * build(deps): bump google.golang.org/grpc in /images/jail/gpubench (#216) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.68.0 to 1.68.1. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.68.0...v1.68.1) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump github.com/onsi/ginkgo/v2 from 2.21.0 to 2.22.0 (#217) Bumps [github.com/onsi/ginkgo/v2](https://github.com/onsi/ginkgo) from 2.21.0 to 2.22.0. - [Release notes](https://github.com/onsi/ginkgo/releases) - [Changelog](https://github.com/onsi/ginkgo/blob/master/CHANGELOG.md) - [Commits](onsi/ginkgo@v2.21.0...v2.22.0) --- updated-dependencies: - dependency-name: github.com/onsi/ginkgo/v2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump k8s.io/client-go from 0.31.2 to 0.31.3 (#219) Bumps [k8s.io/client-go](https://github.com/kubernetes/client-go) from 0.31.2 to 0.31.3. - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](kubernetes/client-go@v0.31.2...v0.31.3) --- updated-dependencies: - dependency-name: k8s.io/client-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump github.com/stretchr/testify from 1.9.0 to 1.10.0 (#218) Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.9.0 to 1.10.0. - [Release notes](https://github.com/stretchr/testify/releases) - [Commits](stretchr/testify@v1.9.0...v1.10.0) --- updated-dependencies: - dependency-name: github.com/stretchr/testify dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump github.com/onsi/gomega from 1.35.1 to 1.36.0 (#220) Bumps [github.com/onsi/gomega](https://github.com/onsi/gomega) from 1.35.1 to 1.36.0. - [Release notes](https://github.com/onsi/gomega/releases) - [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md) - [Commits](onsi/gomega@v1.35.1...v1.36.0) --- updated-dependencies: - dependency-name: github.com/onsi/gomega dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring (#221) Bumps [github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring](https://github.com/prometheus-operator/prometheus-operator) from 0.78.0 to 0.78.2. - [Release notes](https://github.com/prometheus-operator/prometheus-operator/releases) - [Changelog](https://github.com/prometheus-operator/prometheus-operator/blob/main/CHANGELOG.md) - [Commits](prometheus-operator/prometheus-operator@v0.78.0...v0.78.2) --- updated-dependencies: - dependency-name: github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * MSP-3635: add generate rbac (#222) * MSP-3642: add some configmap values (#223) * MSP-3642: add some configmap values * build(deps): bump alpine from `1e42bbe` to `21dc606` (#224) Bumps alpine from `1e42bbe` to `21dc606`. --- updated-dependencies: - dependency-name: alpine dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * MSP-3541: removed unused crd values (#225) * build(deps): bump golang.org/x/crypto from 0.29.0 to 0.30.0 (#226) Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.29.0 to 0.30.0. - [Commits](golang/crypto@v0.29.0...v0.30.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * 3578: moving to native sidecar (#227) * 3578: moving to native sidecar * HOTFIX: fix ServiceAccount name for role binding (#229) * HOTFIX: fix ServiceAccount name for role binding * build(deps): bump github.com/onsi/gomega from 1.36.0 to 1.36.1 (#230) Bumps [github.com/onsi/gomega](https://github.com/onsi/gomega) from 1.36.0 to 1.36.1. - [Release notes](https://github.com/onsi/gomega/releases) - [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md) - [Commits](onsi/gomega@v1.36.0...v1.36.1) --- updated-dependencies: - dependency-name: github.com/onsi/gomega dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * set compute instance name as slurm InstanceId * MSP-3272: Support GDRCopy + preinstall more tools * build(deps): bump k8s.io/client-go from 0.31.3 to 0.31.4 (#233) Bumps [k8s.io/client-go](https://github.com/kubernetes/client-go) from 0.31.3 to 0.31.4. - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](kubernetes/client-go@v0.31.3...v0.31.4) --- updated-dependencies: - dependency-name: k8s.io/client-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump softprops/action-gh-release from 2.1.0 to 2.2.0 Bumps [softprops/action-gh-release](https://github.com/softprops/action-gh-release) from 2.1.0 to 2.2.0. - [Release notes](https://github.com/softprops/action-gh-release/releases) - [Changelog](https://github.com/softprops/action-gh-release/blob/master/CHANGELOG.md) - [Commits](softprops/action-gh-release@01570a1...7b4da11) --- updated-dependencies: - dependency-name: softprops/action-gh-release dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump actions/setup-go from 5.1.0 to 5.2.0 Bumps [actions/setup-go](https://github.com/actions/setup-go) from 5.1.0 to 5.2.0. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@41dfa10...3041bf5) --- updated-dependencies: - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump k8s.io/api in /images/jail/gpubench (#236) Bumps [k8s.io/api](https://github.com/kubernetes/api) from 0.31.3 to 0.31.4. - [Commits](kubernetes/api@v0.31.3...v0.31.4) --- updated-dependencies: - dependency-name: k8s.io/api dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang from `574185e` to `7003184` (#243) Bumps golang from `574185e` to `7003184`. --- updated-dependencies: - dependency-name: golang dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/crypto from 0.30.0 to 0.31.0 (#247) Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.30.0 to 0.31.0. - [Commits](golang/crypto@v0.30.0...v0.31.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * MSP-3578: add sshd to worker (#248) * MSP-3578: add sshd to worker * run enroot containers without root privileges * build(deps): bump go.opentelemetry.io/otel/sdk/metric (#253) Bumps [go.opentelemetry.io/otel/sdk/metric](https://github.com/open-telemetry/opentelemetry-go) from 1.32.0 to 1.33.0. - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.32.0...v1.33.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/otel/sdk/metric dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump google.golang.org/grpc in /images/jail/gpubench (#252) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.68.1 to 1.69.0. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.68.1...v1.69.0) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp (#254) Bumps [go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp](https://github.com/open-telemetry/opentelemetry-go) from 1.32.0 to 1.33.0. - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.32.0...v1.33.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * MSP-3705: Support starting Docker containers from Slurm jobs * description protectedSecret (#258) * NOTIC: Add rights to manage daemonsets * fix error var * run build for release (#265) --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Alexander <[email protected]> Co-authored-by: Pavel Sofrony <[email protected]> Co-authored-by: Pavel Sofronii <[email protected]> Co-authored-by: rdjjke <[email protected]> Co-authored-by: Mikhail Mokrushin <[email protected]>

Uburro added a commit that referenced this issue Nov 28, 2024

fix issue #203 and upgrade version

56fbfaf

Uburro added a commit that referenced this issue Nov 28, 2024

fix issue #203 and upgrade version (#205)

cbc162c

Uburro closed this as completed Dec 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

failed to reconcile Slurm Controllers: slurmdb configs not found #203

failed to reconcile Slurm Controllers: slurmdb configs not found #203

george-kuanli-peng commented Nov 27, 2024

Uburro commented Nov 27, 2024 •

edited

Loading

george-kuanli-peng commented Nov 28, 2024

Uburro commented Nov 28, 2024 •

edited

Loading

george-kuanli-peng commented Nov 28, 2024

Uburro commented Nov 28, 2024

Uburro commented Nov 28, 2024 •

edited

Loading

Uburro commented Nov 28, 2024

Uburro commented Nov 28, 2024 •

edited

Loading

george-kuanli-peng commented Nov 29, 2024 •

edited

Loading

george-kuanli-peng commented Dec 3, 2024

failed to reconcile Slurm Controllers: slurmdb configs not found #203

failed to reconcile Slurm Controllers: slurmdb configs not found #203

Comments

george-kuanli-peng commented Nov 27, 2024

Uburro commented Nov 27, 2024 • edited Loading

george-kuanli-peng commented Nov 28, 2024

Uburro commented Nov 28, 2024 • edited Loading

george-kuanli-peng commented Nov 28, 2024

Uburro commented Nov 28, 2024

Uburro commented Nov 28, 2024 • edited Loading

Uburro commented Nov 28, 2024

Uburro commented Nov 28, 2024 • edited Loading

george-kuanli-peng commented Nov 29, 2024 • edited Loading

george-kuanli-peng commented Dec 3, 2024

Uburro commented Nov 27, 2024 •

edited

Loading

Uburro commented Nov 28, 2024 •

edited

Loading

Uburro commented Nov 28, 2024 •

edited

Loading

Uburro commented Nov 28, 2024 •

edited

Loading

george-kuanli-peng commented Nov 29, 2024 •

edited

Loading