Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No news disks are added to the LVMCluster/LVMvolumegroupnodestatuses #105

Closed
jgato opened this issue Feb 7, 2022 · 4 comments · Fixed by #110
Closed

No news disks are added to the LVMCluster/LVMvolumegroupnodestatuses #105

jgato opened this issue Feb 7, 2022 · 4 comments · Fixed by #110

Comments

@jgato
Copy link

jgato commented Feb 7, 2022

I had one SNO with an LVMCluster created to manage the VG vg1 (/dev/nvme0n1, /dev/nvme1n1, /dev/sda). I wanted to test the addition of new disks. So, I rebooted and created some more disks from the raid I have in the server.

After creating some more disks the SNO is rebooted and:

  • The new disks (/dev/sdc, /dev/sdd) have been added to the vg1 in the node:
[root@master-0 core]# vgs
  VG  #PV #LV #SN Attr   VSize VFree 
  vg1   5   4   0 wz--n- 6.00t <6.00t
[root@master-0 core]# pvs
  PV           VG  Fmt  Attr PSize   PFree  
  /dev/nvme0n1 vg1 lvm2 a--  745.21g 745.21g
  /dev/nvme1n1 vg1 lvm2 a--  745.21g 745.21g
  /dev/sda     vg1 lvm2 a--   <2.73t   2.72t
  /dev/sdc     vg1 lvm2 a--  931.48g 931.48g
  /dev/sde     vg1 lvm2 a--  931.48g 931.48g
  • This new disks are not recognized by the LVMCluster/LVMvolumegroupnodestatuses:
$ oc get lvmvolumegroupnodestatuses -o yaml
apiVersion: v1
items:
- apiVersion: lvm.topolvm.io/v1alpha1
  kind: LVMVolumeGroupNodeStatus
  metadata:
    creationTimestamp: "2022-02-05T18:58:01Z"
    generation: 1
    name: master-0.apollo2.hpecloud.org
    namespace: lvm-operator-system
    resourceVersion: "6276721"
    uid: b6cc0395-e17c-4116-941f-9b95b8c4ed82
  spec:
    nodeStatus:
    - devices:
      - /dev/nvme0n1
      - /dev/nvme1n1
      - /dev/sda
      name: vg1
      status: Ready
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

$ oc get lvmcluster  lvmcluster-sample -o yaml
apiVersion: lvm.topolvm.io/v1alpha1
kind: LVMCluster
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"lvm.topolvm.io/v1alpha1","kind":"LVMCluster","metadata":{"annotations":{},"name":"lvmcluster-sample","namespace":"lvm-operator-system"},"spec":{"deviceClasses":[{"name":"vg1"}]}}
  creationTimestamp: "2022-02-05T18:56:54Z"
  finalizers:
  - lvmcluster.topolvm.io
  generation: 1
  name: lvmcluster-sample
  namespace: lvm-operator-system
  resourceVersion: "6276722"
  uid: fb91728e-1300-4fe1-9041-78d5a8e166d6
spec:
  deviceClasses:
  - name: vg1
status:
  deviceClassStatuses:
  - name: vg1
    nodeStatus:
    - devices:
      - /dev/nvme0n1
      - /dev/nvme1n1
      - /dev/sda
      node: master-0.apollo2.hpecloud.org
      status: Ready
  ready: true

I have created a second LVMCluster because I thought this would collect the new disks. Now I understand it is only one LVMCluster supported, would this have interfere?
@nbalacha
Copy link
Contributor

nbalacha commented Feb 7, 2022

I have created a second LVMCluster because I thought this would collect the new disks. Now I understand it is only one LVMCluster supported, would this have interfere?

It should not. Do you see any errors in the controller-manager or vgmanager logs files?

@jgato
Copy link
Author

jgato commented Feb 7, 2022

From the controller-manager:

{
  "level": "error",
  "ts": 1644229480.7411144,
  "logger": "controller.lvmcluster.lvmcluster-controller",
  "msg": "failed to create or update vgManager daemonset",
  "reconciler group": "lvm.topolvm.io",
  "reconciler kind": "LVMCluster",
  "name": "vg-manager",
  "namespace": "lvm-operator-system",
  "error": "failed to update controller reference on vgManager daemonset \"vg-manager\". Object lvm-operator-system/vg-manager is already owned by another LVMCluster controller lvmcluster-sample",
  "stacktrace": "github.com/red-hat-storage/lvm-operator/controllers.(*LVMClusterReconciler).Reconcile\n\t/workspace/controllers/lvmcluster_controller.go:104\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"
}
{
  "level": "error",
  "ts": 1644229480.7504883,
  "logger": "controller.lvmcluster",
  "msg": "Reconciler error",
  "reconciler group": "lvm.topolvm.io",
  "reconciler kind": "LVMCluster",
  "name": "lvmcluster-sample-2",
  "namespace": "lvm-operator-system",
  "error": "failed reconciling: vg-manager failed to update controller reference on vgManager daemonset \"vg-manager\". Object lvm-operator-system/vg-manager is already owned by another LVMCluster controller lvmcluster-sample",
  "stacktrace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"
}

So it could be the second LVMCCluster creation could be causing problems. Do you want me to delete it?

No errors in the vgmanager.

@nbalacha
Copy link
Contributor

nbalacha commented Feb 7, 2022

So it could be the second LVMCCluster creation could be causing problems. Do you want me to delete it?

Yes, please delete the second LVMCluster.

@jgato
Copy link
Author

jgato commented Feb 9, 2022

Ok, I have cleaned up everything. The second LVMCluster was causing problems and it is something not supported yet.
I have repeated the process:
Phase 1)

  • Everything in a clean stage
  • Operator deployed an LVMCluster created.
  • it detects all the available disks and the VG/PV are created correctly
 Device Class Statuses:
   Name:  vg1
   Node Status:
     Devices:
       /dev/nvme0n1
       /dev/nvme1n1
       /dev/sda
       /dev/sdc
       /dev/sdd
       /dev/sdg
     Node:    master-0.apollo2.hpecloud.org

[root@master-0 core]# vgs                                                                                                                                                                                                                                     
  VG  #PV #LV #SN Attr   VSize VFree                                                                                                                                                                                                                          
  vg1   6   6   0 wz--n- 6.91t 6.90t       
[root@master-0 core]# pvs                                                                                                                                                                                                                                     
  PV           VG  Fmt  Attr PSize   PFree                                                                                                                                                                                                                    
  /dev/nvme0n1 vg1 lvm2 a--  745.21g 745.21g                                                                                                                                                                                                                  
  /dev/nvme1n1 vg1 lvm2 a--  745.21g 745.21g                                                                                                                                                                                                                  
  /dev/sda     vg1 lvm2 a--   <2.73t  <2.72t                                                                                                                                                                                                                  
  /dev/sdc     vg1 lvm2 a--  931.48g 931.48g                                                                                                                                                                                                                  
  /dev/sdd     vg1 lvm2 a--  931.48g 931.48g                                                                                                                                                                                                                  
  /dev/sdg     vg1 lvm2 a--  931.48g 931.48g

Phase 2)

  • Notice /dev/sde and /dev/sdf were not included initialy. This is ok, because these were not empty.
  • Cleaning both disks but these are still not included into the VG
  • Reboot the sno
  • Now the disks have been included in the VG
# pvs
  PV           VG  Fmt  Attr PSize   PFree  
  /dev/nvme0n1 vg1 lvm2 a--  745.21g 745.21g
  /dev/nvme1n1 vg1 lvm2 a--  745.21g 745.21g
  /dev/sda     vg1 lvm2 a--   <2.73t  <2.72t
  /dev/sdc     vg1 lvm2 a--  931.48g 931.48g
  /dev/sdd     vg1 lvm2 a--  931.48g 931.48g
  /dev/sde     vg1 lvm2 a--  931.48g 931.48g
  /dev/sdf     vg1 lvm2 a--  931.48g 931.48g
  /dev/sdg     vg1 lvm2 a--  931.48g 931.48g
  • But still not recognizable by the LVMCluster
  Device Class Statuses:
    Name:  vg1
    Node Status:
      Devices:
        /dev/nvme0n1
        /dev/nvme1n1
        /dev/sda
        /dev/sdc
        /dev/sdd
        /dev/sdg

@nbalacha nbalacha mentioned this issue Feb 15, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: updates lvmvolumegroupnodestatus for vg
2 participants
@jgato @nbalacha