Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]VolumeExpansion ops of polardbx cn/cdc is always running #5274

Closed
ahjing99 opened this issue Sep 26, 2023 · 1 comment
Closed

[BUG]VolumeExpansion ops of polardbx cn/cdc is always running #5274

ahjing99 opened this issue Sep 26, 2023 · 1 comment
Assignees
@shanshanying
Labels
bug kind/bug Something isn't working Stale
Milestone
Release 0.9.3

Comments

@ahjing99
Copy link
Collaborator

➜ ~ kbcli version
Kubernetes: v1.27.3-gke.100
KubeBlocks: 0.7.0-alpha.18
kbcli: 0.7.0-alpha.18

The ops status is always running, the pv size has changed but pvc size does not change

➜  ~ helm repo add kubeblocks https://apecloud.github.io/helm-charts
^@^@"kubeblocks" has been added to your repositories
➜  ~ helm upgrade --install polardbx kubeblocks/polardbx --version 0.7.0-alpha.18
Release "polardbx" does not exist. Installing it now.
NAME: polardbx
LAST DEPLOYED: Tue Sep 26 16:46:20 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thanks for installing PolarDB-X using KubeBlocks!

1. Run the following command to create your first PolarDB-X cluster:

kbcli cluster create pxc --cluster-definition polardbx


2. Port-forward service to localhost and connect to PolarDB-X cluster:

kubectl port-forward svc/pxc-cn 3306:3306
mysql -h127.0.0.1 -upolardbx_root

➜  ~ kbcli cluster create test  --cluster-definition=polardbx
Info: --cluster-version is not specified, ClusterVersion polardbx-v1.4.1 is applied by default
Cluster test created

➜  ~ kbcli cluster describe test
Name: test	 Created Time: Sep 26,2023 16:47 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION           STATUS     TERMINATION-POLICY
default     polardbx             polardbx-v1.4.1   Updating   Delete

Endpoints:
COMPONENT   MODE        INTERNAL                                  EXTERNAL
gms         ReadWrite   test-gms.default.svc.cluster.local:3306   <none>
                        test-gms.default.svc.cluster.local:9104
dn          ReadWrite   test-dn.default.svc.cluster.local:3306    <none>
cn          ReadWrite   test-cn.default.svc.cluster.local:3306    <none>
                        test-cn.default.svc.cluster.local:9104
cdc         ReadWrite   test-cdc.default.svc.cluster.local:3306   <none>
                        test-cdc.default.svc.cluster.local:9104

Topology:
COMPONENT   INSTANCE     ROLE       STATUS    AZ              NODE                                                CREATED-TIME
cdc         test-cdc-0   <none>     Running   us-central1-c   gke-yjtest-default-pool-16b6e83c-hsgn/10.128.0.37   Sep 26,2023 16:47 UTC+0800
cn          test-cn-0    <none>     Running   us-central1-c   gke-yjtest-default-pool-16b6e83c-hsgn/10.128.0.37   Sep 26,2023 16:47 UTC+0800
dn          test-dn-0    leader     Running   us-central1-c   gke-yjtest-default-pool-16b6e83c-xqs0/10.128.0.39   Sep 26,2023 16:47 UTC+0800
dn          test-dn-1    follower   Running   us-central1-c   gke-yjtest-default-pool-16b6e83c-pbkd/10.128.0.26   Sep 26,2023 16:47 UTC+0800
dn          test-dn-2    follower   Running   us-central1-c   gke-yjtest-default-pool-16b6e83c-rxv2/10.128.0.38   Sep 26,2023 16:54 UTC+0800
gms         test-gms-0   leader     Running   us-central1-c   gke-yjtest-default-pool-16b6e83c-xqs0/10.128.0.39   Sep 26,2023 16:47 UTC+0800
gms         test-gms-1   follower   Running   us-central1-c   gke-yjtest-default-pool-16b6e83c-pbkd/10.128.0.26   Sep 26,2023 16:47 UTC+0800
gms         test-gms-2   follower   Running   us-central1-c   gke-yjtest-default-pool-16b6e83c-rxv2/10.128.0.38   Sep 26,2023 16:47 UTC+0800

Resources Allocation:
COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
gms         false       1 / 1                1Gi / 1Gi               data:21Gi      kb-default-sc
dn          false       1 / 1                1Gi / 1Gi               data:21Gi      kb-default-sc
cn          false       1 / 1                1Gi / 1Gi               data:21Gi      kb-default-sc
cdc         false       1 / 1                1Gi / 1Gi               data:20Gi      kb-default-sc

Images:
COMPONENT   TYPE   IMAGE
gms         gms    polardbx/polardbx-engine-2.0:latest
dn          dn     polardbx/polardbx-engine-2.0:latest
cn          cn     polardbx/polardbx-sql:latest
cdc         cdc    polardbx/polardbx-cdc:latest

Show cluster events: kbcli cluster list-events -n default test
➜  ~ kbcli cluster describe test
Name: test	 Created Time: Sep 26,2023 16:47 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION           STATUS     TERMINATION-POLICY
default     polardbx             polardbx-v1.4.1   Updating   Delete

Endpoints:
COMPONENT   MODE        INTERNAL                                  EXTERNAL
gms         ReadWrite   test-gms.default.svc.cluster.local:3306   <none>
                        test-gms.default.svc.cluster.local:9104
dn          ReadWrite   test-dn.default.svc.cluster.local:3306    <none>
cn          ReadWrite   test-cn.default.svc.cluster.local:3306    <none>
                        test-cn.default.svc.cluster.local:9104
cdc         ReadWrite   test-cdc.default.svc.cluster.local:3306   <none>
                        test-cdc.default.svc.cluster.local:9104

Topology:
COMPONENT   INSTANCE     ROLE       STATUS    AZ              NODE                                                CREATED-TIME
cdc         test-cdc-0   <none>     Running   us-central1-c   gke-yjtest-default-pool-16b6e83c-hsgn/10.128.0.37   Sep 26,2023 16:47 UTC+0800
cn          test-cn-0    <none>     Running   us-central1-c   gke-yjtest-default-pool-16b6e83c-hsgn/10.128.0.37   Sep 26,2023 16:47 UTC+0800
dn          test-dn-0    leader     Running   us-central1-c   gke-yjtest-default-pool-16b6e83c-xqs0/10.128.0.39   Sep 26,2023 16:47 UTC+0800
dn          test-dn-1    follower   Running   us-central1-c   gke-yjtest-default-pool-16b6e83c-pbkd/10.128.0.26   Sep 26,2023 16:47 UTC+0800
dn          test-dn-2    follower   Running   us-central1-c   gke-yjtest-default-pool-16b6e83c-rxv2/10.128.0.38   Sep 26,2023 16:54 UTC+0800
gms         test-gms-0   leader     Running   us-central1-c   gke-yjtest-default-pool-16b6e83c-xqs0/10.128.0.39   Sep 26,2023 16:47 UTC+0800
gms         test-gms-1   follower   Running   us-central1-c   gke-yjtest-default-pool-16b6e83c-pbkd/10.128.0.26   Sep 26,2023 16:47 UTC+0800
gms         test-gms-2   follower   Running   us-central1-c   gke-yjtest-default-pool-16b6e83c-rxv2/10.128.0.38   Sep 26,2023 16:47 UTC+0800

Resources Allocation:
COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
gms         false       1 / 1                1Gi / 1Gi               data:21Gi      kb-default-sc
dn          false       1 / 1                1Gi / 1Gi               data:21Gi      kb-default-sc
cn          false       1 / 1                1Gi / 1Gi               data:21Gi      kb-default-sc
cdc         false       1 / 1                1Gi / 1Gi               data:20Gi      kb-default-sc

Images:
COMPONENT   TYPE   IMAGE
gms         gms    polardbx/polardbx-engine-2.0:latest
dn          dn     polardbx/polardbx-engine-2.0:latest
cn          cn     polardbx/polardbx-sql:latest
cdc         cdc    polardbx/polardbx-cdc:latest

Show cluster events: kbcli cluster list-events -n default test

➜  ~ kbcli cluster volume-expand test  --components cn --volume-claim-templates data --storage 21Gi
Please type the name again(separate with white space when more than one): test
OpsRequest test-volumeexpansion-z5hcc created successfully, you can view the progress:

kbcli cluster describe-ops test-volumeexpansion-z5hcc -n default


➜  ~ kbcli cluster describe-ops test-volumeexpansion-z5hcc -n default
Spec:
  Name: test-volumeexpansion-z5hcc	NameSpace: default	Cluster: test	Type: VolumeExpansion

Command:
  kbcli cluster volume-expand test --components=cn --volume-claim-template-names=data --storage=21Gi --namespace=default

Last Configuration:
COMPONENT   VOLUME-CLAIM-TEMPLATE   STORAGE
cn          data                    20Gi

Status:
  Start Time:         Sep 26,2023 17:19 UTC+0800
  Duration:           6m55s
  Status:             Running
  Progress:           0/1
                      OBJECT-KEY                 STATUS       DURATION    MESSAGE
                      PVC/data-test-cn-0(data)   Processing   <Unknown>   Start expanding volume: PVC/data-test-cn-0 in Component: cn

Conditions:
LAST-TRANSITION-TIME         TYPE              REASON                         STATUS   MESSAGE
Sep 26,2023 17:19 UTC+0800   Progressing       OpsRequestProgressingStarted   True     Start to process the OpsRequest: test-volumeexpansion-z5hcc in Cluster: test
Sep 26,2023 17:19 UTC+0800   Validated         ValidateOpsRequestPassed       True     OpsRequest: test-volumeexpansion-z5hcc is validated
Sep 26,2023 17:19 UTC+0800   VolumeExpanding   VolumeExpansionStarted         True     Start to expand the volumes in Cluster: test

Warning Events: <none>



➜  ~ k describe pvc data-test-cn-0
Name:          data-test-cn-0
Namespace:     default
StorageClass:  kb-default-sc
Status:        Bound
Volume:        pvc-dafc3f90-eb97-4989-8ed0-4e928e84b0b5
Labels:        app.kubernetes.io/instance=test
               app.kubernetes.io/managed-by=kubeblocks
               app.kubernetes.io/name=polardbx
               apps.kubeblocks.io/component-name=cn
               apps.kubeblocks.io/vct-name=data
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
               volume.kubernetes.io/selected-node: gke-yjtest-default-pool-16b6e83c-hsgn
               volume.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      20Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       test-cn-0
Conditions:
  Type                      Status  LastProbeTime                     LastTransitionTime                Reason  Message
  ----                      ------  -----------------                 ------------------                ------  -------
  FileSystemResizePending   True    Mon, 01 Jan 0001 00:00:00 +0000   Tue, 26 Sep 2023 17:19:42 +0800           Waiting for user to (re-)start a pod to finish file system resize of volume on node.
Events:
  Type     Reason                    Age                From                                                                                              Message
  ----     ------                    ----               ----                                                                                              -------
  Normal   WaitForFirstConsumer      40m                persistentvolume-controller                                                                       waiting for first consumer to be created before binding
  Normal   ExternalProvisioning      40m (x2 over 40m)  persistentvolume-controller                                                                       waiting for a volume to be created, either by external provisioner "pd.csi.storage.gke.io" or manually created by system administrator
  Normal   Provisioning              40m                pd.csi.storage.gke.io_gke-e81b3d5226fe4e088e54-819b-0aa3-vm_8d2cefde-e802-44df-94b2-d1edb676b67a  External provisioner is provisioning volume for claim "default/data-test-cn-0"
  Normal   ProvisioningSucceeded     40m                pd.csi.storage.gke.io_gke-e81b3d5226fe4e088e54-819b-0aa3-vm_8d2cefde-e802-44df-94b2-d1edb676b67a  Successfully provisioned volume pvc-dafc3f90-eb97-4989-8ed0-4e928e84b0b5
  Warning  ExternalExpanding         7m40s              volume_expand                                                                                     waiting for an external controller to expand this PVC
  Normal   Resizing                  7m39s              external-resizer pd.csi.storage.gke.io                                                            External resizer is resizing volume pvc-dafc3f90-eb97-4989-8ed0-4e928e84b0b5
  Normal   FileSystemResizeRequired  7m32s              external-resizer pd.csi.storage.gke.io                                                            Require file system resize of volume on node

➜  ~ k describe pv pvc-dafc3f90-eb97-4989-8ed0-4e928e84b0b5
Name:              pvc-dafc3f90-eb97-4989-8ed0-4e928e84b0b5
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: pd.csi.storage.gke.io
                   volume.kubernetes.io/provisioner-deletion-secret-name:
                   volume.kubernetes.io/provisioner-deletion-secret-namespace:
Finalizers:        [kubernetes.io/pv-protection external-attacher/pd-csi-storage-gke-io]
StorageClass:      kb-default-sc
Status:            Bound
Claim:             default/data-test-cn-0
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          21Gi
Node Affinity:
  Required Terms:
    Term 0:        topology.gke.io/zone in [us-central1-c]
Message:
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            pd.csi.storage.gke.io
    FSType:            xfs
    VolumeHandle:      projects/kubeblocks/zones/us-central1-c/disks/pvc-dafc3f90-eb97-4989-8ed0-4e928e84b0b5
    ReadOnly:          false
    VolumeAttributes:      storage.kubernetes.io/csiProvisionerIdentity=1695697785164-6118-pd.csi.storage.gke.io
Events:                <none>

➜ ~ kbcli report cluster test --with-logs --all-containers
reporting cluster information to report-cluster-test-2023-09-26-17-31-59.zip
processing manifests OK
processing events OK
process pod logs OK
➜ ~ kbcli report kubeblocks --with-logs --all-containers --output yaml
reporting KubeBlocks information to report-kubeblocks-2023-09-26-17-33-00.zip
processing manifests OK
processing events OK
process pod logs
report-kubeblocks-2023-09-26-17-33-00.zip
report-cluster-test-2023-09-26-17-31-59.zip
@ahjing99 ahjing99 added the kind/bug Something isn't working label Sep 26, 2023
@ahjing99 ahjing99 added this to the Release 0.7.0 milestone Sep 26, 2023
@github-actions
Copy link

This issue has been marked as stale because it has been open for 30 days with no activity

@github-actions github-actions bot added the Stale label Oct 30, 2023
@ahjing99 ahjing99 modified the milestones: Release 0.7.0, Release 0.8.0 Nov 6, 2023
@ahjing99 ahjing99 modified the milestones: Release 0.8.0, Release 0.9.0 Jan 12, 2024
@ahjing99 ahjing99 modified the milestones: Release 0.9.0, Release 0.9.1 Jul 8, 2024
@github-actions github-actions bot modified the milestones: Release 0.9.1, Release 0.9.2 Aug 8, 2024
@github-actions github-actions bot modified the milestones: Release 0.9.2, Release 0.8.5 Oct 8, 2024
@github-actions github-actions bot modified the milestones: Release 0.9.2, Release 0.9.3 Jan 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shanshanying shanshanying

Labels
bug kind/bug Something isn't working Stale
Projects
None yet
Milestone
Release 0.9.3
Development

No branches or pull requests
3 participants
@ahjing99 @shanshanying @apecloud-bot