Cannot set a different default storage class #118

rtyler · 2018-01-11T18:57:01Z

#48 was resolved, and believe the upgrade I performed a week or two ago from Kubernetes 1.8.1 to 1.8.2 caused the "fix" to come into effect.

Unfortunately I had already set managed-premium as the IsDefaultClass, and now AKS has two default storage classes set:

Name:            default
IsDefaultClass:  Yes
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1beta1","kind":"StorageClass","metadata":{"annotations":{"storageclass.beta.kubernetes.io/is-default-class":"true"},"labels":{"kubernetes.io/cluster-service":"true"},"name":"default","namespace":""},"parameters":{"kind":"Managed","storageaccounttype":"Standard_LRS"},"provisioner":"kubernetes.io/azure-disk"}
,storageclass.beta.kubernetes.io/is-default-class=true
Provisioner:  kubernetes.io/azure-disk
Parameters:   kind=Managed,storageaccounttype=Standard_LRS
Events:       <none>


Name:            managed-premium
IsDefaultClass:  Yes
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1beta1","kind":"StorageClass","metadata":{"annotations":{},"labels":{"kubernetes.io/cluster-service":"true"},"name":"managed-premium","namespace":""},"parameters":{"kind":"Managed","storageaccounttype":"Premium_LRS"},"provisioner":"kubernetes.io/azure-disk"}
,storageclass.kubernetes.io/is-default-class=true
Provisioner:  kubernetes.io/azure-disk
Parameters:   kind=Managed,storageaccounttype=Premium_LRS
Events:       <none>

I don't know if this is a transitory issue when upgrading from 1.8.1 to 1.8.2, but any change which adjusts the storage clases should check that a default storage class has not already been set.

The text was updated successfully, but these errors were encountered:

rtyler · 2018-01-11T18:58:03Z

Of course, two default storageclasses causes a problem, in case that wasn't already obvious:

create Claim jenkins-dduportal-jenkins-dduportal-0 for Pod jenkins-dduportal-0 in StatefulSet jenkins-dduportal failed error: persistentvolumeclaims "jenkins-dduportal-jenkins-dduportal-0" is forbidden: Internal error occurred: 2 default StorageClasses were found

rtyler · 2018-01-11T20:01:19Z

I've attempted to correct this a few times, but it appears that I cannot unset default as the default storage class.

wut.

andyzhangx · 2018-01-12T08:24:49Z

what's the result of

kubectl patch storageclass default -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

it does not work?

rtyler · 2018-01-12T15:53:59Z

If you check the storage class after a minute or two, you should observe that it has returned to it's default state:

➜  codevalet git:(master) ✗ KUBECONFIG=$PWD/.kube/config ~/bin/kubectl patch storageclass default -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
storageclass "default" patched
➜  codevalet git:(master) ✗ KUBECONFIG=$PWD/.kube/config ~/bin/kubectl describe storageclass default                                                                              
Name:            default
IsDefaultClass:  Yes
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1beta1","kind":"StorageClass","metadata":{"annotations":{"storageclass.beta.kubernetes.io/is-default-class":"true"},"labels":{"kubernetes.io/cluster-service":"true"},"name":"default","namespace":""},"parameters":{"kind":"Managed","storageaccounttype":"Standard_LRS"},"provisioner":"kubernetes.io/azure-disk"}
,storageclass.beta.kubernetes.io/is-default-class=true,storageclass.kubernetes.io/is-default-class=false
Provisioner:  kubernetes.io/azure-disk
Parameters:   kind=Managed,storageaccounttype=Standard_LRS
Events:       <none>

rabenhorst · 2018-01-15T13:21:56Z

I just observed the exact same behavior.

andyzhangx · 2018-01-15T13:28:12Z

I just realized that the built-in storage class are all in /etc/kubernetes/addons/azure-storage-classes.yaml in master node, unless you could change this file, otherwise you could not change the default setting. The point is you could not access file /etc/kubernetes/addons/azure-storage-classes.yaml...
@slack

jzakotnik · 2018-03-26T13:58:13Z

We observe the same issue. Workaround is to change the file above.
@gonzochick

jimmykarily · 2018-04-17T21:54:37Z

Let me just link the 2 issues (they are similar): kubernetes/kubernetes#53428

seanknox · 2018-08-02T19:11:19Z

Closing due inactivity. Feel free to re-open if still an issue.

kwaazaar · 2019-03-11T09:36:52Z

Why is this closed? The proposed fix requires me to change config on the master, but with AKS my master is managed (away), so how could this work? I really like to specify my own default storage class, so I have a little more control. Currently I cannot.

ddarling85 · 2019-04-23T22:35:33Z

Hello, has anyone been able to get around this issue yet? I just attempted to change my default storage class also. Is there a way to dynamically assign storage using the default class?

stevendstark · 2019-07-31T23:46:39Z

Hello, I'm also running into the same issue. Has anyone been able to change the default storage class from AKS?

huangyingting · 2019-10-28T23:21:38Z

@andyzhangx and @jnoller , do we have any updates on this to change default storageclass from AKS?

sharkymcdongles · 2019-11-05T00:16:39Z

I also need a fix for this @andyzhangx pretty plz <3

andyliddle · 2020-01-08T13:11:10Z

is there any fix for this yet? I don't want azure-disk to be default in my cluster

gkaleta · 2020-02-19T11:07:50Z

@andyzhangx @jnoller can you follow up on this? @palma21 FYI

berndverst · 2020-04-22T00:22:27Z

Just adding my thoughts here -- for plenty of use cases, including KubeFlow, it would be preferable to change the default storage class.

For example in KubeFlow I would prefer using Azure Files (which I can mount as Read Write Many) instead of Azure Disks that can only be attached to a single node.

sblack4 · 2020-05-12T19:33:30Z

This looks like a duplicate of #1075, but the good news is it's been added to the roadmap :) https://github.com/Azure/AKS/projects/1#card-34416796

andyzhangx · 2020-05-13T09:17:29Z

Here is the way how to change the default storage class on AKS:

prepare default storage class and save to your local env, e.g. default.yaml

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    kubernetes.io/cluster-service: "true"
  name: default
parameters:
  cachingmode: ReadOnly
  kind: Managed
  storageaccounttype: Premium_LRS
provisioner: kubernetes.io/azure-disk
reclaimPolicy: Delete
volumeBindingMode: Immediate

Run following two commands consecutively:

kubectl delete sc default && kubectl apply -f default.yaml

check whether default storage class is changed:

kubectl get sc default -o yaml

By this way, you could change other built-in storage classes too.

berndverst · 2020-05-19T04:01:56Z

@andyzhangx I'm pretty sure this is exactly what I tried multiple times but it didn't work.

If I do kubectl get sc default -o yaml everything is fine, but once I run kubectl get sc it has reverted.

andyzhangx · 2020-06-08T11:57:14Z

@andyzhangx I'm pretty sure this is exactly what I tried multiple times but it didn't work.

If I do kubectl get sc default -o yaml everything is fine, but once I run kubectl get sc it has reverted.

@berndverst what's your aks version? I tried on AKS 1.16 and 1.17 clusters, it works as expected. The key point is run following two command immediately:

kubectl delete sc default && kubectl apply -f default.yaml

DanielSCon40 · 2020-07-01T14:34:16Z

@andyzhangx we are using v1.16.9 and it still does revert to initial state. Now I have two defaults.

andyzhangx · 2020-07-03T06:53:04Z

@andyzhangx we are using v1.16.9 and it still does revert to initial state. Now I have two defaults.

pls make sure you have prepared default.yaml file first(see above example: #118 (comment)) and then run following command:

kubectl delete sc default && kubectl apply -f default.yaml

And if you run kubectl delete sc default first and run kubectl apply -f default.yaml command after a while, it may not work. Pls make sure kubectl delete and kubectl apply command runs consecutively. Also when there is cluster upgrade, the default class setting would be reverted. So this is only a workaround.

egoienola · 2020-07-17T14:09:52Z

Having tested it with 1.16, 1.17 and 1.18, I can confirm that any changes I make to the default sc get reverted within a minute. This is true for all storage classes provided by AKS. e.g. azurefile.

$ kubectl get sc
NAME                PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
azurefile           kubernetes.io/azure-file   Delete          Immediate           true                   37m
azurefile-premium   kubernetes.io/azure-file   Delete          Immediate           true                   37m
default (default)   kubernetes.io/azure-disk   Delete          Immediate           true                   37m
managed-premium     kubernetes.io/azure-disk   Delete          Immediate           true                   37m

$ kubectl describe sc azurefile
Name:            azurefile
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1beta1","kind":"StorageClass","metadata":{"annotations":{},"labels":{"kubernetes.io/cluster-service":"true"},"name":"azurefile"},"parameters":{"skuName":"Standard_LRS"},"provisioner":"kubernetes.io/azure-file"}

Provisioner:           kubernetes.io/azure-file
Parameters:            skuName=Standard_LRS
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>

$ cat << 'EOF' > azurefile.yaml
> kind: StorageClass
> apiVersion: storage.k8s.io/v1
> metadata:
>   name: azurefile
> provisioner: kubernetes.io/azure-file
> mountOptions:
>   - dir_mode=0777
>   - file_mode=0777
>   - uid=1001
>   - gid=1001
> parameters:
>   skuName: Standard_LRS
> EOF

$ kubectl delete sc azurefile && kubectl apply -f azurefile.yaml
storageclass.storage.k8s.io "azurefile" deleted
storageclass.storage.k8s.io/azurefile created

$ date
Tue Jul 14 11:40:49 EDT 2020

$ kubectl delete sc azurefile && kubectl apply -f azurefile.yaml
storageclass.storage.k8s.io "azurefile" deleted
storageclass.storage.k8s.io/azurefile created
sukezg@cldlgn05:~/viya4-azure$ kubectl describe sc azurefile
Name:            azurefile
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"azurefile"},"mountOptions":["dir_mode=0777","file_mode=0777","uid=1001","gid=1001"],"parameters":{"skuName":"Standard_LRS"},"provisioner":"kubernetes.io/azure-file"}

Provisioner:           kubernetes.io/azure-file
Parameters:            skuName=Standard_LRS
AllowVolumeExpansion:  <unset>
MountOptions:
  dir_mode=0777
  file_mode=0777
  uid=1001
  gid=1001
ReclaimPolicy:      Delete
VolumeBindingMode:  Immediate
Events:             <none>

$ date
Tue Jul 14 11:41:50 EDT 2020

$ kubectl describe sc azurefile
Name:            azurefile
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1beta1","kind":"StorageClass","metadata":{"annotations":{},"labels":{"kubernetes.io/cluster-service":"true"},"name":"azurefile"},"parameters":{"skuName":"Standard_LRS"},"provisioner":"kubernetes.io/azure-file"}

Provisioner:           kubernetes.io/azure-file
Parameters:            skuName=Standard_LRS
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>

Note how in a matter of 1 minute, all my MountOptions customisations get reverted back. I encounter the exact same behaviour trying to modify the default sc, so have to end up creating a new sc and patching all my resources accordingly.

danishsamad · 2020-09-14T04:48:24Z

I have the same issue. Changing the default storage class only lasts for a minute or so, reverting back to using the default storage class ending in two storage classes marked as default

sharkymcdongles · 2020-09-14T05:04:15Z

I finally solved this by switching to a nonterrible provider in GKE. Since the switch we have opened 0 issue tickets compared to the weekly ones we opened for AKS.

…

On Mon, 14 Sep 2020, 06:48 danishsamad, ***@***.***> wrote: I have the same issue. Changing the default storage class only lasts for a minute or so, reverting back to using the default storage class ending in two storage classes marked as default — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#118 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG6KEUJ4MWKTHKHPZAGLVJDSFWOCPANCNFSM4ELNFEOA> .

andyzhangx · 2020-09-23T07:43:49Z

Update: original storage class is set as addonmanager.kubernetes.io/mode=Reconcile, we are now switching to use addonmanager.kubernetes.io/mode=EnsureExists, the rollout would happen next Monday, it could be completed in around Mid Oct.
Try to use a non default storage class now is the workaround.

andyzhangx · 2020-10-14T08:48:36Z

With the fix roll out, here is the way how to change the default storage class on AKS:

prepare default storage class and save to your local env, e.g. default.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    kubernetes.io/cluster-service: "true"
  name: default
parameters:
  storageaccounttype: Premium_LRS
provisioner: disk.csi.azure.com
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate

Run following two commands consecutively:

kubectl delete sc default && kubectl apply -f default.yaml

check whether default storage class is changed:

kubectl get sc default -o yaml

You could also use same way to to change other built-in storage classes, make sure storageclass.beta.kubernetes.io/is-default-class: "true" is removed in the new storage class config.

Check whether your default class addonmanager setting is correct(should be EnsureExists other than Reconcile):

# k get sc default -o yaml | grep addonmanager
    addonmanager.kubernetes.io/mode: EnsureExists

andyzhangx · 2020-10-14T08:51:42Z

@palma21

rtyler changed the title ~~Upgrade should consider whether a default StorageClass has already been set.~~ Cannot set a different default storage class Jan 11, 2018

rtyler mentioned this issue Jan 11, 2018

Add a different Storage Class CodeValet/codevalet#84

Merged

seanknox closed this as completed Aug 2, 2018

andyzhangx reopened this Mar 11, 2019

jnoller added the known-issue label Apr 2, 2019

rummens mentioned this issue Jul 23, 2019

Update documentation to take care of two default storage classes after NFS Provisioner instalation kubeflow/website#964

Closed

thbkrkr mentioned this issue Oct 9, 2019

Update volumeBindingMode of the default storageClass on AKS elastic/cloud-on-k8s#1948

Closed

palma21 self-assigned this Apr 21, 2020

ghost added the action-required label Aug 11, 2020

andyzhangx closed this as completed Oct 14, 2020

ghost locked as resolved and limited conversation to collaborators Nov 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot set a different default storage class #118

Cannot set a different default storage class #118

rtyler commented Jan 11, 2018

rtyler commented Jan 11, 2018

rtyler commented Jan 11, 2018

andyzhangx commented Jan 12, 2018

rtyler commented Jan 12, 2018

rabenhorst commented Jan 15, 2018

andyzhangx commented Jan 15, 2018

jzakotnik commented Mar 26, 2018

jimmykarily commented Apr 17, 2018

seanknox commented Aug 2, 2018

kwaazaar commented Mar 11, 2019 •

edited

Loading

ddarling85 commented Apr 23, 2019

stevendstark commented Jul 31, 2019

huangyingting commented Oct 28, 2019

sharkymcdongles commented Nov 5, 2019

andyliddle commented Jan 8, 2020

gkaleta commented Feb 19, 2020

berndverst commented Apr 22, 2020

sblack4 commented May 12, 2020

andyzhangx commented May 13, 2020 •

edited

Loading

berndverst commented May 19, 2020

andyzhangx commented Jun 8, 2020 •

edited

Loading

DanielSCon40 commented Jul 1, 2020

andyzhangx commented Jul 3, 2020

egoienola commented Jul 17, 2020 •

edited

Loading

danishsamad commented Sep 14, 2020

sharkymcdongles commented Sep 14, 2020 via email

andyzhangx commented Sep 23, 2020

andyzhangx commented Oct 14, 2020 •

edited

Loading

andyzhangx commented Oct 14, 2020

Cannot set a different default storage class #118

Cannot set a different default storage class #118

Comments

rtyler commented Jan 11, 2018

rtyler commented Jan 11, 2018

rtyler commented Jan 11, 2018

andyzhangx commented Jan 12, 2018

rtyler commented Jan 12, 2018

rabenhorst commented Jan 15, 2018

andyzhangx commented Jan 15, 2018

jzakotnik commented Mar 26, 2018

jimmykarily commented Apr 17, 2018

seanknox commented Aug 2, 2018

kwaazaar commented Mar 11, 2019 • edited Loading

ddarling85 commented Apr 23, 2019

stevendstark commented Jul 31, 2019

huangyingting commented Oct 28, 2019

sharkymcdongles commented Nov 5, 2019

andyliddle commented Jan 8, 2020

gkaleta commented Feb 19, 2020

berndverst commented Apr 22, 2020

sblack4 commented May 12, 2020

andyzhangx commented May 13, 2020 • edited Loading

berndverst commented May 19, 2020

andyzhangx commented Jun 8, 2020 • edited Loading

DanielSCon40 commented Jul 1, 2020

andyzhangx commented Jul 3, 2020

egoienola commented Jul 17, 2020 • edited Loading

danishsamad commented Sep 14, 2020

sharkymcdongles commented Sep 14, 2020 via email

andyzhangx commented Sep 23, 2020

andyzhangx commented Oct 14, 2020 • edited Loading

andyzhangx commented Oct 14, 2020

kwaazaar commented Mar 11, 2019 •

edited

Loading

andyzhangx commented May 13, 2020 •

edited

Loading

andyzhangx commented Jun 8, 2020 •

edited

Loading

egoienola commented Jul 17, 2020 •

edited

Loading

andyzhangx commented Oct 14, 2020 •

edited

Loading