Question about size of etcd disk default size

[yaron-idan]

Is this a request for help?:
No

Is this an ISSUE or FEATURE REQUEST? (choose one):
ISSUE

What version of aks-engine?:
0.32.3

Kubernetes version:
1.11.8

What happened:
We realised clusters are turning up with 1024GB etcd disks, and then found out this is expected behaviour according to the docs, even though this seems like a very large disk for ~20 nodes.

What you expected to happen:
According to the etcd docs, 60GB is an adequate size for very large clusters. So I'm wondering what makes aks-enging provision such a large disk by default.
Furthermore, the etcd docs stress that the critical factor for these disks is speed, so I was also looking for the possibility of using SSD drives for etcd, but couldn't find this config in aks-engine api-model.

How to reproduce it (as minimally and precisely as possible):
Spin up a cluster with default values.

[welcome]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it.

[CecileRobertMichon]

For context, here's the PR that set those defaults: Azure/acs-engine#2435

@khenidak to see if he has any additional guidance.

[yaron-idan]

Not sure I'm following...
The PR seems to indicate that the bigger disks are allocated to provide better IOPS ratio, but looking at the pricing page for managed disks, it seems only SSD are getting better IOPS as their size increase - while HDD are always marked as up to 500.
Am I missine something here?

[zhiweiv]

In my k8s cluster, there are 7 nodes and about 60 pods, I allocated only 60G Premium SSD for ectd disk. In azure monitor, Avg Data Disk Queue Depth/Avg Data Disk Write Operations/Sec of ectd disk are very low.

Back to the doc https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/hardware.md, 1500~3000 IOPS could handle up to 50 nodes, that fits my case, 500 IOPS is enough for 5 nodes.

In Azure/acs-engine#2510, it used etcd 2.x, aks-e uses 3.x now, maybe it improved performance, I think aks-e may allocate too aggressive ectd disk size now.

[yaron-idan]

Thanks @zhiweiv, this is valuable input.
Can you please elaborate on how to assign premium SSD for etcd drives? I can't find that parameter in the aks configuration.

[zhiweiv]

I didn't specify it explicitly, I think if master VM's SKU supports premium storage, it will create Premium SSD etcd automatically.
Here is part of my template.

{
	"apiVersion": "vlabs",
	"properties": {
		"orchestratorProfile": {
			"orchestratorType": "Kubernetes",
			"orchestratorRelease": "1.13",
			"kubernetesConfig": {
				"etcdDiskSizeGB": "120"
			}
		},
		"masterProfile": {
			"count": 1,
			"dnsPrefix": "test",
			"vmSize": "Standard_B2s"
		}
	}
}

[yaron-idan]

Amazing, turns out we were using machines that don't support premium storage to run as masters. I'm gonna change it in our next cluster deployment. Thanks alot!