Kubernetes CoreOS failed to start kubelet

[IvanovOleg]

Is this a request for help?:
YES

Is this an ISSUE or FEATURE REQUEST? (choose one):
ISSUE

What version of acs-engine?:
0.11.0

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes 1.8.4

What happened:
Unable to connect to the kubernetes cluster with CoreOS as a distro after deployment. Kubelet can't be started because of the file system permission issue:

Dec 20 14:32:24 k8s-master-XXXXXX-0 docker[2794]: /run/torcx/bin/docker: Error response from daemon: error while creating mount source path '/usr/libexec/kubernetes/kubelet-plugins': mkdir /usr/libexec/kubernetes: read-only file system.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):
{ "apiVersion": "vlabs", "properties": { "orchestratorProfile": { "orchestratorType": "Kubernetes", "orchestratorVersion": "1.8.4", "kubernetesConfig": { "networkPolicy": "azure", "enableRbac": true } }, "masterProfile": { "count": 1, "dnsPrefix": "yourdnsprefix", "vmSize": "Standard_D2_v2", "storageProfile" : "ManagedDisks", "distro": "coreos" }, "agentPoolProfiles": [ { "name": "agentpool1", "count": 3, "vmSize": "Standard_DS2_v2", "availabilityProfile": "AvailabilitySet", "storageProfile" : "ManagedDisks", "distro": "coreos" } ], "linuxProfile": { "adminUsername": "admin", "ssh": { "publicKeys": [ { "keyData": "" } ] } }, "servicePrincipalProfile": { "clientId": "", "secret": "" } } }

Anything else we need to know:
Tried several times

[IvanovOleg]

UPD1: I've removed the plugins volume line from the kubelet.service and it works.
UPD2:
Dec 20 15:45:19 k8s-agentpool1-39482401-3 docker[1124]: 'docker daemon' is not supported on Linux. Please run 'dockerd' directly

On agent node docker daemon doesn't start(((

UPD3: Copied the content of the exec_start.conf from master node and it fixed the error.
UPD4: Agent is not ready without that plugins line((( With that line the read-only error.

[andyzhangx]

@jackfrancis does our acs-engine support coreos too? I have checked, In CoreOS the /usr partition is read-only by design

[mwieczorek]

@andyzhangx It was supported before #1909 :)
There's some examples: https://github.com/Azure/acs-engine/tree/master/examples/coreos
Can we change the folder from:

--volume=/usr/libexec/kubernetes/kubelet-plugins:/usr/libexec/kubernetes/kubelet-plugins

to

--volume=/etc/kubernetes/plugins:/etc/kubernetes/plugins

and pass it to kubelet as parameter:

--volume-plugin-dir=/etc/kubernetes/plugins

instead of using the default?

[andyzhangx]

@mwieczorek Thanks for the suggestion. Sounds like there is a trade-off if we also want to support coreos, we need to use /etc/kubernetes/plugins instead of standard plugin dir /usr/libexec/kubernetes/kubelet-plugins for all linux distros, because we use only one
https://github.com/Azure/acs-engine/blob/master/parts/k8s/artifacts/kuberneteskubelet.service for all linux distros.
@jackfrancis @JiangtianLi what's your opinion?
Flexvolume doc: https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md#prerequisites

[andyzhangx]

I would like to change the default flexvolume plugin dir in Linux to /etc/kubernetes/volumeplugins and in Windows, the plugin dir would be c:\k\volumeplugins

[richardjortega]

I will try and look into this soon, I integrated CoreOS a while back but think it needs updates after Azure CNI was introduced. You can put stuff in /opt, which is what the CoreOS implementation does (puts kubelet in /opt) — I believe unless it was changed recently.

[andyzhangx]

@richardjortega, I made a PR#1991 to set flexvolume plugin dir as /etc/kubernetes/volumeplugins in Linux.
BTW, have anyone tried coreos examples under https://github.com/Azure/acs-engine/tree/master/examples/coreos, it never succeeded, last error was due to docker could not start up in coreos, is there any fix to acs-engine?

[richardjortega]

@andyzhangx I wrote those CoreOS examples. I haven’t tried them in about a month or two so things could’ve changed that sideaffected it.

[IvanovOleg]

@andyzhangx If you get 'docker daemon' is not supported on Linux. Please run 'dockerd' directly on the agent nodes, just copy exec_start.conf from master.

[scher200]

I tried to deploy with kubespray coreos and noticed my kubelet_flexvolumes_plugins_dir: /var/lib/kubelet/volumeplugins variable was not picked up during deployment.

In every deployed host, I had to edit /etc/kubernetes/kubelet.env
and add this line:

KUBELET_VOLUME_PLUGIN="--volume-plugin-dir=/var/lib/kubelet/volumeplugins"

after that run systemctl restart kubelet

Working now