Mixed OS cluster attempting to schedule components on Windows nodes

[ghost]

Is this a request for help?:

Yes

Is this an ISSUE or FEATURE REQUEST? (choose one):

Issue

What version of acs-engine?:

All

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)

Kubernetes 1.10.4

What happened:

As many of you know, for acs-engine clusters with Linux and Windows nodes, we are in a constant state of whack-a-mole with adding nodeSelector: statements everywhere to try and get workloads not to schedule on Windows nodes - 99% of the time things are unable to run, and most solutions dont even have windows container images to run yet (Prometheus, Elasticsearch, Kibana, Grafana, etc etc).

What you expected to happen:

I expect there to be a way with acs-engine to configure the default node type from a specific agentpool for all resources to be scheduled by default. Is there away to achieve this through a scheduler default within Kubernetes?

How to reproduce it (as minimally and precisely as possible):

Deploy anything and dont define nodeSelector, you'll probably get scheduled on a Windows node and the container will fail to deploy.

Anything else we need to know:

[sam-cogan]

This isn't really an acs-engine issue, it's a Kubernetes one. It doesn't offer the functionality to select a default node.

What I have ended up doing is apply a taint to my windows nodes which prevents any pods being run unless I specify a toleration on them. Yes it does mean I have to add that toleration to my Windows pods, but at least I am only doing it on one OS, and it deals with things like helm packages that are not configured to allow a node selector.

[PatrickLang]

#3407 will fix some of the acs-engine addons that were missing node selectors.

For other workloads, I think you'll have to rely on either adding node selectors or using taints as Sam suggested. I usually go ahead and put the node selector in my helm charts.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contribution. Note that acs-engine is deprecated--see https://github.com/Azure/aks-engine instead.