[kube-batch] task still dispatched to the node marked with 'SchedulingDisabled'

[runqch]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

What happened: task still dispatched to the node marked with 'SchedulingDisabled'

What you expected to happen: task should NOT dispatched to the node marked with 'SchedulingDisabled'

How to reproduce it (as minimally and precisely as possible):

1. kubectl cordon ib22b12 to mark ib22b12 as SchedulingDisabled
2. submit job, requiring slots more than the avaiable slots in cluster
3. check pod running status, will found pod still runs on node marked with SchedulingDisabled.

 runqch@ib22b10-779: kubectl get nodes
NAME      STATUS                     ROLES     AGE       VERSION
ib22b10   Ready                      master    36d       v1.11.3
**ib22b12   Ready,SchedulingDisabled**   <none>    36d       v1.11.3
ib22b13   Ready                      <none>    36d       v1.11.3

runqch@ib22b10-777: cat job1.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: qj-1
spec:
  backoffLimit: 120
  completions: 120
  parallelism: 120
  template:
    metadata:
      annotations:
        scheduling.k8s.io/group-name: qj-1
    spec:
      containers:
      - image: busybox
        imagePullPolicy: IfNotPresent
        name: busybox
        command:
           - sleep
           - "1000"
        resources:
          requests:
            cpu: "1"
      restartPolicy: Never
      schedulerName: kube-batch
---
apiVersion: scheduling.incubator.k8s.io/v1alpha1
kind: PodGroup
metadata:
  name: qj-1
spec:
  minMember: 1 

 runqch@ib22b10-772: kubectl create -f ./job1.yaml
job.batch/qj-1 created

 runqch@ib22b10-775: kubectl get pods -o wide | grep Running | grep ib22b12
qj-1-27jqj   1/1       Running   0          1m        192.168.1.22   ib22b12   <none>
qj-1-2cjqd   1/1       Running   0          1m        192.168.1.6    ib22b12   <none>
qj-1-2v42l   1/1       Running   0          1m        192.168.1.21   ib22b12   <none>
qj-1-48vtz   1/1       Running   0          1m        192.168.1.8    ib22b12   <none>
qj-1-5m7bf   1/1       Running   0          1m        192.168.1.14   ib22b12   <none>
qj-1-8n8x8   1/1       Running   0          1m        192.168.1.20   ib22b12   <none>
qj-1-8qsfz   1/1       Running   0          1m        192.168.1.32   ib22b12   <none>
qj-1-bbvqg   1/1       Running   0          1m        192.168.1.11   ib22b12   <none>
qj-1-bgn76   1/1       Running   0          1m        192.168.1.30   ib22b12   <none>
qj-1-c7sqf   1/1       Running   0          1m        192.168.1.15   ib22b12   <none>

Tried default scheduler, pod will not running on SchedulingDisabled node.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[k82cn]

Which k8s version do we use? We did not check node.spec.Unschedulable right now; after enabled TaintNodeByCondtion, ScheduleDisabled node will be tainted with unschedule:NoSchedule and then kube-batch will not schedule pod to it. If we want to support node.Spec.Unschedulable, we need a PR to check it explicitly.

[k82cn]

@jeefy , would you also help to add an e2e test for it?

[jeefy]

Definitely! I'll PR that soon.