scheduler should take action when receiving TASK_LOST for REASON_SLAVE_REMOVED

[jdef]

the mesos master has given up on the slave at this point and if the slave process starts up again on the same node it will get a new slave ID. all prior tasks are recorded as LOST so the scheduler should delete the related pods so that they can be rescheduled

[jdef]

found while debugging #778

[jdef]

should also delete any mirror pods associated with the slave

[jdef]

fix: kubernetes/kubernetes#21366

[jdef]

I've tried deleting the mirror pods, and they do go away but if...

1. the slave process dies --> mesos fires slave-lost
2. the slave stays dead
3. the scheduler pod GC deletes all pods associated with {host,slaveID}
4. there are static pods running on the kubelet

the kubelet-executor will stay running until suicide timeout. this argues in favor of a lower value for the default suicide timeout threshold. #465

[jdef]

FWIW the apiserver has this status about the node (I'm wondering if the replication controller's pod GC is supposed to kick in here?):

status:
  addresses:
  - address: 10.2.0.6
    type: LegacyHostIP
  - address: 10.2.0.6
    type: InternalIP
  allocatable:
    cpu: "2"
    memory: 3745Mi
    pods: "40"
  capacity:
    cpu: "2"
    memory: 3745Mi
    pods: "40"
  conditions:
  - lastHeartbeatTime: 2016-02-17T03:48:23Z
    lastTransitionTime: 2016-02-17T03:44:02Z
    message: mesos reports ready status 
    reason: SlaveReady
    status: "True"
    type: Ready
  - lastHeartbeatTime: 2016-02-17T01:39:40Z
    lastTransitionTime: 2016-02-17T03:44:01Z
    message: Kubelet stopped posting node status.
    reason: NodeStatusUnknown
    status: Unknown
    type: OutOfDisk

[jdef]

we need to track additional mesos state as a node condition. upstream doesn't provide a mechanism with which we can inject such a condition. I filed a PR to add such: kubernetes/kubernetes#21521

• waiting for merge of the above
• MERGED on 2016-02-24