virtual nodes requires explicit pod command and dns configs

[mgsnuno]

What happened:
dask deployment in azure aks kubernetes cluster with virtual nodes (virtual-kubelet) fails.

Minimal Complete Verifiable Example:

import dask
from dask.distributed import Client
from dask_kubernetes import KubeCluster, KubeConfig, make_pod_spec

dask.config.set({"kubernetes.scheduler-service-type": "LoadBalancer"})
dask.config.set({"distributed.comm.timeouts.connect": 180})
image = "daskdev/dask"
cluster = "aks-cluster-prod3"
virtual_config = {
    "nodeSelector": {
        "kubernetes.io/role": "agent",
        "beta.kubernetes.io/os": "linux",
        "type": "virtual-kubelet",
    },
    "tolerations": [
        {"key": "virtual-kubelet.io/provider", "operator": "Exists"},
        {"key": "azure.com/aci", "effect": "NoSchedule"},
    ],
    # "dnsConfig": {
    #     "options": [{"name": "ndots", "value": "5"}],
    #     "searches": [
    #         "default.svc.cluster.local",
    #         "svc.cluster.local",
    #         "cluster.local",
    #     ],
    # },
}

# copied from: https://github.com/dask/dask-docker/blob/master/base/Dockerfile#L25
command_entrypoint_explicit = {
    "command": ["tini", "-g", "--", "/usr/bin/prepare.sh"],
}

pod_spec = make_pod_spec(
    image=image,
    extra_pod_config=virtual_config,
    extra_container_config=command_entrypoint_explicit,
    memory_limit="2G",
    memory_request="2G",
    cpu_limit=1,
    cpu_request=1,
    threads_per_worker=1,  # same as cpu
)

# az aks get-credentials --name aks-cluster1 --resource-group resource_group1
# cp ~/.kube/config ./aksconfig.yaml
auth = KubeConfig(config_file="./aksconfig.yaml", context=cluster,)
cluster = KubeCluster(
    pod_spec,
    auth=auth,
    deploy_mode="remote",
    scheduler_service_wait_timeout=180,
    n_workers=1,
)
client = Client(cluster)

Explanation
As explained here there are two problems:

1. in the pod configuration azure ACI expects command + args and dask only supplies args.
2. dns name resolution in virtual-kubelet is slightly different and causes workers not being able to resolve the scheduler location by tcp://{name}.{namespace}:{port}

To fix 1 I explicitly added the entrypoint command as in here:

command_entrypoint_explicit = {
    "command": ["tini", "-g", "--", "/usr/bin/prepare.sh"],
}

To fix 2 there are two options:

• edit the line bellow into: tcp://{name}.{namespace}.svc.cluster.local:{port}
  

  dask-kubernetes/dask_kubernetes/core.py

  Line 180 in b7ccae7

  self.address = "tcp://{name}.{namespace}:{port}".format(

  
  I verified that this works both for virtual and VM deployments.
  References: AKS Connector KubeDNS Resolution and Service IP resolution is not working virtual-kubelet/virtual-kubelet#456 (comment), ACI pod cannot communicate with other pods Azure/AKS#868 (comment), virtual-kubelet has different DNS resolution behaviour to standard kubelet  virtual-kubelet/azure-aci#46

• use the dnsConfig options in virtual_config above

Question
What is the best way to address this and incorporate this into dask-kubernetes so that the only thing one needs to do to run using virtual nodes is:

virtual_config = {
    "nodeSelector": {
        "kubernetes.io/role": "agent",
        "beta.kubernetes.io/os": "linux",
        "type": "virtual-kubelet",
    },
    "tolerations": [
        {"key": "virtual-kubelet.io/provider", "operator": "Exists"},
        {"key": "azure.com/aci", "effect": "NoSchedule"},
    ],
}

The references mentioned seem to indicate that the proper naming is tcp://{name}.{namespace}.svc.cluster.local:{port}, should this be default?

Also while developing this it was hard to find in the documentation that we can have a fully remote deployment of workers and scheduler by setting dask.config.set({"kubernetes.scheduler-service-type": "LoadBalancer"}) with deploy_mode="remote".

Versions

• Dask version: 2.21.0
• Python version: 3.8.3
• Operating System: linux (manjaro)
• Install method (conda, pip, source): conda

[jacobtomlinson]

Thanks for taking this time to raise this @mgsnuno. It is always nice to read a well-written issue.

in the pod configuration azure ACI expects command + args and dask only supplies args.

We should update make_pod_spec to take a command kwarg and set it with an appropriate default.

The references mentioned seem to indicate that the proper naming is tcp://{name}.{namespace}.svc.cluster.local:{port}, should this be default?

Happy for that to be the default.

It's a shame that virtual kubelet doesn't honour tcp://{name}.{namespace}. I feel having the DNS search set correctly is a reasonable assumption. But providing the full address for clusters who do not configure the search domain automatically shouldn't cause issues elsewhere.

Also while developing this it was hard to find in the documentation that we can have a fully remote deployment of workers and scheduler by setting dask.config.set({"kubernetes.scheduler-service-type": "LoadBalancer"}) with deploy_mode="remote".

This is a pretty experimental feature. It's still early days but I agree the documentation is rather lacking at this point.

@mgsnuno do you have any interest in raising PRs to address these issues?

[mgsnuno]

Thank you @jacobtomlinson

After some more digging virtual-kubelet/azure-aci#8 seems to fix an issue so that we don't need to explicitly define DNS search in dnsConfig.

I agree with not changing tcp://{name}.{namespace} and having the issue fixed in virtual-kubelet. For now I've been using dnsConfig to set the search and sometimes it works and sometimes doesn't. I'll update on this.

We should update make_pod_spec to take a command kwarg and set it with an appropriate default.

I got the default from dask's Dockerfile, not sure how to dynamically insert it in make_pod_spec instead of hardcoded. Suggestions?

Sure, I can PR the documentation and make_pod_spec. Any pointers for the documentation?

[jacobtomlinson]

I agree with not changing tcp://{name}.{namespace}

I am in favour of changing. I was just saying we shouldn't really have to. But it doesn't seem like there is a downside to doing so.

I got the default from dask's Dockerfile, not sure how to dynamically insert it in make_pod_spec instead of hardcoded. Suggestions?

Hardcoded is fine.

Sure, I can PR the documentation and make_pod_spec. Any pointers for the documentation?

Setting deploy_mode="remote" is similar to setting the deploy_mode in dask-yarn. You may wish to refer to that documentation.

Instead of creating a scheduler locally in the Python process it launches a pod for the scheduler and creates a service for clients and workers to connect to. So the client and workers will need to be able to connect to the service. As you suggested setting the service type to LoadBalancer is one way of exposing it.

[mgsnuno]

As you suggested setting the service type to LoadBalancer is one way of exposing it.

What are other ways of exposing it?

I am in favour of changing. I was just saying we shouldn't really have to. But it doesn't seem like there is a downside to doing so.

I've been working with it for the last 2 weeks and I randomly have DNS problems, it doesn't matter if I use the dnsConfig or change the scheduler address explicitly. When virtual-kubelet/azure-aci#8 gets merged, dnsConfig won't be necessary even so I think we wait and do not change.

We should update make_pod_spec to take a command kwarg and set it with an appropriate default.

I'll see how to best implement it. args are not really exposed anyway, should we expose both?

[jacobtomlinson]

What are other ways of exposing it?

Kubectl proxy and node ports are two common ways. Various service meshes may also have their own way of doing things.

args are not really exposed anyway, should we expose both?

Sure

[jacobtomlinson]

The classic KubeCluster was removed in #890. All users will need to migrate to the Dask Operator. Closing.