Multi-node Prototype

[pbitty]

This is a working prototype I made (not to be merged), inspired by #94. I am very interested in adding this feature. If it is something that the maintainers are interested in adding, I'd love to work to work out the design with whoever is interested and help implement it.

I've made notes on the prototype in docs/multi-node-prototype.md.

The code in this branch is hacky, but it does work (on my machine at least😄 ).

I'm not sure if this is the best way to discuss the feature, but I figure it's a way to start off the discussion, and if there is interest I could submit a more specific design doc/proposal.

Here's a demo:

[miry]

@pbitty It seems go fmt was not run for this file.

[pbitty]

@afbjorklund , thanks for the feedback.

is the basic contract (run a cluster on your laptop) still fulfilled (minikube start), and the number of nodes that you want is just yet another parameter ?

I would lean toward this option, as to not complicate the experience for single-node users. My assumption is that most of the time people will be operating in single-node mode, so it should probably remain the core focus of the minikube UX.

My suggestion would be to start this out small, as an experimental feature and see how it gets used.

the various hostpath solutions are not going to cut it anymore in a multi-node solution...

Please tell me more. I am not sure why the current solutions will not cut it.

I think that this feature can require kubeadm and "libvirt"

Sounds good to me. If this an experimental feature, it can be limited in scope and support, and grow as needed.

Not even sure if it is worth keep doing the minikube.iso distro, or if it should move to a standard one ?

I don't think a change in distro would be needed for this feature. Currently it only requires an image that runs systemd and docker (and maybe some subtle details I am overlooking). Everything else is provided by the kubeadm bootstrapper. For simplicity, we might want to keep the choice of distro as a separate (but related?) issue.

[afbjorklund]

You are going to need distributed storage, what if your upgraded pod ends up on another node ? Then all host paths are "gone". You might even need a registry

Ignore the comment about the distro, I was just chasing down yet another bug specific to minikube and was annoyed... Those other images are still a lot larger, by default

[pbitty]

You are going to need distributed storage, what if your upgraded pod ends up on another node ? Then all host paths are "gone". You might even need a registry.

Ah, I understand - you are referring to hostPath volume type, right? I had assumed "host path" to mean the path on the host OS that minikube normally mounts on the VMs. eg. /Users on MacOS. If all VMs have access to a folder on the host, that could be the shared location, at least as an initial solution.

[r2d4]

@pbitty Thanks for the PR. It is much appreciated. I'll give it a deep look soon. We should also set up some time to discuss this in more depth, since its something we'd be interested in.

[afbjorklund]

@pbitty : I think a good start would be to have it work just-as-it-does-now for the master node. And then add some kind of addons, to make it possible to share storage and images to the worker nodes...
For instance by running an NFS server and a Registry server, on the master node. This way, all the current workflows would remain as-is while still providing a way to get away from those anti-patterns(*)

• with k8s, you are not supposed to map your home directory or to push things to the docker daemons

So once you grow out of single-node clusters, you will have to move to persistent volumes and registry ?

[nebril]

I am trying to run this, but I ran into several problems with starting nodes:

• I need to manually disable swap on node vms, as kubeadm join demands this
• I need to manually add --discovery-token-unsafe-skip-ca-verification to kubeadm join, otherwise I get discovery: Invalid value: "": using token-based discovery without DiscoveryTokenCACertHashes can be unsafe. set --discovery-token-unsafe-skip-ca-verification to continue error

Still have some kind of cert error, investigating.

[pbitty]

@nebril I only tested this with kubernetes-version: v1.8.0. Is this the version you are using?

[pbitty]

@r2d4 If you haven't started looking at this yet, I have a cleaner proposal doc coming in another PR.
I would love to schedule a time to chat about this. Sometime next week would be great. We can coordinate by email.

[afbjorklund]

kubeadm join probably needs the same kind of workarounds as kubeadm init ?

	// We use --ignore-preflight-errors=DirAvailable since we have our own custom addons
	// that we also stick in /etc/kubernetes/manifests
	// We use --ignore-preflight-errors=Swap since minikube.iso allocates a swap partition.
	// (it should probably stop doing this, though...)

See #2403 and --ignore-preflight-errors=all

[nebril]

@pbitty thanks for pointing this out, I was trying to run 1.9.

With 1.8 I also keep getting
[discovery] Failed to request cluster info, will try again: [Unauthorized] error from kubeadm join.

[nebril]

Ok, I messed up, I started master node with localkube bootstrapper. Starting master node with --kubernetes-version v1.8.0 --bootstrapper kubeadm options worked for me. Thanks for this PR!

[afbjorklund]

I tried to update the code to later versions, think the code is somewhat OK but some tests are still failing:

https://github.com/afbjorklund/minikube/tree/multi-node

Most of the original comments still hold, and there's some additional discussions in the proposal.

[christopher-avila]

Any news on this?
Looks promising! :)

[afbjorklund]

@christopher-avila : I updated my branch, but neither me nor @pbitty have time to continue with it...

[Albertmoiseev]

Really wish to see it completed&merged someday...

[pbitty]

Hi All, I'm really sorry for the long silence. I built this prototype pretty quickly (and messily) to see how it would look. After chatting about it more with @r2d4 and @afbjorklund, I realized that working with the community to implement it and maintain it would take up much more time than I have available.

If anyone is interested in taking this on, I suggest talking about it with @r2d4 . I am happy to contribute ideas and give my own feedback as well

Sorry if I wasted anyone's time. I am closing this for now.

[afbjorklund]

@pbitty: thank you for your effort!

I tried to merge the previous code with v0.29.0 (5e1776b), but now the tests don't pass anymore...

The previous versions (3cf034a 27fdea4 ccb1fce a6f5e77) were in a happier place, but if you are going to add multi-node support to minikube now it is probably better to start over with the implementation.

I don't think I will spend any more time on this, if you are interested in doing multiple nodes you are probably better off without minikube: provision the machines yourself and just run kubeadm on them yourself ?

Documentation for doing that is at: https://kubernetes.io/docs/setup/independent/install-kubeadm/

[pbitty]

@afbjorklund , sorry I missed your last message for some reason. Thanks for the reply. If I need multi-node before it is supported, I'll definitely do that. Sorry I couldn't follow through with a clean implementation.

[afbjorklund]

@pbitty : no worries, I just meant that a small typo in the help text is probably not what is left for multi-node. Thank you for initial implementation, and we still have multi-node on the roadmap even if it has competition:

• VM-less (0 VM, run all pods and containers on the host)
• Multi-node (1 VM per node, all virtual machines on host)
• Kata containers (1 VM per pod, or even 1 per container*)

* based on the trust level and other configuration, see docs

I personally think that multi-node is one of the most important aspects when learning Kubernetes, and if we can provide that locally (without disturbing the single-node workflow) it would be excellent! As per #94.