Add flag to pass cluster CIDR to bootkube render

[dhawal55]

When I run bootkube to render assets, it assumes service-cluster-ip-range to be 10.3.0.0/24 and allocates etcd clusterIP and kube-dns clusterIP from that range. Make this as a parameter that can be passed to bootkube render.

Why does kube-proxy daemonset need to pass --cluster-cidr to hyperkube proxy image? Documentation says "It is used to bridge traffic coming from outside of the cluster". How does it work when using flanneld (its value is same as that set in etcd for /coreos.com/network/config Network value) and can this be made a parameter too so if not passed, it will not be set?

[aaronlevy]

The --cluster-cidr is so that the proxy can properly setup iptables MASQUERADE rules for the pod network (cluster cidr is for the pod-network, where service-cluster-ip-range is for the service IPs)

[dhawal55]

@aaronlevy thanks for clarifying this.

[aaronlevy]

From an offline discussion I wanted to track some of the plans for this:

The first step is going to be to change bootkube start to try and introspect the expected pod/service cidrs by directly reading the rendered control-plane templates. Assuming all assets have been rendered correctly, the internal bootkube control-plane will just use these same values (which protects us from having to make sure bootkube render and bootkube start both use the same flags -- which could lead to problems if you happen to use the wrong flags with existing assets).

Initially this just means that render will still output the same default values, but start will get those values from the rendered templates, rather than being hard-coded. @dghubble is going to start working on this soon.

The next step would be to update bootkube render to support custom pod/service CIDRs - and properly rendering those options into all affected components (apiserver, controller-manager, proxy, kubelet, dns, etc.) - then once that step is done, bootkube will just use those updated values.

As a note, some of the other options discussed:

• Rendering a temporary control plane as static pods, rather than using compiled-in components (Possibly use static pods for temporary control-plane rather than compiled-in components #168). Then just using the rendered templates directly (with all chosen values).

The issue with this approach is that it makes TLS bootstrapping difficult. The kubelet would start, then wait to be able to contact an api-server - but that api-server is a static pod waiting to be started by the kubelet. This would mean we would either need to do a 2 step process with the kubelet (let it start static pods, then kill it and let it start normally to do TLS bootstrap) - or mimic the TLS bootstrap functionality in bootkube itself. This seemed like a lot of new complexity which we don't need to immediately solve.

• Another option discussed was generating a config file which would be used by both render and start. This essentially would mean doing something like bootkube init, which would generate a config with default values. A user could then change those values, run bootkube render to generate all assets, then also run bootkube start which would use the same config file (so all state could be persisted between commands). This also seemed overly complex initially, even though we may want something like this in the future (similar discussed more in: Provide a generic mechanism for flag customizations #106)

/cc @sym3try @Quentin-M

[aaronlevy]

This should be closed by: #318

[dhawal55]

This is awesome!! Thanks