Add comment clarifying network allocation and sizes

[mikejoh]

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:
In the inventory/sample/groups_vars/k8s-cluster/k8s-cluster.yml there's a typo that may lead to confusion regarding the kube_network_node_prefix and how it relates to the kube_pods_subnet. This was fixed as seen here:

kubespray/roles/kubespray-defaults/defaults/main.yaml

Lines 100 to 110 in 240d419

	# internal network node size allocation (optional). This is the size allocated
	# to each node on your network. With these defaults you should have
	# room for 64 nodes with 254 pods per node.
	# Example: Up to 256 nodes, 100 pods per node (/16 network):
	# - kube_service_addresses: 10.233.0.0/17
	# - kube_pods_subnet: 10.233.128.0/17
	# - kube_network_node_prefix: 25
	# Example: Up to 4096 nodes, 100 pods per node (/12 network):
	# - kube_service_addresses: 10.192.0.0/13
	# - kube_pods_subnet: 10.200.0.0/13
	# - kube_network_node_prefix: 25

This PR adds the comment from defaults/main.yml to the sample k8s-cluster.yml file.

Which issue(s) this PR fixes:

N/A

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

[k8s-ci-robot]

Hi @mikejoh. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[floryut]

/ok-to-test
/lgtm

[EppO]

I'm really confused by the docs.

This is the size allocated to each node on your network.

for pod allocation! It's used for the controller-manager to allocate per-node CIDRs from the cluster CIDR for pod networking (when --allocate-node-cidrs=true).

Example: Up to 256 nodes, 100 pods per node (/16 network)
Example: Up to 4096 nodes, 100 pods per node (/12 network)

max pods per node depends on the Pod subnet size and kubelet_max_pods (which is 110 as default today), but It's not 100 in both cases.

[mikejoh]

I'm really confused by the docs.

This is the size allocated to each node on your network.

for pod allocation! It's used for the controller-manager to allocate per-node CIDRs from the cluster CIDR for pod networking (when --allocate-node-cidrs=true).

Example: Up to 256 nodes, 100 pods per node (/16 network)
Example: Up to 4096 nodes, 100 pods per node (/12 network)

max pods per node depends on the Pod subnet size and kubelet_max_pods (which is 110 as default today), but It's not 100 in both cases.

Thansk for the feedback!

You're right, it's confusing when reading it again and given that kube_max_pods isn't even considered (or rather mentioned) doesn't make it any clearer. I also think it's interesting to dig into the hard limit of 110 Pods per node, here's some insight into why that is.

Not sure how to rewrite this at the moment, i mean tuning the kube_max_pods to maximize the number of addresses in e.g. a /24 might still not be what we want to do, or recommend without a disclaimer. Maybe it's better to add a clarifying doc or point to a official one over at kubernetes.io.

Also you mentioned the --allocate-node-cidrs option, which i believe also is in-scope here. To maybe add something about, but in the context of kubespray that might be a corner case. Everything configured today, as defaults, more or less assumes you have a cloud somewhere and that you want to use one of the network CNI plugins.

[EppO]

I think it would be more precise to say "up to x or kubelet_max_pods, whatever the smaller is" or something more constructed. kubelet_max_pods is configurable today in kubespray so you may change that depending on your horsepower.

[mikejoh]

I think it would be more precise to say "up to x or kubelet_max_pods, whatever the smaller is" or something more constructed. kubelet_max_pods is configurable today in kubespray so you may change that depending on your horsepower.

Right, so maybe something like this:

# internal network node size allocation (optional). This is the size allocated
# to each node for pod IP address allocation. Note that the number of pods per node is
# also limited by the kubelet_max_pods variable which defaults to 110.
#
# Example: 
# Up to 64 nodes and up to 254 or kubelet_max_pods (the lowest of the two) pods per node:
#  - kube_pods_subnet: 10.233.64.0/18
#  - kube_network_node_prefix: 24
#  - kubelet_max_pods: 110
#
# Example:
# Up to 128 nodes and up to 126 or kubelet_max_pods (the lowest of the two) pods per node:
#  - kube_pods_subnet: 10.233.64.0/18
#  - kube_network_node_prefix: 25
#  - kubelet_max_pods: 110

[EppO]

I think it would be more precise to say "up to x or kubelet_max_pods, whatever the smaller is" or something more constructed. kubelet_max_pods is configurable today in kubespray so you may change that depending on your horsepower.

Right, so maybe something like this:

# internal network node size allocation (optional). This is the size allocated
# to each node for pod IP address allocation. Note that the number of pods per node is
# also limited by the kubelet_max_pods variable which defaults to 110.
#
# Example: 
# Up to 64 nodes and up to 254 or kubelet_max_pods (the lowest of the two) pods per node:
#  - kube_pods_subnet: 10.233.64.0/18
#  - kube_network_node_prefix: 24
#  - kubelet_max_pods: 110
#
# Example:
# Up to 128 nodes and up to 126 or kubelet_max_pods (the lowest of the two) pods per node:
#  - kube_pods_subnet: 10.233.64.0/18
#  - kube_network_node_prefix: 25
#  - kubelet_max_pods: 110

way better, thanks a lot!

[mikejoh]

Note that i've also changed the comment in the roles/kubespray-defaults/defaults/main.yaml file.

[woopstar]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mikejoh, woopstar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [woopstar]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment