Tweaks to network config

[cknowles]

Fixes #1407.

For now just document the podCIDR aspects but we should probably switch the default based on the networking setup, possibly combine this config with the selfHosting config into a networking section.

[cknowles]

@davidmccormick I cannot assign you for review but worth a look.

[codecov-io]

Codecov Report

Merging #1408 into master will increase coverage by 0.73%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1408      +/-   ##
==========================================
+ Coverage   37.53%   38.26%   +0.73%     
==========================================
  Files          74       74              
  Lines        4534     4565      +31     
==========================================
+ Hits         1702     1747      +45     
+ Misses       2597     2576      -21     
- Partials      235      242       +7

Impacted Files	Coverage Δ
model/node_pool_config.go	20.58% <0%> (ø)	⬆️
model/custom_file.go	64.51% <0%> (ø)	⬆️
model/iamconfig.go	0% <0%> (ø)	⬆️
cfnstack/provisioner.go	0% <0%> (ø)	⬆️
core/controlplane/config/config.go	63.62% <0%> (+0.04%)	⬆️
core/controlplane/cluster/cluster.go	50% <0%> (+0.69%)	⬆️
core/nodepool/cluster/cluster.go	46.51% <0%> (+1.27%)	⬆️
model/etcd_cluster.go	26.31% <0%> (+26.31%)	⬆️
model/etcd.go	34.02% <0%> (+34.02%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 77c43b5...b63a183. Read the comment docs.

[cknowles]

@mumoshu please hold off on merging this, something is causing nodes in a couple of our clusters to shift into SchedulingDisabled status a bit at random and not yet sure if it's this change or the one in #1397. I believe it to be this change because we've been running the other PR in two clusters for a week without issue and it seems to only be when we rolled this change that nodes starting becoming randomly unhealthy. Note, this is even with a totally fresh stack (as we did change the pod CIDR change so wanted a fresh one).

[cknowles]

@mumoshu I believe we identified the cause of my noted issues above as unrelated now. I've corrected my cluster.yaml notes according to @davidmccormick's feedback in #1407. I think this is ready to merge.

[mumoshu]

LGTM. Thank you very much for sharing your hard-won insights 🙏