Enabling --non-masquerade-cidr option #1361

dbourasseau · 2017-08-28T09:39:05Z

What this PR does / why we need it:

This PR enables configuration of the --non-masquerade-cidr option at the kubelet level.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #1360

Special notes for your reviewer:

Release note:

This change is

lachie83 · 2017-08-29T17:38:45Z

parts/kuberneteskubelet.service

@@ -50,6 +50,7 @@ ExecStart=/usr/bin/docker run \
        --network-plugin=${KUBELET_NETWORK_PLUGIN} \
        --max-pods=${KUBELET_MAX_PODS} \
        --node-status-update-frequency=${KUBELET_NODE_STATUS_UPDATE_FREQUENCY} \
+        --non-masquerade-cidr=${KUBELET_NON_MASQUERADE_CIDR} \


I don't see this as a valid option on the documentation. Can you confirm? -- https://kubernetes.io/docs/admin/kubelet/

This page at least mentions it: https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/

Pretty sure I saw it somewhere on the kubelet options previously.

lachie83 · 2017-08-29T17:39:11Z

parts/kubernetesagentcustomdata.yml

@@ -115,6 +115,7 @@ write_files:
    KUBE_CTRL_MGR_POD_EVICTION_TIMEOUT={{WrapAsVariable "kubernetesCtrlMgrPodEvictionTimeout"}}
    KUBE_CTRL_MGR_ROUTE_RECONCILIATION_PERIOD={{WrapAsVariable "kubernetesCtrlMgrRouteReconciliationPeriod"}}
 {{if IsKubernetesVersionGe "1.6.0"}}
+    KUBELET_NON_MASQUERADE_CIDR={{WrapAsVariable "kubernetesNonMasqueradeCIDR"}}


Depending on what the option can take. Should this be an array of CIDR blocks?

dbourasseau · 2017-09-04T12:14:38Z

Any news ? Have you watched the pull request? must we do something else?

mrosterm · 2017-09-25T12:14:28Z

Whats the status on this one?

dbourasseau · 2017-09-25T14:50:19Z

We are still waiting for your response

SaaldjorMike · 2017-09-27T20:19:04Z

Linking this with issue #425

SaaldjorMike · 2017-09-27T20:26:16Z

According to https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/options/options.go#L339 --non-masquerade-cidr is deprecated. See more here kubernetes/kubernetes#46554

jackfrancis · 2017-10-03T21:57:20Z

Closing this as --non-masquerade-cidr is deprecated. Happy to iterate on a more current solution if you'd like to file an issue that describes the high order intention.

bathizte · 2017-10-04T14:15:27Z

Hello @jackfrancis

The high order intention is hopefully explained in #1360 linked issue.

We understand the option has been deprecated a month ago in master and will be removed in a future version but :

K8s deployments using pre-1.8 version still need some parts of this PR if deploying on a private subnet (unless we have missed something).
We may want to check that along with deprecating the option, a fix or an other mechanism has been commted to deploy a k8s cluster in a private network and disable masquerading on the pod cidr.
In the case a new option allows this, let's update the PR specifying the new option for Acs engine k8s v>=1.8 deployments.

jackfrancis · 2017-10-04T21:15:22Z

@bathizte We did some further investigation, and this is still the only way (as far as we know) to do this in v1.8, so we'll proceed. Thanks!

… in a network that has already used the 10.0.0.0/8, and therefore activated NAT of a pod to an address in 10.0.0.0/8

* clean * Review change

jackfrancis

lgtm

sylr · 2017-10-05T08:25:29Z

I had a similar PR for this (#1142) but I closed it because if I think I saw that the --non-masquerade-cidr option of kubelet is deprecated.

lachie83 · 2017-10-06T22:44:14Z

@jackfrancis when can we get this merged?

jackfrancis · 2017-10-06T22:54:04Z

Been having trouble getting it to pass Windows e2e, will re-poke.

jackfrancis

lgtm

karataliu · 2017-10-10T02:07:39Z

We'd better begin seeking for new solutions, as @sylr mentioned this flag is deprecated:

https://github.com/kubernetes/kubernetes/blob/v1.8.0/cmd/kubelet/app/options/options.go#L340

tanner-bruce · 2017-12-11T21:12:23Z

@jackfrancis @karataliu I just came across the ip-masq-agent today

EDIT
Confirmed: I tested ip-masq-agent on my ACS-Engine provisioned cluster.

I've edited my vnet to be 10.240.0.0/16. My Pod CIDR is 10.244.0.0/16. My third-party vnet is 10.128.0.0/16. With the default config, I was unable to talk to 10.128/16.

With this configmap, and the basic ip-masq-agent provided by the authors, everything works nicely:

apiVersion: v1
data:
  config: |
    nonMasqueradeCIDRs:
    - 10.244.0.0/16
    masqLinkLocal: false
    resyncInterval: 10s
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: ip-masq-config
  namespace: kube-system
  selfLink: /api/v1/namespaces/kube-system/configmaps/ip-masq-config

msftclas added cla-required and removed cla-required labels Aug 28, 2017

lachie83 reviewed Aug 29, 2017

View reviewed changes

jchauncey added area/engine area/templates needs approval size/small waiting for ci needs rebase labels Sep 12, 2017

msftgits removed the cla-signed label Sep 26, 2017

Azure deleted a comment from msftclas Sep 26, 2017

jackfrancis closed this Oct 3, 2017

jackfrancis mentioned this pull request Oct 4, 2017

Improve Custom VNet support #1150

Merged

jackfrancis reopened this Oct 4, 2017

ghost assigned jackfrancis Oct 4, 2017

ghost added the in progress label Oct 4, 2017

dbourasseau added 4 commits October 4, 2017 14:24

add kubernetes Non Masquerade CIDR in order to be able to install K8s…

e0d0cb5

… in a network that has already used the 10.0.0.0/8, and therefore activated NAT of a pod to an address in 10.0.0.0/8

gofmted

bcee096

rename CIDR to Cidr according to the exciting code

e77ce11

fix kubernetesNonMasqueradeCidr to KUBELET_NON_MASQUERADE_CIDR on master

a49e242

* clean * Review change

dbourasseau and others added 2 commits October 4, 2017 14:26

change CIDR to Cidr

abd6387

json lint

dac6a2a

jackfrancis previously approved these changes Oct 4, 2017

View reviewed changes

added NonMasqueradeCidr validation

7656388

jackfrancis dismissed their stale review via 7656388 October 4, 2017 21:52

documentation

5f6ab1c

jackfrancis mentioned this pull request Oct 4, 2017

Add nodeCidrMask parameter to allow running in a smaller IP vnet. #495

Closed

jackfrancis approved these changes Oct 9, 2017

View reviewed changes

jackfrancis merged commit 8281069 into Azure:master Oct 9, 2017

ghost removed the in progress label Oct 9, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enabling --non-masquerade-cidr option #1361

Enabling --non-masquerade-cidr option #1361

dbourasseau commented Aug 28, 2017 •

edited by acs-bot

Loading

lachie83 Aug 29, 2017

SaaldjorMike Aug 29, 2017

lachie83 Aug 29, 2017

dbourasseau commented Sep 4, 2017

mrosterm commented Sep 25, 2017

dbourasseau commented Sep 25, 2017

SaaldjorMike commented Sep 27, 2017

SaaldjorMike commented Sep 27, 2017 •

edited

Loading

jackfrancis commented Oct 3, 2017

bathizte commented Oct 4, 2017

jackfrancis commented Oct 4, 2017

jackfrancis left a comment

sylr commented Oct 5, 2017

lachie83 commented Oct 6, 2017

jackfrancis commented Oct 6, 2017

jackfrancis left a comment

karataliu commented Oct 10, 2017

tanner-bruce commented Dec 11, 2017 •

edited

Loading

Enabling --non-masquerade-cidr option #1361

Enabling --non-masquerade-cidr option #1361

Conversation

dbourasseau commented Aug 28, 2017 • edited by acs-bot Loading

lachie83 Aug 29, 2017

Choose a reason for hiding this comment

SaaldjorMike Aug 29, 2017

Choose a reason for hiding this comment

lachie83 Aug 29, 2017

Choose a reason for hiding this comment

dbourasseau commented Sep 4, 2017

mrosterm commented Sep 25, 2017

dbourasseau commented Sep 25, 2017

SaaldjorMike commented Sep 27, 2017

SaaldjorMike commented Sep 27, 2017 • edited Loading

jackfrancis commented Oct 3, 2017

bathizte commented Oct 4, 2017

jackfrancis commented Oct 4, 2017

jackfrancis left a comment

Choose a reason for hiding this comment

sylr commented Oct 5, 2017

lachie83 commented Oct 6, 2017

jackfrancis commented Oct 6, 2017

jackfrancis left a comment

Choose a reason for hiding this comment

karataliu commented Oct 10, 2017

tanner-bruce commented Dec 11, 2017 • edited Loading

dbourasseau commented Aug 28, 2017 •

edited by acs-bot

Loading

SaaldjorMike commented Sep 27, 2017 •

edited

Loading

tanner-bruce commented Dec 11, 2017 •

edited

Loading