This repository has been archived by the owner on Jan 11, 2023. It is now read-only.
Documentation about how to run Kubernetes in an hybrid env #2804
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CecileRobertMichon
suggested changes
Apr 30, 2018
docs/README.md
Outdated
| @@ -20,3 +20,4 @@ This cluster definition examples demonstrate how to create a customized Docker E | |||
| * [Attached Disks](../examples/disks-storageaccount) - shows how to attach up to 4 disks per node | |||
| * [Managed Disks](../examples/disks-managed) (under private preview) - shows how to use managed disks | |||
| * [Large Clusters](../examples/largeclusters) - shows how to create cluster sizes of up to 1200 nodes | |||
| * [Running Kubernetes in an hybrid environment](kubernetes/hybrid-environment.md) - considerations around how you will configure your cluster, and plan your deployment when running Kubernetes in a Cloud/On-Premise environment | |||
There was a problem hiding this comment.
Should be in a hybrid environment
|
|
||
| Note : We do not cover a Kubernetes cluster _spanning_ a hybrid cloud network, but rather how to run a cluster in the cloud and interact with an existing local network seamlessly. | ||
|
|
||
|  |
There was a problem hiding this comment.
in a hybrid cloud network
|
|
||
| ### Topology | ||
|
|
||
| The network topology must be well defined beforehand to enable peering between the different VNET. This means that the subnet ip range musty be defined before deploying kubernetes. It cannot be changed afterwards. |
There was a problem hiding this comment.
typo here musty --> must
| In a hybrid environment, you usually want to integrate with your on-premises DNS. There is two aspects to this. The first one is to register the VMs forming the cluster, and using your local search domain when resolving other services. The second is getting the services running on Kubernetes to use the external DNS. | ||
| To benefit the scaling capabilities of the cluster and to ensure resiliency to machine failure, every node configuration needs to be scripted and part of the initial template that acs-engine will deploy. To register the nodes in your DNS at startup, you need to define [an acs-engine extension](../extensions.md) that will run your [dns registration script](https://github.com/tesharp/acs-engine/blob/register-dns-extension/extensions/register-dns/v1/register-dns.sh). | ||
|
|
||
| In addition, you might want cluster services to address urls outside the cluster using your on-premise dns. To achieve this you need to configure KubeDNS to use your existing nameservice as upstream. [This setup is well documented on kubernetes blog](https://kubernetes.io/blog/2017/04/configuring-private-dns-zones-upstream-nameservers-kubernetes) |
| In a hybrid environment, you usually want to integrate with your on-premises DNS. There is two aspects to this. The first one is to register the VMs forming the cluster, and using your local search domain when resolving other services. The second is getting the services running on Kubernetes to use the external DNS. | ||
| To benefit the scaling capabilities of the cluster and to ensure resiliency to machine failure, every node configuration needs to be scripted and part of the initial template that acs-engine will deploy. To register the nodes in your DNS at startup, you need to define [an acs-engine extension](../extensions.md) that will run your [dns registration script](https://github.com/tesharp/acs-engine/blob/register-dns-extension/extensions/register-dns/v1/register-dns.sh). | ||
|
|
||
| In addition, you might want cluster services to address urls outside the cluster using your on-premise dns. To achieve this you need to configure KubeDNS to use your existing nameservice as upstream. [This setup is well documented on kubernetes blog](https://kubernetes.io/blog/2017/04/configuring-private-dns-zones-upstream-nameservers-kubernetes) |
|
|
||
| ### Network | ||
|
|
||
| Deploying Acs-engine on azure, you have 3 options of network policy. Azure CNI, Kubenet, or Calico. |
There was a problem hiding this comment.
Acs-engine --> ACS-Engine
|
|
||
| #### Azure CNI | ||
|
|
||
| By default, acs-engine is using the [**azure cni** network policy](../../examples/networkpolicy/README.md#azure-container-networking-default) plugin. This has some advantages and some consequences that must be considered when defining the network where we deploy the cluster. CNI provides an integration with azure subnet ip addressing so that every pod created ny kubernetes is assigned an ip address from the corresponding subnet. |
There was a problem hiding this comment.
every pod created ny kubernetes --> every pod created by kubernetes
| #### Kubenet | ||
|
|
||
| The built-in kubernetes network plugin is [Kubenet](https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/#kubenet). | ||
| Kubenet assigns virtual ips to the pods running in the cluster that are not part of the physical network infrastructure. The nodes are then configured to forward and masquerade the network calls using iptables rules. This means you can plan for a much smaller address space on your network as only the nodes will get an ip address. |
There was a problem hiding this comment.
Let's change all the occurrences of ip in the file to IP for consistency
|
|
||
| By default, acs-engine is using the [**azure cni** network policy](../../examples/networkpolicy/README.md#azure-container-networking-default) plugin. This has some advantages and some consequences that must be considered when defining the network where we deploy the cluster. CNI provides an integration with azure subnet ip addressing so that every pod created ny kubernetes is assigned an ip address from the corresponding subnet. | ||
| All IP addresses are pre-allocated at provisionning time. By default, [acs-engine will pre-allocate 128 ips per node](https://github.com/Azure/azure-container-networking/blob/master/docs/acs.md#enabling-azure-vnet-plugins-for-an-acs-kubernetes-cluster) on the subnet. | ||
| While this can be configured, new addresses will not be allocated dynamically. you need to plan for maximum scale. |
There was a problem hiding this comment.
you need to plan for maximum scale. incomplete sentence?
Codecov Report
@@ Coverage Diff @@
## master #2804 +/- ##
=======================================
Coverage 46.93% 46.93%
=======================================
Files 86 86
Lines 12786 12786
=======================================
Hits 6001 6001
Misses 6231 6231
Partials 554 554Continue to review full report at Codecov.
|
|
@CecileRobertMichon thanks for the review. All typos are corrected now. |
jackfrancis
approved these changes
May 2, 2018
ghost
removed
the
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it: adds documentation that explains considerations to have before using acs-engine to run a Kubernetes cluster in an hybrid cloud/on-premise environment
If applicable: