Skip to content

Commit

Permalink
Revise Multi-cluster Gateway descriptions in user docs
Browse files Browse the repository at this point in the history 
Revise the descriptions about Multi-cluster Gateway configuration.
Add descriptions about the `gateway-ip` annotation.
Remove unnecessary descriptions about implementations from the user
documents.

Signed-off-by: Jianjun Shen <[email protected]>
  • Loading branch information
@jianjuns
jianjuns committed Jun 15, 2022
1 parent 2e35c5f commit 44ff64e
Show file tree
Hide file tree
Showing 3 changed files with 56 additions and 45 deletions.
6 changes: 3 additions & 3 deletions docs/multicluster/architecture.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ Common Area of the leader cluster, decapsulates the resources from them, and
creates the resources (e.g. Services, Endpoints, Antrea ClusterNetworkPolicies,
ClusterInfoImports) in the member cluster.

For more information about Multi-cluster Service export/import, please also check
For more information about multi-cluster Service export/import, please also check
the [Service Export and Import](#service-export-and-import) section.

## Multi-cluster Service
Expand Down Expand Up @@ -170,7 +170,7 @@ Endpoints:
When the client Pod `pod-a` on cluster A tries to access the multi-cluster
Service `antrea-mc-nginx`, the request packet will first go through the Service
load balancing pipeline on the source Node `node-a2`, with one endpoint of the
Multi-cluster Service being chosen as the destination. Let's say endpoint
multi-cluster Service being chosen as the destination. Let's say endpoint
`10.11.12.33` from cluster C is chosen, then the request packet will be DNAT'd
with IP `10.11.12.33` and tunnelled to the local Gateway Node `node-a1`.
`node-a1` knows from the destination IP (`10.11.12.33`) the packet is
Expand All @@ -187,7 +187,7 @@ to `pod-c`.
## Antrea Multi-cluster NetworkPolicy

At this moment, Antrea does not support Pod-level policy enforcement for
cross-cluster traffic. Access towards Multi-cluster Services can be regulated
cross-cluster traffic. Access towards multi-cluster Services can be regulated
with Antrea ClusterNetworkPolicy `toService` rules. In each member cluster,
users can create an Antrea ClusterNetworkPolicy selecting Pods in that cluster,
with the imported Mutli-cluster Service name and Namespace in an egress
Expand Down
7 changes: 5 additions & 2 deletions docs/multicluster/quick-start.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ To use the latest version of Antrea Multi-cluster from the Antrea main branch,
you can change the YAML manifest path to: `https://github.com/antrea-io/antrea/tree/main/multicluster/build/yamls/`
when applying or downloading an Antrea YAML manifest.

Antrea must be deployed in both cluster A and cluster B. To configure Antrea
Antrea must be deployed in both cluster A and cluster B. To set up Antrea
Multi-cluster Gateways, the `Multicluster` feature of `antrea-agent` must be
feature enabled in both cluster A and B. Multi-cluster Gateways are required for
routing multi-cluster Service traffic across the member clusters. Set the
Expand All @@ -42,6 +42,9 @@ antrea-agent.conf: |
namespace: ""
```
At the moment, Multi-cluster Gateway only works with the Antrea `encap` traffic
mode, and all member clusters in a ClusterSet must use the same tunnel type.

## Set up Leader and Member in Cluster A

### Step 1 - deploy Antrea Multi-cluster Controllers for leader and member
Expand Down Expand Up @@ -114,7 +117,7 @@ information about Multi-cluster Gatweay, please refer to the [Multi-cluster
User Guide](user-guide.md#multi-cluster-gateway-configuration).

Assuming K8s Node `node-a1` is selected for the Multi-cluster Gateway, run
the following command to annotate the Node with
the following command to annotate the Node with:
`multicluster.antrea.io/gateway=true` (so Antrea can know it is the Gateway
Node from the annotation):

Expand Down
88 changes: 48 additions & 40 deletions docs/multicluster/user-guide.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ multi-cluster ClusterSet admin can define ClusterNetworkPolicies to be replicate
entire ClusterSet and enforced in all member clusters. Antrea Multi-cluster is introduced in
Antrea v1.5.0, and the ClusterNetworkPolicy replication feature is supported since Antrea
v1.6.0. In Antrea v1.7.0, the Multi-cluster Gateway feature is added that supports routing
Multi-cluster Service traffic through tunnels among clusters.
multi-cluster Service traffic through tunnels among clusters.

## Quick Start

Expand All @@ -31,7 +31,7 @@ you can change the YAML manifest path to: `https://github.com/antrea-io/antrea/t
when applying or downloading an Antrea YAML manifest.

Multi-cluster Services require an Antrea Multi-cluster Gateway to be set up in
each member cluster, so the Multi-cluster Service traffic can be routed across
each member cluster, so the multi-cluster Service traffic can be routed across
clusters by the Gateways. To support Multi-cluster Gateways, `antrea-agent` must
be deployed with the `Multicluster` feature enabled in a member cluster. You can
set the following configuration parameters in `antrea-agent.conf` of the Antrea
Expand All @@ -49,6 +49,9 @@ antrea-agent.conf: |
namespace: ""
```
At the moment, Multi-cluster Gateway only works with the Antrea `encap` traffic
mode, and all member clusters in a ClusterSet must use the same tunnel type.

### Deploy Antrea Mulit-cluster Controller

A Multi-cluster ClusterSet is comprised of a single leader cluster and at least
Expand Down Expand Up @@ -359,12 +362,16 @@ spec:

## Multi-cluster Gateway Configuration

Mulit-cluster Gateways are required to support multi-cluster Service access
across member clusters. Each member cluster should have one Node be specified as
its Multi-cluster Gateway. Multi-cluster Service traffic is routed among clusters
through the tunnels between Gateways.

After a member cluster joins a ClusterSet, and the `Multicluster` feature is
enabled for `antrea-agent`, one K8s Node in the member cluster can be specified
to serve as the Multi-cluster Gateway of the cluster. An annotation -
`multicluster.antrea.io/gateway=true` - should be added to the Node to tell
Antrea it is the Gateway Node. For example, you can run the following command to
annotate Node `node-1` as the Multi-cluster Gateway:
enabled on `antrea-agent`, you can select one Node of the cluster to serve as
the Mulit-cluster Gateway by adding an annotation:
`multicluster.antrea.io/gateway=true` to the K8s Node. For example, you can run
the following command to annotate Node `node-1` as the Multi-cluster Gateway:

```bash
$kubectl annotate node node-1 multicluster.antrea.io/gateway=true
Expand All @@ -387,29 +394,36 @@ internalIP: 10.17.27.55

`internalIP` of the Gateway is used for the tunnels between the Gateway Node and
other Nodes in the local cluster, while `gatewayIP` is used for the tunnels to
remote Gateways of other member clusters. By default, Multi-cluster Controller
will use the K8s Node `InternalIP` of the Gateway Node as the `gatewayIP`. If
you want to use `ExternalIP` of the Gateway Node instead, you can change the
configuration option `gatewayIPPrecedence` in ConfigMap
`antrea-mc-controller-config-***` to value `public`, when you deploy the member
Multi-cluster Controller. When selecting the Multi-cluster Gateway Node, you
need to make sure the resulted `gatewayIP` can be reached from the remote
Gateways.

After the `Gateway` CR is created, Multi-cluster Controller will be responsible
for exporting the cluster's network information to the leader cluster including
the Gateway IPs and `serviceCIDR` (the cluster's Service ClusterIP range).
Multi-cluster Controller will try to discover `serviceCIDR` automatically, but
you can also define the `serviceCIDR` manually in the Antrea Multi-cluster
ConfigMap `antrea-mc-controller-config-***`.

The Multi-cluster resource export/import pipeline will replicate the exported
cluster network information to all member clusters in the ClusterSet. For
example, in other member clusters, you can see a `ClusterInfoImport` CR with
name `test-cluster-east-clusterinfo` is created for `test-cluster-east` with
its network information. You can check the `ClusterInfoImport` with command:
`kubectl get clusterinfoimport test-cluster-east-clusterinfo -o yaml`, which
should show information like:
remote Gateways of other member clusters. Multi-cluster Controller discovers the
IP addresses from the K8s Node resource of the Gateway Node. It will always use
`InternalIP` of the K8s Node as the Gateway's `internalIP`. For `gatewayIP`,
there are several possibilities:

* By default, the K8s Node's `InternalIP` is used as `gatewayIP` too.
* You can choose to use the K8s Node's `ExternalIP` as `gatewayIP`, by changing
the configuration option `gatewayIPPrecedence` to value: `public`, when
deploying the member Multi-cluster Controller. The configration option is
defined in ConfigMap `antrea-mc-controller-config-***` in `antrea-multicluster-member.yml`.
* When the Gateway Node has a separate IP for external communication or is
associated with a public IP (e.g. an Elastic IP on AWS), but the IP is not added
to the K8s Node, you can still choose to use the IP as `gatewayIP`, by adding an
annotation: `multicluster.antrea.io/gateway-ip=<ip-address>` to the K8s Node.

When selecting the Multi-cluster Gateway Node, you need to make sure the
resulted `gatewayIP` can be reached from the remote Gateways.

After the Gateway is detected, Multi-cluster Controller will be responsible
for exporting the cluster's network information to other member clusters
through the leader cluster, including the cluster's Gateway IP and Service
CIDR. Multi-cluster Controller will try to discover the cluster's Service CIDR
automatically, but you can also manually specify the `serviceCIDR` option in
ConfigMap `antrea-mc-controller-config-***`. In other member clusters, a
`ClusterInfoImport` CR will be created for the cluster which includes the
exported network information. For example, in cluster `test-cluster-west`, you
you can see a `ClusterInfoImport` CR with name `test-cluster-east-clusterinfo`
is created for cluster `test-cluster-east`. You can check the
`ClusterInfoImport` with command: `kubectl get clusterinfoimport -o yaml`,
which should show information like:

```yaml
apiVersion: multicluster.crd.antrea.io/v1alpha1
Expand All @@ -424,16 +438,10 @@ spec:
serviceCIDR: 110.96.0.0/20
```

Make sure you repeat the same steps to assign a Gateway Node in all member
clusters, then `antrea-agent` will set up Geneve tunnels among the Gateway Nodes
of member clusters based on the local `Gateway` and the `ClusterInfoImport`
resources, and route Multi-cluster Service traffic across clusters through the
tunnels. `antrea-agent` on regular Nodes will route cross-cluster traffic from
local Pods to the Gateway Node of the member cluster.

Once you confirm that all `Gateway` and `ClusterInfoImport` resources are
created correctly, you can follow the [Multi-cluster Service](#multi-cluster-service)
section to create Multi-cluster Services and verify cross-cluster Service
Make sure you repeat the same step to assign a Gateway Node in all member
clusters. Once you confirm that all `Gateway` and `ClusterInfoImport` resources
are created correctly, you can follow the [Multi-cluster Service](#multi-cluster-service)
section to create multi-cluster Services and verify cross-cluster Service
access.

## Multi-cluster Service
Expand Down

0 comments on commit 44ff64e

Please sign in to comment.