Feat | add cloud provider & network plugin

[chrisliu1995]

Background

Multi-cloud hybrid cloud expands the boundary of cloud and has become one of the trends of cloud-native development. In order to reduce the user's access cost to cloud infrastructure (such as network), we designed the Cloud Provider module, which aims to integrate with cloud infrastructure The coupling part is pluggably integrated into OpenKruiseGame (hereinafter referred to as OKG). At the beginning of the design of OKG, we believed that the game server network is an important function that cannot be ignored for game operation and maintenance, so we first supported the network plugin of Cloud Provider. Users can specify the network type and configuration information by defining the network field of the workload GameServerSet, and get the corresponding network information in the GameServer object.

Architecture

Cloud Provider is integrated in the webhook of kruise-game-manager in an in-tree manner, and its architecture diagram is as follows:

• provider manager: The manager of cloud provider, which can manage multiple different cloud provider objects
• cloud provider: OKG currently plans to support native Kubernetes and Alibaba Cloud. Each cloud provider can manage several different network plugins.
• network plugin: OKG currently plans to support Kubernetes' HostPort plugin and Alibaba Cloud's NatGw plugin. Developers can customize cloud providers and network plugins that meet their own needs through interface specifications.

Use

Create network

1. Specify the network type and corresponding network configuration in GameServerSet, and send it when GameServerSet is created

...

spec:
  network:
    networkType: HostPortNetwork #This example uses the HostPort network
    networkConf: 
    #The network configuration is imported in the form of K-V, and each plugin specifies the incoming format
    #The format of HostPort is as follows, {containerName}:{port1}/{protocol1},{port2}/{protocol2},...
    - name: ContainerPorts 
      value: game-server:25565/TCP 

...

2. After the creation is complete, you can view the corresponding network status in GameServer

...

status:
  networkStatus: 
    createTime: "2022-11-10T14:26:43Z"
    #The current state, as determined by the network plugin
    currentNetworkState: Ready 
    #The desired state, Ready/NotReady, is related to the GameServer.Spec.NetworkDisabled field. 
    #When NetworkDisabled is true, the expected state is NotReady; 
    #When NetworkDisabled is false, it is Ready, and the default is Ready when created.
    desiredNetworkState: Ready 
    externalAddresses:
    - ip: 38.111.149.177
      ports:
      - name: game-server-25565
        port: 8870
        protocol: TCP
    internalAddresses:
    - ip: 192.168.0.88
      ports:
      - name: game-server-25565
        port: 25565
        protocol: TCP
    lastTransitionTime: "2022-11-10T14:26:43Z"
    networkType: HostPortNetwork

...

Network Isolation

OKG supports users to set the network to be temporarily unavailable to achieve the effect of interrupting network traffic

1. Set the NetworkDisabled field of GameServer

...

spec:
  networkDisabled: true

...

2. Check the NetworkStatus of the GameServer

...

status:
  networkStatus: 
    createTime: "2022-11-10T14:26:43Z"
    currentNetworkState: NotReady #The network plugin senses the disabled operation, and returns to the current state after 
    completing the network isolation
    desiredNetworkState: NotReady 
    externalAddresses:
    - ip: 38.111.149.177
      ports:
      - name: game-server-25565
        port: 8870
        protocol: TCP
    internalAddresses:
    - ip: 192.168.0.88
      ports:
      - name: game-server-25565
        port: 25565
        protocol: TCP
    lastTransitionTime: "2022-11-10T14:29:01Z"
    networkType: HostPortNetwork

...

Principle

Create network

As shown in the figure above, when the user defines the network field when creating the GameServerSet:

(1) gameserverset-controler initiates a create pod request by creating an advanced statefulset, and the webhook intercepts and calls the OnPodCreate() function to create network. After the create request passes, the pod is successfully created

(2) The gameserver-controller perceives that the pod has a corresponding network status annotation, and writes it back to the GameServer Status

(3) When the GameServer is Ready, if the gameserver-controller finds that the desiredNetworkState is consistent with the currentNetworkState, the reconcile is terminated; when the two fields are inconsistent, the reconcile is retriggered, an update pod request is initiated, and the webhook intercepts the request and calls the OnPodUpdate() function. The network plugin returns the current up-to-date network status. The tuning interval is 5s, and the total waiting time is one minute. If the status is still inconsistent after more than one minute, the req will be abandoned.

Let's take a look at the changes in the fields of each object during the process:

(1) First, the user specifies the type and config of the GameServerSet.

(2) Specify the corresponding encoded annotation in the spec.template of the newly created Advanced StatefulSet.

(3) Through the pod mutating webhook, the pod adds the encoded status annotation. The network status fields that need to be returned by the plugin include externalAddress, internalAddress, and currentNetworkState.

(4) GameServer generates status.netwrokStatus for the status annotation of the Pod, in which externalAddress, internalAddress, and currentNetworkState are inherited from the annotations of pod, and others are generated or modified during the controller reconcile process.

(5) When the currentNetworkState is inconsistent with the desiredNetworkState, modify the Pod network-trigger-time annotation to the current time, trigger the update request(trigger interval defaults to 5 seconds), until the timeout (1 minute by default) or reaching the same state.

Network Isolation

As shown in the figure above, the networkDisabled field of the GameServer is specified when the user want to isolate/unisolate the network

(1) The gameserver-controller synchronizes the networkDisabled field to the annotation corresponding to the pod, triggers the update request intercepted by the webhook, and calls the OnPodUpdate() function. The network plugin will perform network isolation/unisolation according to the pod networkDisabled annotation, and change the currentNetworkState. The updated Pod will be with the latest status annotation, which includes whether the current network status is Ready.

(2) The gameserver-controller perceives the status change of the pod and synchronizes the networkStatus to the GameServer

(3) Similar to when creating process, decide whether to continue to initiate reconcile according to the comparison of network status consistency

Changes in the fields of each object during the process:

The meaning of the fields is similar to that of creating network, so I won’t repeat it in words.

Delete network

The process of deleting the network is very simple, as shown in the figure below, after the pod deletion request is intercepted by the webhook, the network plug-in calls the OnPodDelete() function to delete the network resources

Development Guide

OKG supports developers to customize cloud providers and network plugins according to their own needs. First of all, let's take a look at the call relationship diagram of each module in the webhook, so as to understand the meaning of each interface of the network plugin:

(1) When the webhook registers the mutating pod Handler, it initializes the provider manager and the corresponding cloud providers. At this point, the network plugins will register itself in the map of the corresponding cloud provider, waiting to be accessed.

(2) When the pod mutating req is triggered, the handler will first extract the network plugin name corresponding to the pod, and find the corresponding network plugin object according to the network plugin name. Then call the corresponding function according to the action of req:

• The create request corresponds to calling OnPodCreate()
• The update request corresponds to calling OnPodUpdate()
• The delete request corresponds to calling OnPodDelete()

network plugin development

The network plugin needs to implement the following func

type Plugin interface {
	Name() string
	Alias() string
	Init(client client.Client) error
	OnPodAdded(client client.Client, pod *corev1.Pod) (*corev1.Pod, error)
	OnPodUpdated(client client.Client, pod *corev1.Pod) (*corev1.Pod, error)
	OnPodDeleted(client client.Client, pod *corev1.Pod) error
}

The significance of OnPodCreate(), OnPodUpdate(), OnPodDelete() will not be described in detail.

• Name() returns the name of the network plugin, which is also the corresponding name when the user specifies the network type of GameServerSet
• Alias() returns the network plugin alias. In order to support the scenario where multiple clouds and hybrid clouds use the same workload, different cloud providers have different implementations for a certain plugin. In this case, the network type needs to be unified. So for a common network mode, you need to specify a common name 【TODO】
• Init() realizes the initialization of the network plugin. Some network plugins need to initialize their own cache according to the status of the cluster resource object. Init() helps to realize this function. It is worth noting the developer needs to judge whether the initialization has been performed inside the function to avoid repeated initialization. For network plugins that do not need to be initialized, return nil directly
• OnPodCreate(), OnPodUpdate(), and OnPodDelete() methods are called according to pod mutating requests, so update/delete may be called multiple times, and developers need to pay attention to the data consistency of the cache.

cloud provider development

If OKG does not currently support the cloud provider you expect, you can access it by implementing the following methods

type CloudProvider interface {
	Name() string
	ListPlugins() (map[string]Plugin, error)
}

• Name() returns the name of the cloud provider
• ListPlugins() lists all corresponding plugins
• Call func RegisterCloudProvider() in webhook's func NewProviderManager() to register the corresponding cloud provider