Skip to content

Latest commit

 

History

History
405 lines (332 loc) · 8.79 KB

cdc-usage.md

File metadata and controls

405 lines (332 loc) · 8.79 KB

Milvus-CDC Usage

Limitation

  1. The deployment supports only Pulsar and Kafka Message Queues, as the built-in RocksMQ is inaccessible from the source side.
  2. All collections must currently be synchronized from the source cluster, requiring CDC tasks to be specified as *.
  3. The following operations are supported for synchronization from the source cluster to the target cluster via CDC:
    • Create/Drop Collection
    • Insert/Delete
    • Create/Drop Partition
    • Create/Drop Index
    • Load/Release/Flush
    • Load/Release Partitions
    • Create/Drop Database

Only mentioned items are supported, and Milvus CDC exclusively supports synchronized data, for inquiries on active and backup disaster recovery functions, please contact us.

Configuration

The configuration consists of two main parts: the Milvus configuration for Milvus-Target and the CDC startup configuration.

Furthermore, please ensure that both the source and target Milvus versions are 2.3.2 or higher. Any other versions are not supported.

  1. Milvus-Target Config

    Set the value of common.ttMsgEnabled in the milvus.yaml file in the milvus-target cluster to false.

  2. CDC Startup Config

    The following is the cdc startup configuration file template. You can see this file in the server/configs directory.

# cdc server address
address: 0.0.0.0:8444
# max task num
maxTaskNum: 100
# max task name length
maxNameLength: 256

# cdc meta data config
metaStoreConfig:
  # the metastore type, available value: etcd, mysql
  storeType: mysql
  # etcd address
#  etcd:
#    address:
#      - http://127.0.0.1:2379
#    enableAuth: false
#    username: root
#    password: root123456
#    enableTLS: false
#    tlsCertPath: deployment/cert/client.pem # path to your cert file
#    tlsKeyPath: deployment/cert/client.key # path to your key file
#    tlsCACertPath: deployment/cert/ca.pem # path to your CACert file
  # mysql connection address
  mysqlSourceUrl: root:root@tcp(127.0.0.1:3306)/milvus-cdc?charset=utf8
  # meta data prefix, if multiple cdc services use the same store service, you can set different rootPaths to achieve multi-tenancy
  rootPath: cdc

# milvus-source config, these settings are basically the same as the corresponding configuration of milvus.yaml in milvus source.
sourceConfig:
  # etcd config
  etcd:
    address:
      - http://127.0.0.1:2379
    rootPath: by-dev
    metaSubPath: meta
    enableAuth: false
    username: root
    password: root123456
    enableTLS: false
    tlsCertPath: deployment/cert/client.pem # path to your cert file
    tlsKeyPath: deployment/cert/client.key # path to your key file
    tlsCACertPath: deployment/cert/ca.pem # path to your CACert file
    tlsMinVersion: 1.3
  etcdRootPath: by-dev
  etcdMetaSubPath: meta
  # default partition name
  defaultPartitionName: _default
  # read buffer length, mainly used for buffering if writing data to milvus-target is slow.
  readChanLen: 4
  # milvus replicate channel name, which is `{msgChannel.chanNamePrefix.cluster}/{msgChannel.chanNamePrefix.replicateMsg}` in the milvus.yaml file
  replicateChan: by-dev-replicate-msg
  # milvus-source mq config, which is pulsar or kafka
  pulsar:
    address: pulsar://localhost:6650
    webAddress: localhost:80
    maxMessageSize: 5242880
    tenant: public
    namespace: default
#    authPlugin: org.apache.pulsar.client.impl.auth.AuthenticationToken
#    authParams: token:xxx
#  kafka:
#    address: 127.0.0.1:9092

After completing these two steps of configuration, do not rush to start CDC. Need to ensure the following points:

  1. Whether the common.ttMsgEnabled configuration value of milvus-target is false;
  2. Confirm that the mq type configured by cdc is the same as the type configured by milvus-source;
  3. Ensure that the network environment where the cdc service is located can correctly connect to the mq and etcd addresses in the milvus-source in the configuration;

Usage

All http APIs need to comply with the following rules:

  • Request method: POST
  • Request path:/cdc
  • Request body:
{
     "request_type": "",
     "request_data": {}
}

Different requests are distinguished by request_type, which currently includes: create, delete, pause, resume, get and list.

The format response is:

{
  "code": 200,
  "message": "",
  "data": {}
}

If the request fails, the code is not 200, and the error message will be displayed in the message field; if the request is successful, the returned data is in data field

create request

  • milvus_connect_param, the connection params of the milvus-target server;
  • collection_infos, the collection information that needs to be synchronized, which currently only supports *;
POST localhost:8444/cdc
Content-Type: application/json

body:
{
  "request_type":"create",
  "request_data":{
    "milvus_connect_param":{
      "uri":"http://localhost:19530",
      "token":"root:Milvus",
      "connect_timeout":10
    },
    "collection_infos":[
      {
        "name":"*"
      }
    ]
  }
}

After success, the task_id will be returned, such as:

{
  "code": 200,
  "data": {
    "task_id":"6623ae52d35842a5a2c9d89b16ed7aa1"
  }
}

If there is an exception, a http error will appear.

Tips: If you want to know how to use tls to connect to the target, you can add the dial_config parameter to the milvus_connect_param parameter. For specific usage, refer to the following section.

create request with tls one-way authentication

Prerequisite:

  1. Place server.pem in a directory on the cdc server.
  2. Know the server name when the certificate is generated, which will be filled in dial_config.server_name.
POST localhost:8444/cdc
Content-Type: application/json

body:
{
  "request_type":"create",
  "request_data":{
    "milvus_connect_param":{
      "uri":"https://localhost:19530",
      "token":"root:Milvus",
      "connect_timeout":10,
      "dial_config": {
        "server_name": "localhost",
        "server_pem_path": "/xxx/server.pem"
      }
    },
    "collection_infos":[
      {
        "name":"*"
      }
    ]
  }
}

create request with tls two-way authentication

Prerequisite:

  1. Place ca.pem/client.pem/client.key in a directory on the cdc server.
  2. Know the server name when the certificate is generated, which will be filled in dial_config.server_name.
POST localhost:8444/cdc
Content-Type: application/json

body:
{
  "request_type":"create",
  "request_data":{
    "milvus_connect_param":{
      "uri":"https://localhost:19530",
      "token":"root:Milvus",
      "connect_timeout":10,
      "dial_config": {
        "server_name": "localhost",
        "ca_pem_path": "/xxx/ca.pem",
        "client_pem_path": "/xxx/client.pem",
        "client_key_path": "/xxx/client.key"
      }
    },
    "collection_infos":[
      {
        "name":"*"
      }
    ]
  }
}

delete request

delete a cdc task.

request

POST localhost:8444/cdc
Content-Type: application/json

body:
{
  "request_type":"delete",
  "request_data": {
    "task_id": "f84605ae48fb4170990ab80addcbd71e"
  }
}

response

{
  "code": 200,
  "data": {}
}

pause request

pause a cdc task.

request

POST localhost:8444/cdc
Content-Type: application/json

body:
{
  "request_type":"pause",
  "request_data": {
    "task_id": "4d458a58b0f74e85b842b1244dc69546"
  }
}

response

{
  "code": 200,
  "data": {}
}

resume request

resume a cdc task.

request

POST localhost:8444/cdc
Content-Type: application/json

body:
{
  "request_type":"resume",
  "request_data": {
    "task_id": "4d458a58b0f74e85b842b1244dc69546"
  }
}

response

{
  "code": 200,
  "data": {}
}

get request

get a cdc task info

request

POST localhost:8444/cdc
Content-Type: application/json

body:
{
  "request_type":"get",
  "request_data": {
    "task_id": "4d458a58b0f74e85b842b1244dc69546"
  }
}

response

{
  "code": 200,
  "data": {
    "Task": {
      "collection_infos": [
        {
          "name": "*"
        }
      ],
      "milvus_connect_param": {
        "uri":"http://localhost:19530",
        "port": 19541
      },
      "reason": "manually pause through http interface",
      "state": "Paused",
      "task_id": "728070fdf999499da869fc3a896217b0"
    }
  }
}

list request

list the info of all cdc tasks.

request

POST localhost:8444/cdc
Content-Type: application/json

body:
{
  "request_type":"list"
}

response

{
  "code": 200,
  "data": {
    "tasks": [
      {
        "task_id": "728070fdf999499da869fc3a896217b0",
        "milvus_connect_param": {
          "uri":"https://localhost:19530",
          "connect_timeout": 10
        },
        "collection_infos": [
          {
            "name": "*"
          }
        ],
        "state": "Running"
      }
    ]
  }
}