[Feature] Design of Source/Sink connector

[xwm1992]

Search before asking

• I had searched in the issues and found no similar issues.

Feature Request

Design of Source/Sink Connector in EventMesh

Introduction

EventMesh Connector allows you to skip the complex integration with external services by offering out-of-the-box connectors. Each connector enabling you to have faster integrations with no codes. All data produced or processed by our connectors conforms to the CloudEvents specification.

Functional Requirements

• Connector - A connector is an image that interacts with a specific external service or underlying data source (e.g., Databases) on behalf of user applications. A connector is either a Source or a Sink.
• Source - A source connector obtains data from an underlying data producer and delivers it to targets, after original data has been transformed into CloudEvents. It doesn't limit the way how a source retrieves data. (e.g., A source may pull data from a message queue or act as an HTTP server waiting for data sent to it).
• Sink - A sink connector receives CloudEvents and does some specific business logics. (e.g., A MySQL Sink extracts useful data from CloudEvents and writes them to a MySQL database).
• CloudEvents - A specification for describing event data in common formats to provide interoperability across services, platforms and systems.

Technical Solution

Structure and process

Design Detail

Describe

Worker
Worker is divided into Source Worker and Sink Worker, which are triggered by the Application class and implement the methods of the ConnectorWorker interface respectively, which include the worker's running life cycle, and the worker carries the running of the connector. Workers can be lightweight and independent through mirroring Running, the eventmesh-sdk-java module is integrated internally, and the cloudevents protocol is used to interact with eventmesh. Currently, the tcp client is used by default. In the future, support for dynamic configuration can be considered

Connector
Connectors are divided into Source Connector and Sink Connector. Connectors have their own configuration files and run independently. Workers perform reflective loading and configuration analysis to complete Connector initialization and subsequent operation. Source Connector implements the poll method, and Sink Connector implements The put method uniformly uses ConnectorRecord to carry data. Both Source Connector and Sink Connector can operate independently.

ConnectorRecord with CloudEvents
ConnectorRecord is a connector layer data protocol. When workers interact with eventmesh, a protocol adapter needs to be developed to convert ConnectorRecord to CloudEvents protocol.

Registry
The Registry module is responsible for storing the synchronization progress of synchronizing data of different Connector instances, ensuring high availability between multiple Connector images or instances.

介绍

EventMesh 连接器允许您通过提供开箱即用的连接器来跳过与外部服务的复杂集成。每个连接器都可以让您无需代码即可实现更快的集成。我们的连接器生成或处理的所有数据都符合 CloudEvents 规范。

功能

• Connector - Connector是代表用户应用程序与特定外部服务或底层数据源（例如数据库）交互的镜像或实例。Connector可以是 Source 或 Sink。
• Source - Source Connector从底层数据生产者获取数据并将原始数据转换为 CloudEvents 后，交付给目标。它不限制从事件源获取数据的方式。 （例如，可以从消息队列中拉取数据或充当HTTP服务器等待发送给它的数据）。
• Sink - Sink Connector接收 CloudEvents 并执行一些特定的业务逻辑。 （例如，MySQL Sink 从 CloudEvents 中提取有用的数据并将它们写入 MySQL 数据库）。
• CloudEvents - 以通用格式描述事件数据以提供跨服务、平台和系统的互操作性的规范。

技术方案

结构与处理流程

详细设计

描述

Worker
Worker分为Source Worker与Sink Worker，由Application类进行触发运行，分别实现了ConnectorWorker接口的方法，其中包含了worker的运行生命周期，worker承载了connector的运行。Worker可以通过镜像的方式轻量的独立运行，内部集成了eventmesh-sdk-java模块，采用cloudevents协议与eventmesh进行交互，目前默认采用tcp客户端，后续可以考虑支持动态可配

Connector
Connector分为Source Connector与Sink Connector，connector有各自的配置文件，以及独立运行的方式，通过worker进行反射加载与配置解析，完成Connector的初始化以及后续运行工作，其中Source Connector实现poll方法，Sink Connector实现put方法，统一使用ConnectorRecord承载数据。Source Connector与Sink Connector均可独立运行。

ConnectorRecord with CloudEvents
ConnectorRecord为connector层数据协议，当worker与eventmesh进行交互时需开发协议适配器进行ConnectorRecord到CloudEvents的协议转换。

Registry
Registry模块负责存储同步不同Connector实例的数据的同步进度，确保多个Connector镜像或实例之间的高可用。

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[chenzhiguo]

• ConnectorRecord with CloudEvents，为什么不直接使用CloudEvents？
• Registry没太理解是什么角色？是类似于Kafka Broker？

[xwm1992]

• ConnectorRecord with CloudEvents，为什么不直接使用CloudEvents？
• Registry没太理解是什么角色？是类似于Kafka Broker？

cloudevents协议限制比较多，对一些属性字段要求的比较严格，直接定义在source接口上需要直接对事件源数据进行数据协议转换，因此在source接口上定义了ConnectorRecord更加通用；
Registry为注册中心的角色，类似etcd、nacos等