Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mysql 主键不在第一行，增量导入doris会报错 #400

Closed
gyp617280763 opened this issue Dec 13, 2024 · 1 comment
Closed

mysql 主键不在第一行，增量导入doris会报错 #400

gyp617280763 opened this issue Dec 13, 2024 · 1 comment
Milestone
v4.1.0

Comments

@gyp617280763
Copy link

gyp617280763 commented Dec 13, 2024 •
edited
Loading

两张相同结构的mysql表，唯一差异就在主键位置，A表在第一行，flinkcdc增量没问题。B表主键（主键为：id）不在第一行，flink会报错：

A表DDL：

CREATE TABLE `aa_vendor_test` (
  `id` bigint(20) NOT NULL COMMENT '主键',
  `iOrgId` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '创建组织id',
  `iCustID` bigint(20) DEFAULT NULL COMMENT '创建商家id',
  `iclass_id` bigint(20) DEFAULT NULL COMMENT '供应商分类',
  `cCreditCode` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '统一社会信用代码',
  `iTaxItems` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '税目',
  `cVendorPhone` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '固定电话',
  `cVendorAddress` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '通信地址',
  `cVendorZipCode` varchar(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '邮政编码',
  `cVendorEmail` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '电子邮箱',
  `cContact` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '联系人',
  `cContactPhone` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '联系人电话',
  `cContactMobile` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '移动电话',
  `cErpCode` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT 'ERP编码',
  `cRemark` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci,
  `cDataSource` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '档案来源',
  `bInternalUnit` bit(1) DEFAULT b'0' COMMENT '内部单位',
  `iCorrespondingOrgId` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '对应组织',
  `cWechatPublicAccount` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '微信公众号',
  `cRegion` bigint(20) DEFAULT NULL COMMENT '注册地区',
  `cRegionCode` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '注册地区代码',
  `cVendorRegisterAddress` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '注册地址',
  `iCreatorType` smallint(6) DEFAULT NULL COMMENT '创建者类型',
  `code` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT '编码',
  `name` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT '名称',
  `nameAlias` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '供应商别名,转小写，去空格，转半角',
  `pubts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '时间戳',
  `tenant_id` bigint(20) NOT NULL COMMENT '租户',
  `iCorrespondingCustId` bigint(20) DEFAULT NULL COMMENT '对应客户',
  `iCountryId` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '国家',
  `bFreezeStatus` bit(1) DEFAULT b'0' COMMENT '冻结状态',
  `cAccessStatus` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '供应商状态',
  `cYhtTenant` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '对应租户',
  `cCompanyType` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '企业类型',
  `cTransactions` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '交易方式',
  `cCooperationDate` date DEFAULT NULL COMMENT '首次合作日期',
  `iParentVendor` bigint(20) DEFAULT NULL COMMENT '上级供应商',
  `retailInvestors` bit(1) DEFAULT b'0' COMMENT '散户',
  `iTaxPayingCate` smallint(6) DEFAULT NULL COMMENT '纳税类别',
  `cTimeZoneId` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '时区',
  `cLanguageId` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '语言',
  `name2` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `name3` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `name4` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `name5` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `name6` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `cRemark2` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `cRemark3` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `cRemark4` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `cRemark5` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `cRemark6` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `cRelationTenantId` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '关联租户id',
  `nRelationEnterpriseId` bigint(20) DEFAULT NULL,
  `bcoordination` int(11) DEFAULT '0',
  `ytenant_id` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT '租户id',
  `relation_tenant_usercode` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '关联租户管理员账号',
  `ycn_code` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT 'YCN码',
  `supply_type` smallint(6) DEFAULT NULL COMMENT '供应商类型：企业0、个人1、非盈利组织2',
  `license_type` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '证照类型',
  `registerfund` decimal(28,8) DEFAULT NULL COMMENT '注册资本',
  `register_currency` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '注册币种ID',
  `basis_currency_registerfund` decimal(28,8) DEFAULT NULL COMMENT '注册资金（本币)',
  `basis_currency` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '币种（本币）ID',
  `found_date` datetime DEFAULT NULL COMMENT '企业成立日期',
  `legalbody` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '法定代表人',
  `vendor_fax` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '传真',
  `vendor_web_url` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '网站地址',
  `trade` smallint(6) DEFAULT NULL COMMENT '所属行业',
  `service_range` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci COMMENT '经营范围',
  `custom_category1_id` bigint(20) DEFAULT NULL COMMENT '自定义等级1',
  `custom_category2_id` bigint(20) DEFAULT NULL COMMENT '自定义等级2',
  `mgt_type` smallint(6) DEFAULT '0' COMMENT '管理方式：统管0/分管1',
  `ecotypesincevfive` varchar(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '经济类型',
  `capacity` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci COMMENT '产能/供货能力',
  `ent_profile` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci,
  `bus_performance` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci,
  `investigation_notes` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci COMMENT '考察情况说明',
  `extsystemcode` smallint(6) DEFAULT NULL COMMENT '来源系统标识',
  `extsystemmemberid` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '来源系统主键',
  `supplyapply_id` bigint(20) DEFAULT NULL COMMENT '准入申请单id',
  `apply_date` datetime DEFAULT NULL COMMENT '准入申请日期',
  `approve_date` datetime DEFAULT NULL COMMENT '准入申请审批时间',
  `apply_source` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '供应商申请来源',
  `channel_id` bigint(20) DEFAULT NULL COMMENT '供应商渠道',
  `ent_relationship_id` bigint(20) DEFAULT NULL COMMENT '关系表id',
  `tenant_error_msg` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci COMMENT '生成租户失败原因',
  `finance_access_status` int(11) DEFAULT NULL COMMENT '供应链金融开通状态',
  `re_reject` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci COMMENT '二次审核驳回原因',
  `re_reject_person` varchar(40) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '二次审核驳回人',
  `re_reject_date` date DEFAULT NULL COMMENT '二次审核驳回日期',
  `re_reject_status` int(11) DEFAULT NULL COMMENT '二次审核驳回状态',
  `sync_erp_flag` bit(1) DEFAULT b'0' COMMENT '是否同步ERP',
  `sync_erp_time` datetime DEFAULT NULL COMMENT '同步ERP时间',
  `dr` smallint(6) DEFAULT '0' COMMENT '删除标识',
  `ent_profile_filepath` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '企业介绍附件',
  `bus_performance_filepath` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '经营业绩附件',
  `investigation_filepath` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '考察报告附件',
  `filepath8` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附件8',
  `filepath7` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附件7',
  `filepath6` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附件6',
  `filepath5` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附件5',
  `filepath4` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附件4',
  `filepath3` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附件3',
  `filepath2` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附件2',
  `filepath1` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附件1',
  `businessPartner` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '业务伙伴id',
  `vendorCharacterDefine` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '供应商特征',
  `social_srcid` varchar(22) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '社会化来源id',
  `social_tenantid` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '社会化来源租户',
  `social_mctype` int(11) DEFAULT NULL COMMENT '社会化管控类型，0-只读，1-可更改',
  `transType` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '供应商交易类型',
  `crowd` bigint(20) DEFAULT NULL COMMENT '组织群id',
  `vendorApplyRangeOrgForRefFilter` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '使用组织(仅用于参照过滤,数据库不存数据)',
  `key` smallint(6) DEFAULT NULL COMMENT '核实状态',
  PRIMARY KEY (`id`),
  KEY `IX_AA_tenant_id` (`tenant_id`),
  KEY `idx_test1` (`id`,`tenant_id`),
  KEY `IX_AA_aa_vendor_code_tenant_id` (`tenant_id`,`code`,`pubts`),
  KEY `idx_iOrgId_tenantid` (`iOrgId`,`tenant_id`),
  KEY `idx_ytenant_id` (`ytenant_id`),
  KEY `idx_supentid` (`nRelationEnterpriseId`),
  KEY `idx_region_code` (`cRegionCode`),
  KEY `idx_crowd` (`crowd`) USING BTREE,
  KEY `i_pubts_ytenant_id` (`pubts`,`ytenant_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci COMMENT='供应商档案信息主表';

INSERT INTO dbn_detail.aa_vendor_test
(id,iOrgId, iCustID, iclass_id, cCreditCode, iTaxItems, cVendorPhone, cVendorAddress, cVendorZipCode, cVendorEmail, cContact, cContactPhone, cContactMobile, cErpCode, cRemark, cDataSource, bInternalUnit, iCorrespondingOrgId, cWechatPublicAccount, cRegion, cRegionCode, cVendorRegisterAddress, iCreatorType, code, name, nameAlias,  pubts, tenant_id, iCorrespondingCustId, iCountryId, bFreezeStatus, cAccessStatus, cYhtTenant, cCompanyType, cTransactions, cCooperationDate, iParentVendor, retailInvestors, iTaxPayingCate, cTimeZoneId, cLanguageId, name2, name3, name4, name5, name6, cRemark2, cRemark3, cRemark4, cRemark5, cRemark6, cRelationTenantId, nRelationEnterpriseId, bcoordination, ytenant_id, relation_tenant_usercode, ycn_code, supply_type, license_type, registerfund, register_currency, basis_currency_registerfund, basis_currency, found_date, legalbody, vendor_fax, vendor_web_url, trade, service_range, custom_category1_id, custom_category2_id, mgt_type, ecotypesincevfive, capacity, ent_profile, bus_performance, investigation_notes, extsystemcode, extsystemmemberid, supplyapply_id, apply_date, approve_date, apply_source, channel_id, ent_relationship_id, tenant_error_msg, finance_access_status, re_reject, re_reject_person, re_reject_date, re_reject_status, sync_erp_flag, sync_erp_time, dr, ent_profile_filepath, bus_performance_filepath, investigation_filepath, filepath8, filepath7, filepath6, filepath5, filepath4, filepath3, filepath2, filepath1, businessPartner, vendorCharacterDefine, social_srcid, social_tenantid, social_mctype, transType, crowd, vendorApplyRangeOrgForRefFilter, `key`)
VALUES(455079559430209589,'666666', NULL, 1933079379041583108, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 
'0', 0, NULL, NULL, NULL, NULL, NULL, NULL, '0001000001', '北京天达科技有限公司', '北京天达科技有限公司', 
'2024-10-28 14:49:53', 3671121647931408, NULL, NULL, 0, '2', NULL, NULL, NULL, NULL, NULL, 0, 0, 'Asia/Shanghai',
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 0, 'pa75jxsd', NULL, NULL, 0, 
'businessLicense', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 0, NULL, NULL, NULL, 
NULL, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 0, NULL, 0, NULL, 
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, '2120839277596639235', NULL, NULL, NULL, NULL,
1933079576610078727, NULL, NULL);


B表DDL：
-- dbn_detail.aa_vendor definition

CREATE TABLE `aa_vendor` (
  `iOrgId` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '创建组织id',
  `iCustID` bigint(20) DEFAULT NULL COMMENT '创建商家id',
  `iclass_id` bigint(20) DEFAULT NULL COMMENT '供应商分类',
  `cCreditCode` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '统一社会信用代码',
  `iTaxItems` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '税目',
  `cVendorPhone` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '固定电话',
  `cVendorAddress` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '通信地址',
  `cVendorZipCode` varchar(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '邮政编码',
  `cVendorEmail` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '电子邮箱',
  `cContact` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '联系人',
  `cContactPhone` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '联系人电话',
  `cContactMobile` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '移动电话',
  `cErpCode` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT 'ERP编码',
  `cRemark` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci,
  `cDataSource` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '档案来源',
  `bInternalUnit` bit(1) DEFAULT b'0' COMMENT '内部单位',
  `iCorrespondingOrgId` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '对应组织',
  `cWechatPublicAccount` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '微信公众号',
  `cRegion` bigint(20) DEFAULT NULL COMMENT '注册地区',
  `cRegionCode` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '注册地区代码',
  `cVendorRegisterAddress` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '注册地址',
  `iCreatorType` smallint(6) DEFAULT NULL COMMENT '创建者类型',
  `code` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT '编码',
  `name` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT '名称',
  `nameAlias` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '供应商别名,转小写，去空格，转半角',
  `id` bigint(20) NOT NULL COMMENT '主键',
  `pubts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '时间戳',
  `tenant_id` bigint(20) NOT NULL COMMENT '租户',
  `iCorrespondingCustId` bigint(20) DEFAULT NULL COMMENT '对应客户',
  `iCountryId` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '国家',
  `bFreezeStatus` bit(1) DEFAULT b'0' COMMENT '冻结状态',
  `cAccessStatus` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '供应商状态',
  `cYhtTenant` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '对应租户',
  `cCompanyType` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '企业类型',
  `cTransactions` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '交易方式',
  `cCooperationDate` date DEFAULT NULL COMMENT '首次合作日期',
  `iParentVendor` bigint(20) DEFAULT NULL COMMENT '上级供应商',
  `retailInvestors` bit(1) DEFAULT b'0' COMMENT '散户',
  `iTaxPayingCate` smallint(6) DEFAULT NULL COMMENT '纳税类别',
  `cTimeZoneId` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '时区',
  `cLanguageId` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '语言',
  `name2` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `name3` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `name4` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `name5` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `name6` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `cRemark2` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `cRemark3` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `cRemark4` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `cRemark5` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `cRemark6` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `cRelationTenantId` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '关联租户id',
  `nRelationEnterpriseId` bigint(20) DEFAULT NULL,
  `bcoordination` int(11) DEFAULT '0',
  `ytenant_id` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL COMMENT '租户id',
  `relation_tenant_usercode` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '关联租户管理员账号',
  `ycn_code` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT 'YCN码',
  `supply_type` smallint(6) DEFAULT NULL COMMENT '供应商类型：企业0、个人1、非盈利组织2',
  `license_type` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '证照类型',
  `registerfund` decimal(28,8) DEFAULT NULL COMMENT '注册资本',
  `register_currency` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '注册币种ID',
  `basis_currency_registerfund` decimal(28,8) DEFAULT NULL COMMENT '注册资金（本币)',
  `basis_currency` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '币种（本币）ID',
  `found_date` datetime DEFAULT NULL COMMENT '企业成立日期',
  `legalbody` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '法定代表人',
  `vendor_fax` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '传真',
  `vendor_web_url` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '网站地址',
  `trade` smallint(6) DEFAULT NULL COMMENT '所属行业',
  `service_range` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci COMMENT '经营范围',
  `custom_category1_id` bigint(20) DEFAULT NULL COMMENT '自定义等级1',
  `custom_category2_id` bigint(20) DEFAULT NULL COMMENT '自定义等级2',
  `mgt_type` smallint(6) DEFAULT '0' COMMENT '管理方式：统管0/分管1',
  `ecotypesincevfive` varchar(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '经济类型',
  `capacity` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci COMMENT '产能/供货能力',
  `ent_profile` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci,
  `bus_performance` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci,
  `investigation_notes` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci COMMENT '考察情况说明',
  `extsystemcode` smallint(6) DEFAULT NULL COMMENT '来源系统标识',
  `extsystemmemberid` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '来源系统主键',
  `supplyapply_id` bigint(20) DEFAULT NULL COMMENT '准入申请单id',
  `apply_date` datetime DEFAULT NULL COMMENT '准入申请日期',
  `approve_date` datetime DEFAULT NULL COMMENT '准入申请审批时间',
  `apply_source` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '供应商申请来源',
  `channel_id` bigint(20) DEFAULT NULL COMMENT '供应商渠道',
  `ent_relationship_id` bigint(20) DEFAULT NULL COMMENT '关系表id',
  `tenant_error_msg` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci COMMENT '生成租户失败原因',
  `finance_access_status` int(11) DEFAULT NULL COMMENT '供应链金融开通状态',
  `re_reject` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci COMMENT '二次审核驳回原因',
  `re_reject_person` varchar(40) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '二次审核驳回人',
  `re_reject_date` date DEFAULT NULL COMMENT '二次审核驳回日期',
  `re_reject_status` int(11) DEFAULT NULL COMMENT '二次审核驳回状态',
  `sync_erp_flag` bit(1) DEFAULT b'0' COMMENT '是否同步ERP',
  `sync_erp_time` datetime DEFAULT NULL COMMENT '同步ERP时间',
  `dr` smallint(6) DEFAULT '0' COMMENT '删除标识',
  `ent_profile_filepath` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '企业介绍附件',
  `bus_performance_filepath` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '经营业绩附件',
  `investigation_filepath` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '考察报告附件',
  `filepath8` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附件8',
  `filepath7` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附件7',
  `filepath6` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附件6',
  `filepath5` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附件5',
  `filepath4` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附件4',
  `filepath3` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附件3',
  `filepath2` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附件2',
  `filepath1` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附件1',
  `businessPartner` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '业务伙伴id',
  `vendorCharacterDefine` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '供应商特征',
  `social_srcid` varchar(22) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '社会化来源id',
  `social_tenantid` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '社会化来源租户',
  `social_mctype` int(11) DEFAULT NULL COMMENT '社会化管控类型，0-只读，1-可更改',
  `transType` varchar(36) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '供应商交易类型',
  `crowd` bigint(20) DEFAULT NULL COMMENT '组织群id',
  `vendorApplyRangeOrgForRefFilter` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '使用组织(仅用于参照过滤,数据库不存数据)',
  `key` smallint(6) DEFAULT NULL COMMENT '核实状态',
  PRIMARY KEY (`id`),
  KEY `IX_AA_tenant_id` (`tenant_id`),
  KEY `idx_test1` (`id`,`tenant_id`),
  KEY `IX_AA_aa_vendor_code_tenant_id` (`tenant_id`,`code`,`pubts`),
  KEY `idx_iOrgId_tenantid` (`iOrgId`,`tenant_id`),
  KEY `idx_ytenant_id` (`ytenant_id`),
  KEY `idx_supentid` (`nRelationEnterpriseId`),
  KEY `idx_region_code` (`cRegionCode`),
  KEY `idx_crowd` (`crowd`) USING BTREE,
  KEY `i_pubts_ytenant_id` (`pubts`,`ytenant_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci COMMENT='供应商档案信息主表';

INSERT INTO dbn_detail.aa_vendor
(iOrgId, iCustID, iclass_id, cCreditCode, iTaxItems, cVendorPhone, cVendorAddress, cVendorZipCode, cVendorEmail, cContact, cContactPhone, cContactMobile, cErpCode, cRemark, cDataSource, bInternalUnit, iCorrespondingOrgId, cWechatPublicAccount, cRegion, cRegionCode, cVendorRegisterAddress, iCreatorType, code, name, nameAlias, id, pubts, tenant_id, iCorrespondingCustId, iCountryId, bFreezeStatus, cAccessStatus, cYhtTenant, cCompanyType, cTransactions, cCooperationDate, iParentVendor, retailInvestors, iTaxPayingCate, cTimeZoneId, cLanguageId, name2, name3, name4, name5, name6, cRemark2, cRemark3, cRemark4, cRemark5, cRemark6, cRelationTenantId, nRelationEnterpriseId, bcoordination, ytenant_id, relation_tenant_usercode, ycn_code, supply_type, license_type, registerfund, register_currency, basis_currency_registerfund, basis_currency, found_date, legalbody, vendor_fax, vendor_web_url, trade, service_range, custom_category1_id, custom_category2_id, mgt_type, ecotypesincevfive, capacity, ent_profile, bus_performance, investigation_notes, extsystemcode, extsystemmemberid, supplyapply_id, apply_date, approve_date, apply_source, channel_id, ent_relationship_id, tenant_error_msg, finance_access_status, re_reject, re_reject_person, re_reject_date, re_reject_status, sync_erp_flag, sync_erp_time, dr, ent_profile_filepath, bus_performance_filepath, investigation_filepath, filepath8, filepath7, filepath6, filepath5, filepath4, filepath3, filepath2, filepath1, businessPartner, vendorCharacterDefine, social_srcid, social_tenantid, social_mctype, transType, crowd, vendorApplyRangeOrgForRefFilter, key)
VALUES('666666', NULL, 1933079379041583108, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
'0', 0, NULL, NULL, NULL, NULL, NULL, NULL, '0001000001', '北京科技有限公司', '北京科技有限公司', 355079559430209589,
'2024-10-28 14:49:53', 3671121647931408, NULL, NULL, 0, '2', NULL, NULL, NULL, NULL, NULL, 0, 0, 'Asia/Shanghai',
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 0, 'pa75jxsd', NULL, NULL, 0,
'businessLicense', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 0, NULL, NULL, NULL,
NULL, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 0, NULL, 0, NULL,
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, '2120839277596639235', NULL, NULL, NULL, NULL,
1933079576610078727, NULL, NULL);

B表报错信息：

2024-12-13 12:48:41,195 ERROR com.dtstack.chunjun.sink.format.BaseRichOutputFormat         [] - Writing records failed. com.dtstack.chunjun.throwable.NoRestartException: The dirty consumer shutdown, due to the consumed count exceed the max-consumed [0]
	at com.dtstack.chunjun.dirty.consumer.DirtyDataCollector.addConsumed(DirtyDataCollector.java:105)
	at com.dtstack.chunjun.dirty.consumer.DirtyDataCollector.offer(DirtyDataCollector.java:79)
	at com.dtstack.chunjun.dirty.manager.DirtyManager.collect(DirtyManager.java:140)
	at com.dtstack.chunjun.sink.format.BaseRichOutputFormat.writeSingleRecord(BaseRichOutputFormat.java:486)
	at com.dtstack.chunjun.connector.doris.sink.DorisHttpOutputFormat.lambda$writeRecordInternal$0(DorisHttpOutputFormat.java:131)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
	at com.dtstack.chunjun.connector.doris.sink.DorisHttpOutputFormat.writeRecordInternal(DorisHttpOutputFormat.java:131)
	at com.dtstack.chunjun.sink.format.BaseRichOutputFormat.lambda$initTimingSubmitTask$1(BaseRichOutputFormat.java:460)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

2024-12-13 12:48:41,195 WARN  com.dtstack.chunjun.dirty.log.LogDirtyDataCollector          [] - 
====================Dirty Data=====================
DirtyDataEntry[jobId='62d813fea134a102d8f8ed8128bca0e5', jobName='test1', operatorName='Sink: aa_vendor', dirtyContent='{"rowKind":"INSERT","arity":114}', errorMessage='com.dtstack.chunjun.throwable.WriteRecordException: 
java.lang.RuntimeException: colIdx:0,colName:id
	at com.dtstack.chunjun.connector.doris.sink.DorisHttpOutputFormat.writeSingleRecordInternal(DorisHttpOutputFormat.java:91)
	at com.dtstack.chunjun.sink.format.BaseRichOutputFormat.writeSingleRecord(BaseRichOutputFormat.java:483)
	at com.dtstack.chunjun.connector.doris.sink.DorisHttpOutputFormat.lambda$writeRecordInternal$0(DorisHttpOutputFormat.java:131)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
	at com.dtstack.chunjun.connector.doris.sink.DorisHttpOutputFormat.writeRecordInternal(DorisHttpOutputFormat.java:131)
	at com.dtstack.chunjun.sink.format.BaseRichOutputFormat.lambda$initTimingSubmitTask$1(BaseRichOutputFormat.java:460)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.RuntimeException: colIdx:0,colName:id
	at com.qlangtech.plugins.incr.flink.cdc.RowFieldGetterFactory$BasicGetter.getVal(RowFieldGetterFactory.java:281)
	at com.qlangtech.plugins.incr.flink.cdc.RowFieldGetterFactory$BasicGetter.getFieldOrNull(RowFieldGetterFactory.java:271)
	at com.qlangtech.plugins.incr.flink.chunjun.doris.sink.TISDorisColumnConverter$DorisSerializationConverter.serialize(TISDorisColumnConverter.java:154)
	at com.qlangtech.plugins.incr.flink.chunjun.doris.sink.TISDorisColumnConverter$DorisSerializationConverter.serialize(TISDorisColumnConverter.java:145)
	at com.qlangtech.plugins.incr.flink.chunjun.doris.sink.TISDorisColumnConverter.lambda$wrapNullableExternalConverter$6e5a4709$1(TISDorisColumnConverter.java:135)
	at com.qlangtech.plugins.incr.flink.chunjun.doris.sink.TISDorisColumnConverter.lambda$wrapNullableExternalConverter$6e5a4709$1(TISDorisColumnConverter.java:135)
	at com.qlangtech.plugins.incr.flink.chunjun.doris.sink.TISDorisColumnConverter.toExternal(TISDorisColumnConverter.java:118)
	at com.qlangtech.plugins.incr.flink.chunjun.doris.sink.TISDorisColumnConverter.toExternal(TISDorisColumnConverter.java:43)
	at com.dtstack.chunjun.connector.doris.rest.DorisLoadClient.createInsertVals(DorisLoadClient.java:198)
	at com.dtstack.chunjun.connector.doris.rest.DorisLoadClient.processGenericRowData(DorisLoadClient.java:169)
	at com.dtstack.chunjun.connector.doris.rest.DorisLoadClient.process(DorisLoadClient.java:155)
	at com.dtstack.chunjun.connector.doris.sink.DorisHttpOutputFormat.writeSingleRecordInternal(DorisHttpOutputFormat.java:89)
	... 11 more
Caused by: java.lang.ClassCastException
', fieldName='null', createTime=2024-12-13 12:48:41.195]
@baisui1981
Copy link
Member

先使用flink-sql 方式执行

@baisui1981 baisui1981 added this to the v4.1.0 milestone Dec 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v4.1.0
Development

No branches or pull requests
2 participants
@baisui1981 @gyp617280763