You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have a table with a column video_id as primary key, and i find record with same primary key, have different record_key as shown in the picture below.
Steps to reproduce the behavior:
1.create table with spark sql, with tblproperties
tblproperties (
type = 'mor',
primaryKey = 'video_id'
)
2.insert historical data with spark sql
3.ingest real time incremental data with flink
config in flink ddl:
The _hoodie_record_key that written with spark contains a prefix "video_id:" while data written with flink doesn't
Hudi version : 0.11.0
Spark version : 3.1
flink version : 1.13
Storage (HDFS/S3/GCS..) : hdfs
Running on Docker? (yes/no) : no
The text was updated successfully, but these errors were encountered:
This is a known problem, because Spark uses the ComplexAvroKeyGenerator by default even if the primary key only has one field, while flink would use SimpleAvroKeyGenerator instead when primary key fields is simple, a temporal solution is to set up the key generator for spark as SimpleAvroKeyGenerator instead manually, i would fire a fix for spark soon ~
I have a table with a column
video_id
as primary key, and i find record with same primary key, have different record_key as shown in the picture below.Steps to reproduce the behavior:
1.create table with spark sql, with tblproperties
tblproperties (
type = 'mor',
primaryKey = 'video_id'
)
2.insert historical data with spark sql
3.ingest real time incremental data with flink
config in flink ddl:
![image](https://user-images.githubusercontent.com/8900183/172768975-9695d659-81da-4561-b73c-2fcf43328f9d.png)
The _hoodie_record_key that written with spark contains a prefix "video_id:" while data written with flink doesn't
Hudi version : 0.11.0
Spark version : 3.1
flink version : 1.13
Storage (HDFS/S3/GCS..) : hdfs
Running on Docker? (yes/no) : no
The text was updated successfully, but these errors were encountered: