Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Option to inject kafka metadata #205

Merged
merged 19 commits into from
Apr 13, 2024
Merged

Option to inject kafka metadata #205

merged 19 commits into from
Apr 13, 2024

Conversation

tabmatfournier
Copy link
Contributor

@tabmatfournier tabmatfournier commented Mar 9, 2024 •
edited
Loading

  • adds KafkaMetadataTransform to inject metadata to Struct/Map records
  • adds option to inject Kafka metadata to both DMS and Debezium SMTs w/o an additional record copy (using an internally nested KafkaMetadataAppender)
  • can be configured to inject an (optional) static field such as the (external) kafka cluster address/name

fqtab
fqtab reviewed Apr 2, 2024
View reviewed changes
kafka-connect-transforms/src/main/java/io/tabular/iceberg/connect/transforms/DmsTransform.java Outdated Show resolved Hide resolved
...nect-transforms/src/main/java/io/tabular/iceberg/connect/transforms/util/RecordAppender.java Outdated Show resolved Hide resolved
...ansforms/src/main/java/io/tabular/iceberg/connect/transforms/util/KafkaMetadataAppender.java Outdated Show resolved Hide resolved
...ansforms/src/main/java/io/tabular/iceberg/connect/transforms/util/KafkaMetadataAppender.java Outdated Show resolved Hide resolved
...ansforms/src/main/java/io/tabular/iceberg/connect/transforms/util/KafkaMetadataAppender.java Outdated Show resolved Hide resolved
...ansforms/src/main/java/io/tabular/iceberg/connect/transforms/util/KafkaMetadataAppender.java Outdated Show resolved Hide resolved
...ansforms/src/main/java/io/tabular/iceberg/connect/transforms/util/KafkaMetadataAppender.java Outdated Show resolved Hide resolved
...ansforms/src/main/java/io/tabular/iceberg/connect/transforms/util/KafkaMetadataAppender.java Outdated Show resolved Hide resolved
...t-transforms/src/main/java/io/tabular/iceberg/connect/transforms/KafkaMetadataTransform.java Outdated Show resolved Hide resolved
...ansforms/src/main/java/io/tabular/iceberg/connect/transforms/util/KafkaMetadataAppender.java Outdated Show resolved Hide resolved
fqtab
fqtab reviewed Apr 12, 2024
View reviewed changes
Copy link
Contributor

@fqtab fqtab left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quite close, think we can wrap this up today.
The only major comment is I don't think offset field should be optional.

kafka-connect-transforms/README.md Outdated Show resolved Hide resolved
...t-transforms/src/main/java/io/tabular/iceberg/connect/transforms/KafkaMetadataTransform.java Outdated Show resolved Hide resolved
...t-transforms/src/main/java/io/tabular/iceberg/connect/transforms/KafkaMetadataTransform.java Outdated Show resolved Hide resolved
...t-transforms/src/main/java/io/tabular/iceberg/connect/transforms/KafkaMetadataTransform.java Outdated Show resolved Hide resolved
...t-transforms/src/main/java/io/tabular/iceberg/connect/transforms/KafkaMetadataTransform.java Outdated Show resolved Hide resolved
...t-transforms/src/main/java/io/tabular/iceberg/connect/transforms/KafkaMetadataTransform.java Outdated Show resolved Hide resolved
...t-transforms/src/main/java/io/tabular/iceberg/connect/transforms/KafkaMetadataTransform.java Show resolved Hide resolved
...t-transforms/src/main/java/io/tabular/iceberg/connect/transforms/KafkaMetadataTransform.java Outdated Show resolved Hide resolved
...ansforms/src/main/java/io/tabular/iceberg/connect/transforms/util/KafkaMetadataAppender.java Outdated Show resolved Hide resolved
kafka-connect-transforms/README.md Outdated Show resolved Hide resolved
@tabmatfournier
Copy link
Contributor Author

Quite close, think we can wrap this up today. The only major comment is I don't think offset field should be optional.

Addressed all of this.

fqtab
fqtab approved these changes Apr 12, 2024
View reviewed changes
...t-transforms/src/main/java/io/tabular/iceberg/connect/transforms/KafkaMetadataTransform.java
}

private SinkRecord applySchemaless(SinkRecord record) {
Map<String, Object> value = Requirements.requireMap(record.value(), "KafkaMetadata transform");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Map<String, Object> value = Requirements.requireMap(record.value(), "KafkaMetadata transform");
Map<String, Object> value = Requirements.requireMap(record.value(), "KafkaMetadataTransform");

@tabmatfournier tabmatfournier merged commit 423b4a8 into main Apr 13, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fqtab fqtab fqtab approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@tabmatfournier @fqtab