Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MAP support for ExtractNestedField #65

Open
amirofmn opened this issue Dec 9, 2020 · 0 comments
Open

MAP support for ExtractNestedField #65

amirofmn opened this issue Dec 9, 2020 · 0 comments

Comments

@amirofmn
Copy link

amirofmn commented Dec 9, 2020

Hello,

We are pulling some data using kafka-connect (2.5.0) out of a kafka topic which contains JSON formatted data produced by winlogbeat. We only want the value from a specific key which we initially tried doing using ExtractField but it doesn't appear to support nested key/values. We stumbled upon this plugin which looks very promising but we're getting "MAP is not supported" errors.

Within our kafka-connect configuration, we are using the org.apache.kafka.connect.json.JsonConverter for both the keys and values. We have tried changing the over to the StringConverters to see if there would be any better luck and unfortunately not. The error message changes to "String is not supported".

We've tried chaining a mix of transforms from this plugin, a mix of this plugin and the default transforms (e.g. Flatten, Rename, Whitelist, ExtractField) that come with kafka-connect...every scenario seems to produce some type of error related to the type where it's expecting something specific and can't work with what was passed down.

I'm open to other transform chaining ideas/patterns but I'm kinda reaching out to see if MAP could be supported directly as I think our use-case is simple enough where others could benefit as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant