Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP avro field parser #718

Merged

Conversation

jeremyplichtafc
Copy link
Contributor

Adds the ability to partition by custom fields out of avro messages

@jeremyplichtafc jeremyplichtafc changed the title WIP avro field parser avro field parser May 10, 2019
@jeremyplichtafc jeremyplichtafc changed the title avro field parser WIP avro field parser May 10, 2019
@HenryCaiHaiying
Copy link
Contributor

Looks like a good addition to secor, are you planning to merge this in (since WIP is still in the title)?

@jeremyplichtafc
Copy link
Contributor Author

Thank you - yes, there were a few more modifications I wanted to make and also wanted to see if it wouldn't be too complicated to add an Avro test to this. I will take the WIP off and ping you when ready

@jeremyplichtafc
Copy link
Contributor Author

I originally stopped work on this because the value I was partioning by had really high cardinality and was causing memory issues. Now that I have another use case that is low cardinality I will be testing out and using this code more and take the WIP off when I feel its stable

@jeremyplichtafc jeremyplichtafc force-pushed the jplichta/avro-field-parser branch from 577f9a6 to 7a5aecc Compare January 10, 2020 23:26
call schemaRegistryClient.deserialize
@HenryCaiHaiying HenryCaiHaiying merged commit 5d109e3 into pinterest:master May 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants