Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

enhancement(pulsar sink): introduce encoding schema and pulsar avro schema #5021

Merged
merged 15 commits into from
Dec 1, 2020
Merged

enhancement(pulsar sink): introduce encoding schema and pulsar avro schema #5021

merged 15 commits into from
Dec 1, 2020

Conversation

KannarFr
Copy link
Contributor

@KannarFr KannarFr commented Nov 13, 2020 •
edited
Loading

WIP

@KannarFr KannarFr changed the title Pulsar avro [WIP] Pulsar avro Nov 13, 2020
@KannarFr KannarFr changed the title [WIP] Pulsar avro [WIP] Introduce encoding schema and pulsar avro schema Nov 13, 2020
@KannarFr KannarFr changed the title [WIP] Introduce encoding schema and pulsar avro schema enhancement(sinks, pulsar): [WIP] Introduce encoding schema and pulsar avro schema Nov 13, 2020
@KannarFr
Copy link
Contributor Author

I'd like to make schema type depending on codec type so make encoding initialization parse schema if avro and use it in encode_event.

@KannarFr KannarFr changed the title enhancement(sinks, pulsar): [WIP] Introduce encoding schema and pulsar avro schema enhancement(sinks, pulsar): introduce encoding schema and pulsar avro schema Nov 13, 2020
@KannarFr KannarFr force-pushed the pulsar-avro branch 7 times, most recently from 884d735 to a5524cc Compare November 18, 2020 14:25
@KannarFr KannarFr changed the title enhancement(sinks, pulsar): introduce encoding schema and pulsar avro schema enhancement(pulsar sink): introduce encoding schema and pulsar avro schema Nov 18, 2020
Hoverbear
Hoverbear reviewed Nov 18, 2020
View reviewed changes
src/sinks/pulsar.rs Outdated Show resolved Hide resolved
Hoverbear
Hoverbear reviewed Nov 18, 2020
View reviewed changes
src/sinks/pulsar.rs Outdated Show resolved Hide resolved
Hoverbear
Hoverbear reviewed Nov 18, 2020
View reviewed changes
src/sinks/pulsar.rs Outdated Show resolved Hide resolved
@KannarFr
Copy link
Contributor Author

Invesgatigating

Nov 19 15:46:53.922 ERROR sink{component_kind="sink" component_name=pulsar component_type=pulsar}: vector::topology: An error occurred that vector couldn't handle.

@KannarFr
Copy link
Contributor Author

@Hoverbear fixed!

@Hoverbear
Copy link
Contributor

Great! @KannarFr can you help me by fixing the DCO?

git rebase HEAD~9 --signoff
git push --force-with-lease origin pulsar-avro

I also noted cargo fmt must be run.

Hoverbear
Hoverbear reviewed Nov 20, 2020
View reviewed changes
Cargo.toml Outdated Show resolved Hide resolved
Hoverbear
Hoverbear reviewed Nov 20, 2020
View reviewed changes
src/sinks/pulsar.rs Outdated Show resolved Hide resolved
Hoverbear
Hoverbear reviewed Nov 20, 2020
View reviewed changes
Copy link
Contributor

@Hoverbear Hoverbear left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rest of this PR looks good. :)

@KannarFr KannarFr requested a review from JeanMertz as a code owner November 20, 2020 21:37
@KannarFr
Copy link
Contributor Author

Misscliked for the review request sorry JeanMertz (don't want to ping you more with @).

@KannarFr
Copy link
Contributor Author

@Hoverbear is ok? :)

@KannarFr
Copy link
Contributor Author

@Hoverbear rebased!

Hoverbear
Hoverbear approved these changes Nov 30, 2020
View reviewed changes
Copy link
Contributor

@Hoverbear Hoverbear left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I note clippy is mad. If you could fix that I think this is good to merge!

@Hoverbear Hoverbear mentioned this pull request Nov 30, 2020
Merged
KannarFr added 15 commits December 1, 2020 11:04
@KannarFr
introduce schema encoding
4a43ffc 
Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>
@KannarFr
introduce avro encoding in pulsar sink
39a69c4 
Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>
@KannarFr
add avro test
93323cb 
Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>
@KannarFr
apply suggestion
d5022d1 
Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>
@KannarFr
put built schema to sink level
bc4387a 
Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>
@KannarFr
format
b913bb8 
Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>
@KannarFr
use Result
af6716b 
Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>
@KannarFr
print error
4c9f6a1 
Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>
@KannarFr
format
b0c9e66 
Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>
@KannarFr
make it internal_event
7321c4b 
Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>
@KannarFr
fmt
99e35c0 
Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>
@KannarFr
drop useless
bbafab7 
Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>
@KannarFr
clean
25c4f5f 
Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>
@KannarFr
take care of clippy
4b00722 
Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>
@KannarFr
add rand
1f6cac2 
Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>
@Hoverbear Hoverbear merged commit 10e4273 into vectordotdev:master Dec 1, 2020
mengesb pushed a commit to jacobbraaten/vector that referenced this pull request Dec 9, 2020
@KannarFr
enhancement(pulsar sink): introduce encoding schema and pulsar avro s…
5c8c577 
…chema (vectordotdev#5021)

* introduce schema encoding

Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>

* introduce avro encoding in pulsar sink

Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>

* add avro test

Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>

* apply suggestion

Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>

* put built schema to sink level

Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>

* format

Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>

* use Result

Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>

* print error

Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>

* format

Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>

* make it internal_event

Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>

* fmt

Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>

* drop useless

Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>

* clean

Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>

* take care of clippy

Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>

* add rand

Signed-off-by: Alexandre DUVAL - @KannarFr <[email protected]>
Signed-off-by: Brian Menges <[email protected]>
@fanatid fanatid mentioned this pull request Dec 16, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Hoverbear Hoverbear Hoverbear approved these changes

@JeanMertz JeanMertz Awaiting requested review from JeanMertz

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@KannarFr @Hoverbear