Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Embulk >0.8 #16

Closed
yzp0n opened this issue Feb 21, 2019 · 2 comments
Closed

Support Embulk >0.8 #16

yzp0n opened this issue Feb 21, 2019 · 2 comments

Comments

@yzp0n
Copy link
Contributor

yzp0n commented Feb 21, 2019

Hey. Thanks for publicizing this great piece of software! We are happily using embulk-output-orc in our production pipeline.

We found an incompatibility issue with Embulk >0.8 which enabled "Page Scattering"[release note] by default.
It seems the ScatterExecutor instantiates multiple instances of the plugin and end up running orc.Writer in parallel, while the default orc.MemoryManagerImpl is not thread-safe [code].
With Embulk >0.8, we get a lot of "Wrong Owner Thread" warnings [code], and random crashes when finalizing the write.

If you are open to contributions, we are happy to collaborate on a fix. We have a tentative fix which seems to be working on our prod, which we'll open a PR shortly.

(突然の Issue 失礼いたします。日本語のやりとりを希望される場合は、お申し付けください。)

@yuokada
Copy link
Owner

yuokada commented Feb 25, 2019

@yzp0n Thanks you for your contribution. I merged your PR.

@yzp0n
Copy link
Contributor Author

yzp0n commented Feb 26, 2019

Thank you for the merge! Let me close this issue as resolved.

@yzp0n yzp0n closed this as completed Feb 26, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants