Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Async parquet write #371

Closed
GrandChaman opened this issue Sep 3, 2021 · 3 comments · Fixed by #372
Closed

Async parquet write #371

GrandChaman opened this issue Sep 3, 2021 · 3 comments · Fixed by #372
Labels
no-changelog Issues whose changes are covered by a PR and thus should not be shown in the changelog

Comments

@GrandChaman
Copy link
Contributor

Hi !

First of all thanks for this library !

Would it be possible to add support for writing parquet file asynchronously ?

parquet2 already implements such function.
@GrandChaman GrandChaman mentioned this issue Sep 3, 2021
Merged
@jorgecarleitao jorgecarleitao added the feature A new feature label Sep 3, 2021
@jorgecarleitao
Copy link
Owner

jorgecarleitao commented Sep 3, 2021 •
edited
Loading

Just to desambiguate, there are 4 cases of interest:

  1. consumer: Write; producer: Iterator
  2. consumer: Write; producer: Stream
  3. consumer: AsyncWrite; producer: Iterator
  4. consumer: AsyncWrite; producer: Stream

We currently support 1 and 2. you are thinking in 3 and 4, right?

@jorgecarleitao jorgecarleitao reopened this Sep 3, 2021
@jorgecarleitao
Copy link
Owner

sorry, wrong combination of keys :(; will update comment above

@GrandChaman
Copy link
Contributor Author

Yes, exactly.

From what I understand 1 and 2 already exists.
I think it'd be nice to add support for 4.

Maybe there's no need for 3 as one could make a Stream from an Iterator ?

@jorgecarleitao jorgecarleitao added no-changelog Issues whose changes are covered by a PR and thus should not be shown in the changelog and removed feature A new feature labels Sep 3, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
no-changelog Issues whose changes are covered by a PR and thus should not be shown in the changelog
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Added support for async parquet write
2 participants
@jorgecarleitao @GrandChaman