Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mid-level ArrowFlight Client #3371

Closed
Tracked by #3301
alamb opened this issue Dec 19, 2022 · 0 comments · Fixed by #3391 or #3402
Closed
Tracked by #3301

Mid-level ArrowFlight Client #3371

alamb opened this issue Dec 19, 2022 · 0 comments · Fixed by #3391 or #3402
Assignees
Labels
arrow-flight Changes to the arrow-flight crate enhancement Any new improvement worthy of a entry in the changelog

Comments

@alamb
Copy link
Contributor

alamb commented Dec 19, 2022

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I want to improve the FlightSQL client contributed by @avantgardnerio in #3207

Specifically FlightSQL is in terms of RecordBatches but at the moment to get record batches out out of a stream of FlightDatas is non trivial (especially as the schema and dictionaries are sent individually)

I think this is a common need as I found similar code in IOx for our own flight API as was in the FlightSQL client from #3207

@xudong963 also mentions there is similar code in DataBend: https://github.com/datafuselabs/databend/tree/main/src/query/service/src/api/rpc (uses the crate https://github.com/DataEngineeringLabs/arrow-format)

Describe the solution you'd like
I would like a mid-level flight client that handles these details - the FlightSQL client can then be built on top of this mid level client, and I think the mid level client is useful on its own (for writing clients for other Arrow Flight based APIs).

Describe alternatives you've considered

Additional context
We have a version of this code in https://github.com/influxdata/influxdb_iox/pull/6427

@alamb alamb mentioned this issue Dec 19, 2022
11 tasks
@alamb alamb added arrow-flight Changes to the arrow-flight crate enhancement Any new improvement worthy of a entry in the changelog labels Dec 19, 2022
@alamb alamb changed the title Mid-level FlightClient Mid-level ArrowFlight Client Dec 19, 2022
@alamb alamb self-assigned this Dec 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow-flight Changes to the arrow-flight crate enhancement Any new improvement worthy of a entry in the changelog
Projects
None yet
1 participant