Add optional flag which advertises host for Arrow Flight SQL #418
Labels
enhancement
New feature or request
Comments
DaltonModlin
pushed a commit
to sxt-partners/arrow-ballista
that referenced
this issue
Oct 24, 2022
- Update scheduler_config_spec.toml to include new flag 'advertise_host' - Add advertise_host member variable to SchedulerServer - Add advertise_host argument to new, new_with_policy, new_with_builder, and new_with_state in order to propagate flag - Add None argument to relevant method calls Add optional flag which advertises host for Arrow Flight SQL apache#418 - Update logic in job_to_fetch_part to use advertise-host flag when it exists - Remove default from advertise_host in scheduler_config_spec.toml - Wrap scheduler_server advertise_host variable in Option - Update scheduler's main.rs to reflect advertise_host being wrapped in Option Utilize executor IP for routing to flight results in job_to_fetch_part even when advertise-host flag is set. Add missing variable and ownership stuff
DaltonModlin
pushed a commit
to sxt-partners/arrow-ballista
that referenced
this issue
Oct 24, 2022
- Update scheduler_config_spec.toml to include new flag 'advertise_host' - Add advertise_host member variable to SchedulerServer - Add advertise_host argument to new, new_with_policy, new_with_builder, and new_with_state in order to propagate flag - Add None argument to relevant method calls Add optional flag which advertises host for Arrow Flight SQL apache#418 - Update logic in job_to_fetch_part to use advertise-host flag when it exists - Remove default from advertise_host in scheduler_config_spec.toml - Wrap scheduler_server advertise_host variable in Option - Update scheduler's main.rs to reflect advertise_host being wrapped in Option Utilize executor IP for routing to flight results in job_to_fetch_part even when advertise-host flag is set. Add missing variable and ownership stuff Remove unnecessary output from do_get_fallback
DaltonModlin
pushed a commit
to sxt-partners/arrow-ballista
that referenced
this issue
Oct 31, 2022
- Update scheduler_config_spec.toml to include new flag 'advertise_host' - Add advertise_host member variable to SchedulerServer - Add advertise_host argument to new, new_with_policy, new_with_builder, and new_with_state in order to propagate flag - Add None argument to relevant method calls Add optional flag which advertises host for Arrow Flight SQL apache#418 - Update logic in job_to_fetch_part to use advertise-host flag when it exists - Remove default from advertise_host in scheduler_config_spec.toml - Wrap scheduler_server advertise_host variable in Option - Update scheduler's main.rs to reflect advertise_host being wrapped in Option Utilize executor IP for routing to flight results in job_to_fetch_part even when advertise-host flag is set. Add missing variable and ownership stuff Remove unnecessary output from do_get_fallback Responding to PR feedback - Rename advertise-host to advertise-endpoint - Replace unwrap calls with expect where possible - Add missing error handling when parsing advertise-endpoint flag PR feedback Co-authored-by: Andy Grove <[email protected]> PR Feedback - Using slice rather than array indexing for parsing advertise-endpoint PR Feedback - Fix clippy issue
DaltonModlin
pushed a commit
to sxt-partners/arrow-ballista
that referenced
this issue
Oct 31, 2022
- Update scheduler_config_spec.toml to include new flag 'advertise_host' - Add advertise_host member variable to SchedulerServer - Add advertise_host argument to new, new_with_policy, new_with_builder, and new_with_state in order to propagate flag - Add None argument to relevant method calls Add optional flag which advertises host for Arrow Flight SQL apache#418 - Update logic in job_to_fetch_part to use advertise-host flag when it exists - Remove default from advertise_host in scheduler_config_spec.toml - Wrap scheduler_server advertise_host variable in Option - Update scheduler's main.rs to reflect advertise_host being wrapped in Option Utilize executor IP for routing to flight results in job_to_fetch_part even when advertise-host flag is set. Add missing variable and ownership stuff Remove unnecessary output from do_get_fallback Responding to PR feedback - Rename advertise-host to advertise-endpoint - Replace unwrap calls with expect where possible - Add missing error handling when parsing advertise-endpoint flag PR feedback Co-authored-by: Andy Grove <[email protected]> PR Feedback - Using slice rather than array indexing for parsing advertise-endpoint PR Feedback - Fix clippy issue
DaltonModlin
pushed a commit
to sxt-partners/arrow-ballista
that referenced
this issue
Oct 31, 2022
- Update scheduler_config_spec.toml to include new flag 'advertise_host' - Add advertise_host member variable to SchedulerServer - Add advertise_host argument to new, new_with_policy, new_with_builder, and new_with_state in order to propagate flag - Add None argument to relevant method calls Add optional flag which advertises host for Arrow Flight SQL apache#418 - Update logic in job_to_fetch_part to use advertise-host flag when it exists - Remove default from advertise_host in scheduler_config_spec.toml - Wrap scheduler_server advertise_host variable in Option - Update scheduler's main.rs to reflect advertise_host being wrapped in Option Utilize executor IP for routing to flight results in job_to_fetch_part even when advertise-host flag is set. Add missing variable and ownership stuff Remove unnecessary output from do_get_fallback Responding to PR feedback - Rename advertise-host to advertise-endpoint - Replace unwrap calls with expect where possible - Add missing error handling when parsing advertise-endpoint flag PR feedback Co-authored-by: Andy Grove <[email protected]> PR Feedback - Using slice rather than array indexing for parsing advertise-endpoint PR Feedback - Fix clippy issue Fix pipeline failure
andygrove
pushed a commit
that referenced
this issue
Oct 31, 2022
- Update scheduler_config_spec.toml to include new flag 'advertise_host' - Add advertise_host member variable to SchedulerServer - Add advertise_host argument to new, new_with_policy, new_with_builder, and new_with_state in order to propagate flag - Add None argument to relevant method calls Add optional flag which advertises host for Arrow Flight SQL #418 - Update logic in job_to_fetch_part to use advertise-host flag when it exists - Remove default from advertise_host in scheduler_config_spec.toml - Wrap scheduler_server advertise_host variable in Option - Update scheduler's main.rs to reflect advertise_host being wrapped in Option Utilize executor IP for routing to flight results in job_to_fetch_part even when advertise-host flag is set. Add missing variable and ownership stuff Remove unnecessary output from do_get_fallback Responding to PR feedback - Rename advertise-host to advertise-endpoint - Replace unwrap calls with expect where possible - Add missing error handling when parsing advertise-endpoint flag PR feedback Co-authored-by: Andy Grove <[email protected]> PR Feedback - Using slice rather than array indexing for parsing advertise-endpoint PR Feedback - Fix clippy issue Fix pipeline failure Co-authored-by: Dalton Modlin <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
We have the ability to proxy flights through the Scheduler, but whether or not the client tries to get the Flight from the Scheduler or the Executor is entirely up to us and the IP address we advertise as having the Flight.
Many deployments will be made easier if we provide the ability to proxy all flights through the Scheduler. However, some users may prefer the ability to have the Client retrieve flights directly from Executors, so a flag seems like the appropriate middle ground.
Describe the solution you'd like
Since there are two potential modes of operation, we should use command line flags (or environment variables) to determine which mode to use. Currently, the executor has command line arguments called
--bind-hostand--external-host. Following this pattern, this issue proposes we add a command line argument to the scheduler called --advertise-host. If specified, all Flights will be altered to use that address. If not specified, the IPs of the executors will be left intact.Describe alternatives you've considered
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: