Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ballista register_table does not work with Dataframes that are not TableProviders #1027

Closed
rdettai opened this issue Sep 20, 2021 · 1 comment · Fixed by #1028
Closed

Ballista register_table does not work with Dataframes that are not TableProviders #1027

rdettai opened this issue Sep 20, 2021 · 1 comment · Fixed by #1028
Labels
bug Something isn't working

Comments

@rdettai
Copy link
Contributor

rdettai commented Sep 20, 2021

Describe the bug
The BallistaContext.register_table(name,dataframe) method can register any type of dataframe. But when serializing the plan to send it to the scheduler, it will only work if the value wrapped with the DfTableAdapter is a LogicalPlan::TableScan.

To Reproduce
Steps to reproduce the behavior:

  • create a dataframe with any of the BallistaContext.read_xxx() method
  • add another operation like a projection
  • register the result with BallistaContext.register_table()
  • build an sql query that uses the registered table
  • running this query will raise an error

Expected behavior
Possibilities:

  • we can only register instances of TableProvider instead of DataFrame (just as in Datafusion)
  • we add an extra step that unwraps DfTableAdapter before serialization
  • make it possible in datafusion to register DataFrames and implement the unwrapping in DataFusion
@rdettai rdettai added the bug Something isn't working label Sep 20, 2021
@rdettai
Copy link
Contributor Author

rdettai commented Sep 20, 2021

Note that according to the solution chosen, this will also close #1016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
1 participant