Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow different types of query variables (@@var) rather than just string #1943

Merged
merged 6 commits into from
Mar 12, 2022

Conversation

maxburke
Copy link
Contributor

@maxburke maxburke commented Mar 7, 2022

Closes #1942

@github-actions github-actions bot added datafusion Changes in the datafusion crate sql SQL Planner labels Mar 7, 2022
Copy link
Member

@xudong963 xudong963 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @maxburke -- Could you please add some tests?

@@ -399,7 +399,7 @@ impl fmt::Debug for Expr {
match self {
Expr::Alias(expr, alias) => write!(f, "{:?} AS {}", expr, alias),
Expr::Column(c) => write!(f, "{}", c),
Expr::ScalarVariable(var_names) => write!(f, "{}", var_names.join(".")),
Expr::ScalarVariable(_, var_names) => write!(f, "{}", var_names.join(".")),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'll be better to print the data type, such as
Expr::ScalarVariable(data_type, var_names) => write!(f, "{}, data type: {}", var_names.join("."), data_type),

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I avoided this because it looks like the Debug implementation for Expr tries to print SQL-compatible output, but, for example, "@v0, data type: DataType::Utf8" wouldn't be SQL-like

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I avoided this because it looks like the Debug implementation for Expr tries to print SQL-compatible output, but, for example, "@v0, data type: DataType::Utf8" wouldn't be SQL-like

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it looks like the Debug implementation for Expr tries to print SQL-compatible output

It's a good point, makes sense to me.

@xudong963
Copy link
Member

BTW, ballista indirectly uses ScalarVariable, so you need to process it.

@maxburke
Copy link
Contributor Author

maxburke commented Mar 8, 2022

Thanks @maxburke -- Could you please add some tests?

I thought about that but it looks like the existing tests for the user variables already cover it?

@xudong963
Copy link
Member

xudong963 commented Mar 8, 2022

Thanks @maxburke -- Could you please add some tests?

I thought about that but it looks like the existing tests for the user variables already cover it?

Are you referring to here?
https://github.com/apache/arrow-datafusion/blob/d468177687a596721c0a2af61f395f14c41942cd/datafusion/src/execution/context.rs#L1327

How about trying to test the other type and check the type? I have a vague feeling that there will be some problems of type conversion involved

@maxburke
Copy link
Contributor Author

maxburke commented Mar 8, 2022

How about trying to test the other type and check the type? I have a vague feeling that there will be some problems of type conversion involved

Could you elaborate more on what kinds of issues you are expecting?

The crux of this change is that, previously, the types of all StaticVariable type expressions was hardcoded to be DataType::Utf8; what this change does is add a way of having the variable provider specify the actual type of the value. It does not do any type conversions.

@xudong963
Copy link
Member

How about trying to test the other type and check the type? I have a vague feeling that there will be some problems of type conversion involved

Could you elaborate more on what kinds of issues you are expecting?

The crux of this change is that, previously, the types of all StaticVariable type expressions was hardcoded to be DataType::Utf8; what this change does is add a way of having the variable provider specify the actual type of the value. It does not do any type conversions.

Sorry for the confusion. I went through the original code and the changes made to this ticket. I found the changes will eventually generate Expr::ScalarVariable with data type in the logical plan phase. But In the physical phase, the ticket doesn't do anything with the data type. So I'm confused what problem does this ticket actually solve?

I would appreciate it if you could give an example usage about the ticket to help me understand and learn ❤️, thanks @maxburke

@alamb
Copy link
Contributor

alamb commented Mar 8, 2022

Thank you @maxburke . I agree with @xudong963 that all this PR really needs are some tests and it would be good to go 👍

Maybe you can adapt / update the ones in https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/execution/context.rs#L1286-L1313 ?

@maxburke
Copy link
Contributor Author

Test added / extended!

@alamb alamb changed the title Upstream implementation of typed query variables Allow different types of query variables (@@var) rather than just string Mar 12, 2022
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @maxburke and @xudong963 -- looks great to me

@@ -1300,14 +1317,15 @@ mod tests {
ctx.register_table("dual", provider)?;

let results =
plan_and_collect(&mut ctx, "SELECT @@version, @name FROM dual").await?;
plan_and_collect(&mut ctx, "SELECT @@version, @name, @integer + 1 FROM dual")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@alamb alamb merged commit 8db88ba into apache:master Mar 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datafusion Changes in the datafusion crate sql SQL Planner
Projects
None yet
Development

Successfully merging this pull request may close these issues.

DataFusion hardcodes all query variable types as Utf8
3 participants