Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup type coercion (#3419) #6778

Merged
merged 4 commits into from
Jun 28, 2023
Merged

Cleanup type coercion (#3419) #6778

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
61f004f
Cleanup type coercion (#3419)
tustvold Jun 27, 2023
2a0d407
Further fixes
tustvold Jun 27, 2023
1829ca8
Tweak doc
tustvold Jun 27, 2023
83aa5b4
Review feedback
tustvold Jun 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 9 additions & 10 deletions datafusion/core/src/physical_planner.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1956,7 +1956,7 @@ mod tests {
use fmt::Debug;
use std::collections::HashMap;
use std::convert::TryFrom;
use std::ops::Not;
use std::ops::{BitAnd, Not};
use std::{any::Any, fmt};

fn make_session_state() -> SessionState {
Expand Down Expand Up @@ -2140,18 +2140,17 @@ mod tests {
async fn errors() -> Result<()> {
let bool_expr = col("c1").eq(col("c1"));
let cases = vec![
// utf8 AND utf8
col("c1").and(col("c1")),
// utf8 = utf8
col("c1").eq(col("c1")),
// u8 AND u8
col("c3").and(col("c3")),
// utf8 = bool
col("c1").eq(bool_expr.clone()),
// u32 AND bool
col("c2").and(bool_expr),
col("c3").bitand(col("c3")),
// utf8 = u8
col("c1").eq(col("c3")),
// bool AND bool
bool_expr.clone().and(bool_expr),
];
Comment on lines +2143 to 2151
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not entirely sure what this test was supposed to be testing, but a lot of these expression are ill-formed and now return an error

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps @izveigor remembers 🤔

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not familiar with this test, but it really seems redundant.

for case in cases {
let logical_plan = test_csv_scan().await?.project(vec![case.clone()]);
assert!(logical_plan.is_ok());
test_csv_scan().await?.project(vec![case.clone()]).unwrap();
}
Ok(())
}
Expand Down
2 changes: 1 addition & 1 deletion datafusion/core/tests/sqllogictests/test_files/dates.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ g
h

## Plan error when compare Utf8 and timestamp in where clause
statement error Error during planning: Timestamp\(Nanosecond, Some\("\+00:00"\)\) \+ Utf8 can't be evaluated because there isn't a common type to coerce the types to
statement error DataFusion error: type_coercion\ncaused by\nError during planning: Cannot coerce arithmetic expression Timestamp\(Nanosecond, Some\("\+00:00"\)\) \+ Utf8 to valid types
select i_item_desc from test
where d3_date > now() + '5 days';

Expand Down
34 changes: 26 additions & 8 deletions datafusion/core/tests/sqllogictests/test_files/interval.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -430,13 +430,15 @@ select '1 month'::interval + '1980-01-01T12:00:00'::timestamp;
----
1980-02-01T12:00:00

# Exected error: interval (scalar) - date / timestamp (scalar)

query error DataFusion error: type_coercion\ncaused by\nError during planning: Interval\(MonthDayNano\) \- Date32 can't be evaluated because there isn't a common type to coerce the types to
query D
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making this work wasn't actually intentional, but I don't see a reason why they shouldn't work

select '1 month'::interval - '1980-01-01'::date;
----
1979-12-01

query error DataFusion error: type_coercion\ncaused by\nError during planning: Interval\(MonthDayNano\) \- Timestamp\(Nanosecond, None\) can't be evaluated because there isn't a common type to coerce the types to
query P
select '1 month'::interval - '1980-01-01T12:00:00'::timestamp;
----
1979-12-01T12:00:00

# interval (array) + date / timestamp (array)
query D
Expand All @@ -454,11 +456,19 @@ select i + ts from t;
2000-02-01T00:01:00

# expected error interval (array) - date / timestamp (array)
query error DataFusion error: type_coercion\ncaused by\nError during planning: Interval\(MonthDayNano\) \- Date32 can't be evaluated because there isn't a common type to coerce the types to
query D
select i - d from t;
----
1979-12-01
1990-09-30
1980-01-02

query error DataFusion error: type_coercion\ncaused by\nError during planning: Interval\(MonthDayNano\) \- Timestamp\(Nanosecond, None\) can't be evaluated because there isn't a common type to coerce the types to
query P
select i - ts from t;
----
1999-12-01T00:00:00
1999-12-31T12:11:10
2000-01-31T23:59:00


# interval (scalar) + date / timestamp (array)
Expand All @@ -477,11 +487,19 @@ select '1 month'::interval + ts from t;
2000-03-01T00:00:00

# expected error interval (scalar) - date / timestamp (array)
query error DataFusion error: type_coercion\ncaused by\nError during planning: Interval\(MonthDayNano\) \- Date32 can't be evaluated because there isn't a common type to coerce the types to
query D
select '1 month'::interval - d from t;
----
1979-12-01
1990-09-01
1979-12-02

query error DataFusion error: type_coercion\ncaused by\nError during planning: Interval\(MonthDayNano\) \- Timestamp\(Nanosecond, None\) can't be evaluated because there isn't a common type to coerce the types to
query P
select '1 month'::interval - ts from t;
----
1999-12-01T00:00:00
1999-12-01T12:11:10
2000-01-01T00:00:00

# interval + date
query D
Expand Down
2 changes: 1 addition & 1 deletion datafusion/core/tests/sqllogictests/test_files/timestamps.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1145,7 +1145,7 @@ SELECT ts1 + i FROM foo;
2003-07-12T01:31:15.000123463

# Timestamp + Timestamp => error
query error DataFusion error: type_coercion\ncaused by\nInternal error: Unsupported operation Plus between Timestamp\(Nanosecond, None\) and Timestamp\(Nanosecond, None\)\. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
query error DataFusion error: Arrow error: Cast error: Cannot perform arithmetic operation between array of type Timestamp\(Nanosecond, None\) and array of type Timestamp\(Nanosecond, None\)
SELECT ts1 + ts2
FROM foo;

Expand Down
8 changes: 6 additions & 2 deletions datafusion/core/tests/sqllogictests/test_files/type_coercion.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,13 @@ SELECT '2023-05-01 12:30:00'::timestamp - interval '1 month';
2023-04-01T12:30:00

# interval - date
query error DataFusion error: type_coercion
query D
select interval '1 month' - '2023-05-01'::date;
----
2023-04-01

# interval - timestamp
query error DataFusion error: type_coercion
query P
SELECT interval '1 month' - '2023-05-01 12:30:00'::timestamp;
----
2023-04-01T12:30:00
Loading