-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix SQL Server DATETIMEOFFSET for old dates #19627
Fix SQL Server DATETIMEOFFSET for old dates #19627
Conversation
Is there an (easy) way to disable predicate pushdown for a specific predicate? I've been unable to grok the Maybe instead of disabling it based on predicate type, we instead look at the value within the |
094fc62
to
289b893
Compare
@@ -928,7 +939,8 @@ private static ObjectReadFunction longTimestampWithTimeZoneReadFunction() | |||
return ObjectReadFunction.of( | |||
LongTimestampWithTimeZone.class, | |||
(resultSet, columnIndex) -> { | |||
OffsetDateTime offsetDateTime = resultSet.getObject(columnIndex, OffsetDateTime.class); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this causing some performance penalty?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it possible to perform this conversion only for historical dates ? I'm not sure if adding additional if
would cause additional penalties ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe so, we can get the MSSQL DateTimeOffset
object and if it is before 1583 we can do the String conversion. Presumably it'd more performant that always doing the string parsing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added a check to skip the date parsing if it is before 1583.
for conditional pushdown see https://github.com/hashhar/trino/blob/b8e44b702b8d4f6e1cc987a2175e430a3d05f373/plugin/trino-postgresql/src/main/java/io/trino/plugin/postgresql/PostgreSqlClient.java#L254 the logic can be something like:
cc: @Praveen2112 IIRC you implemented some pushdown logic in past which looked at actual values in the domain or do I misremember. |
ab1450a
to
77a137b
Compare
private static final PredicatePushdownController SQLSERVER_DATE_TIME_PUSHDOWN = (session, domain) -> { | ||
Domain simplifiedDomain = domain.simplify(getDomainCompactionThreshold(session)); | ||
for (Range range : simplifiedDomain.getValues().getRanges().getOrderedRanges()) { | ||
Range disableRange = range.getType().getJavaType().equals(LongTimestampWithTimeZone.class) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: should we keep these two ranges as constants, instead of LONG_DATETIMEOFFSET_DISABLE_VALUE
and SHORT_DATETIMEOFFSET_DISABLE_VALUE
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nineinchnick I had tried that, but the issue with having the Range
as constants is you need to create them like below, but these class types cannot be imported because they aren't public
in io.trino.spi.type
. Please let me know if there is another way to create them.
Range.lessThan(LongTimestampWithTimeZoneType.class, LONG_DATETIMEOFFSET_DISABLE_VALUE)
Range.lessThan(ShortTimestampWithTimeZoneType.class, SHORT_DATETIMEOFFSET_DISABLE_VALUE);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks mostly good to me but @Praveen2112 has better knowledge of the Domains
API so I'd appreciate a second look.
@@ -216,6 +220,13 @@ public class SqlServerClient | |||
|
|||
private static final DateTimeFormatter DATE_FORMATTER = DateTimeFormatter.ofPattern("uuuu-MM-dd"); | |||
|
|||
private static final DateTimeFormatter DATE_TIME_OFFSET_FORMATTER = new DateTimeFormatterBuilder() | |||
.appendPattern("yyyy-MM-dd HH:mm:ss") | |||
.appendFraction(NANO_OF_SECOND, 0, 7, true) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
.appendFraction(NANO_OF_SECOND, 0, 7, true) | |
.appendFraction(NANO_OF_SECOND, 0, MAX_SUPPORTED_TEMPORAL_PRECISION, true) |
String stringValue = resultSet.getString(columnIndex); | ||
OffsetDateTime offsetDateTime = ZonedDateTime.from(DATE_TIME_OFFSET_FORMATTER.parse(stringValue)).toOffsetDateTime(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this change and the one above in shortTimestampWithTimeZoneReadFunction
strictly required in this PR?
I'm unable to understand the significance probably.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, the issue is the driver changes from their internal DateTimeOffset
class to the java.time.OffsetDateTime
, the date shifts from what is stored. This instead gets the string representation of the date and we parse it to OffsetDateTime
.
See microsoft/mssql-jdbc#2246 for examples and their discussion. The root cause of this seems to be due to skipping days/dates when the Gregorian calendar switched. The MSSQL issue is closed, likely nothing that is going to be fixed because maybe it isn't broken? Dates are fun.
Apologies for the delay. I'll take a look at the PR soon |
5667923
to
1913373
Compare
@@ -928,7 +939,8 @@ private static ObjectReadFunction longTimestampWithTimeZoneReadFunction() | |||
return ObjectReadFunction.of( | |||
LongTimestampWithTimeZone.class, | |||
(resultSet, columnIndex) -> { | |||
OffsetDateTime offsetDateTime = resultSet.getObject(columnIndex, OffsetDateTime.class); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it possible to perform this conversion only for historical dates ? I'm not sure if adding additional if
would cause additional penalties ?
@@ -836,6 +837,61 @@ private void testSqlServerDatetimeOffset(ZoneId sessionZone) | |||
.execute(getQueryRunner(), session, sqlServerCreateAndInsert("test_sqlserver_datetimeoffset")); | |||
} | |||
|
|||
@Test | |||
public void testSqlServerDatetimeOffsetOldDates() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Instead of old
we could use historicalDates
as old
is pretty relative
"'1900-01-01 00:00:00.1234567+00:00'"); | ||
|
||
try (TestTable table = new TestTable(onRemoteDatabase(), "test_sqlserver_datetimeoffset_old_date_range_query", "(col0 datetimeoffset(7))", dateTimeOffsetValues)) { | ||
assertQuery("SELECT count(*) FROM " + table.getName(), "SELECT 9"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we also add an assertion to check if the filter is being pushed down or not ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added checks asserting that the filter is pushed down or not
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for addiing additional assertion - Can we merge both of them ? like
assertThat(query(SELECT count(*) FROM " + table.getName()))
.matches("SELECT 9")
.isFullyPushedDown();
assertQuery("SELECT count(*) FROM " + table.getName() + " WHERE col0 <= TIMESTAMP '1582-12-31 23:59:59.9999999+00:00'", "SELECT 3"); | ||
assertQuery("SELECT count(*) FROM " + table.getName() + " WHERE col0 >= TIMESTAMP '1583-01-01 00:00:00+00:00'", "SELECT 6"); | ||
assertQuery("SELECT count(*) FROM " + table.getName() + " WHERE col0 IN (TIMESTAMP '1582-12-31 23:59:59.9999999+00:00', TIMESTAMP '1583-01-01 00:00:00+00:00')", "SELECT 2"); | ||
assertQuery("SELECT count(*) FROM " + table.getName() + " WHERE col0 IN (TIMESTAMP '1583-01-01 00:00:00+00:00', TIMESTAMP '1600-01-01 00:00:00.1234567+00:00')", "SELECT 2"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about additional test cases for NOT IN
and BETWEEN
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added additional test cases
1913373
to
1e9fe12
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall LGTM
"'1900-01-01 00:00:00.1234567+00:00'"); | ||
|
||
try (TestTable table = new TestTable(onRemoteDatabase(), "test_sqlserver_datetimeoffset_old_date_range_query", "(col0 datetimeoffset(7))", dateTimeOffsetValues)) { | ||
assertQuery("SELECT count(*) FROM " + table.getName(), "SELECT 9"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for addiing additional assertion - Can we merge both of them ? like
assertThat(query(SELECT count(*) FROM " + table.getName()))
.matches("SELECT 9")
.isFullyPushedDown();
return DISABLE_PUSHDOWN.apply(session, domain); | ||
} | ||
return FULL_PUSHDOWN.apply(session, simplifiedDomain); | ||
}; | ||
|
||
// Dates prior to the Gregorian calendar switch in 1582 can cause incorrect results when pushed down, | ||
// so we disable predicate push down when the domain contains values prior to 1583 | ||
private static final Instant DISABLE_DATETIMEOFFSET_PUSHDOWN_IF_BEFORE = Instant.parse("1583-01-01T00:00:00Z"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The variable name gives an assumption to be like a boolean
field or some function which returns boolean
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed them to GREGORIAN_SWITCH_INSTANT
and GREGORIAN_SWITCH_DATETIMEOFFSET
, open to other suggestions.
assertThat(query("SELECT count(*) FROM " + table.getName())) | ||
.isFullyPushedDown(); | ||
|
||
assertThat(query("SELECT * FROM " + table.getName() + " WHERE col0 <= TIMESTAMP '1582-12-31 23:59:59.9999999+00:00'")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we also have an additional test case for WHERE col0 <= TIMESTAMP '1990-01-01 00:00:00+00:00
The value returned via the microsoft.sql.DateTimeOffset when converted to an OffsetDateTime is changed for old dates due to an issue in the JDBC driver. This changes retrieving datetimeoffset types from SQL Server to use getString instead of getObject and OffsetDateTime. Predicate pushdown is now disabled for this type due to test failures with IS NOT DISTINCT FROM predicates if the value is before the year 1583.
1e9fe12
to
72376f5
Compare
@Praveen2112 Updated per your last batch of comments. Please take another look when you can, thank you! |
Description
The value returned via the
microsoft.sql.DateTimeOffset
when converted to anOffsetDateTime
is changed for old dates due to an issue in the JDBC driver (see linked issue for example).This changes retrieving datetimeoffset types from SQL Server to use
getString
instead ofgetObject
andOffsetDateTime
.Predicate pushdown is now disabled for this type due to test failures with
IS NOT DISTINCT FROM
predicates.Additional context and related issues
#16559
microsoft/mssql-jdbc#2246
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(X) Release notes are required, with the following suggested text: