Fix SQL Server DATETIMEOFFSET for old dates #19627

adamjshook · 2023-11-03T17:39:01Z

Description

The value returned via the microsoft.sql.DateTimeOffset when converted to an OffsetDateTime is changed for old dates due to an issue in the JDBC driver (see linked issue for example).

This changes retrieving datetimeoffset types from SQL Server to use getString instead of getObject and OffsetDateTime.

Predicate pushdown is now disabled for this type due to test failures with IS NOT DISTINCT FROM predicates.

Additional context and related issues

#16559
microsoft/mssql-jdbc#2246

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(X) Release notes are required, with the following suggested text:

# SQL Server Connector
* Fixed incorrect results for old DATETIMEOFFSET values ({issue}`16559`)

adamjshook · 2023-11-03T18:15:32Z

Is there an (easy) way to disable predicate pushdown for a specific predicate? I've been unable to grok the Domain and what means what.

Maybe instead of disabling it based on predicate type, we instead look at the value within the Domain and disable if it is old but enable if it is past the threshold? (need to determine the specific date, the 1582 date in the linked mssql-java issue would be a good place to start).

kokosing · 2023-11-15T14:02:14Z

plugin/trino-sqlserver/src/main/java/io/trino/plugin/sqlserver/SqlServerClient.java

@@ -928,7 +939,8 @@ private static ObjectReadFunction longTimestampWithTimeZoneReadFunction()
        return ObjectReadFunction.of(
                LongTimestampWithTimeZone.class,
                (resultSet, columnIndex) -> {
-                    OffsetDateTime offsetDateTime = resultSet.getObject(columnIndex, OffsetDateTime.class);


Is this causing some performance penalty?

Is it possible to perform this conversion only for historical dates ? I'm not sure if adding additional if would cause additional penalties ?

I believe so, we can get the MSSQL DateTimeOffset object and if it is before 1583 we can do the String conversion. Presumably it'd more performant that always doing the string parsing.

Added a check to skip the date parsing if it is before 1583.

hashhar · 2023-11-16T07:06:35Z

for conditional pushdown see https://github.com/hashhar/trino/blob/b8e44b702b8d4f6e1cc987a2175e430a3d05f373/plugin/trino-postgresql/src/main/java/io/trino/plugin/postgresql/PostgreSqlClient.java#L254

the logic can be something like:

if the domain is a single value or discrete set (lookup the methods on the Domain class) then we can get the values using getSingleValue or getDiscreteValues etc.
- check if the value is problematic and in that case return DISABLE_PUSHDOWN.apply(session, domain) and in others return FULL_PUSHDOWN.apply(session, domain).

cc: @Praveen2112 IIRC you implemented some pushdown logic in past which looked at actual values in the domain or do I misremember.

adamjshook · 2023-12-04T18:26:14Z

Updated predicate pushdown controller to disable pushdown for values prior to the year 1583. Ready for another look @hashhar @wendigo.

nineinchnick · 2023-12-11T09:09:54Z

plugin/trino-sqlserver/src/main/java/io/trino/plugin/sqlserver/SqlServerClient.java

+    private static final PredicatePushdownController SQLSERVER_DATE_TIME_PUSHDOWN = (session, domain) -> {
+        Domain simplifiedDomain = domain.simplify(getDomainCompactionThreshold(session));
+        for (Range range : simplifiedDomain.getValues().getRanges().getOrderedRanges()) {
+            Range disableRange = range.getType().getJavaType().equals(LongTimestampWithTimeZone.class)


Nit: should we keep these two ranges as constants, instead of LONG_DATETIMEOFFSET_DISABLE_VALUE and SHORT_DATETIMEOFFSET_DISABLE_VALUE?

@nineinchnick I had tried that, but the issue with having the Range as constants is you need to create them like below, but these class types cannot be imported because they aren't public in io.trino.spi.type. Please let me know if there is another way to create them.

Range.lessThan(LongTimestampWithTimeZoneType.class, LONG_DATETIMEOFFSET_DISABLE_VALUE) Range.lessThan(ShortTimestampWithTimeZoneType.class, SHORT_DATETIMEOFFSET_DISABLE_VALUE);

hashhar

Looks mostly good to me but @Praveen2112 has better knowledge of the Domains API so I'd appreciate a second look.

hashhar · 2023-12-12T14:57:41Z

plugin/trino-sqlserver/src/main/java/io/trino/plugin/sqlserver/SqlServerClient.java

@@ -216,6 +220,13 @@ public class SqlServerClient

    private static final DateTimeFormatter DATE_FORMATTER = DateTimeFormatter.ofPattern("uuuu-MM-dd");

+    private static final DateTimeFormatter DATE_TIME_OFFSET_FORMATTER = new DateTimeFormatterBuilder()
+            .appendPattern("yyyy-MM-dd HH:mm:ss")
+            .appendFraction(NANO_OF_SECOND, 0, 7, true)


Suggested change

.appendFraction(NANO_OF_SECOND, 0, 7, true)

.appendFraction(NANO_OF_SECOND, 0, MAX_SUPPORTED_TEMPORAL_PRECISION, true)

hashhar · 2023-12-12T14:59:54Z

plugin/trino-sqlserver/src/main/java/io/trino/plugin/sqlserver/SqlServerClient.java

+                    String stringValue = resultSet.getString(columnIndex);
+                    OffsetDateTime offsetDateTime = ZonedDateTime.from(DATE_TIME_OFFSET_FORMATTER.parse(stringValue)).toOffsetDateTime();


is this change and the one above in shortTimestampWithTimeZoneReadFunction strictly required in this PR?
I'm unable to understand the significance probably.

Yes, the issue is the driver changes from their internal DateTimeOffset class to the java.time.OffsetDateTime, the date shifts from what is stored. This instead gets the string representation of the date and we parse it to OffsetDateTime.

https://github.com/microsoft/mssql-jdbc/blob/main/src/main/java/microsoft/sql/DateTimeOffset.java#L229-L234

See microsoft/mssql-jdbc#2246 for examples and their discussion. The root cause of this seems to be due to skipping days/dates when the Gregorian calendar switched. The MSSQL issue is closed, likely nothing that is going to be fixed because maybe it isn't broken? Dates are fun.

Praveen2112 · 2023-12-12T15:07:34Z

Apologies for the delay. I'll take a look at the PR soon

Praveen2112 · 2023-12-13T08:25:05Z

plugin/trino-sqlserver/src/main/java/io/trino/plugin/sqlserver/SqlServerClient.java

@@ -928,7 +939,8 @@ private static ObjectReadFunction longTimestampWithTimeZoneReadFunction()
        return ObjectReadFunction.of(
                LongTimestampWithTimeZone.class,
                (resultSet, columnIndex) -> {
-                    OffsetDateTime offsetDateTime = resultSet.getObject(columnIndex, OffsetDateTime.class);


Is it possible to perform this conversion only for historical dates ? I'm not sure if adding additional if would cause additional penalties ?

Praveen2112 · 2023-12-13T08:27:23Z

plugin/trino-sqlserver/src/test/java/io/trino/plugin/sqlserver/BaseSqlServerTypeMapping.java

@@ -836,6 +837,61 @@ private void testSqlServerDatetimeOffset(ZoneId sessionZone)
                .execute(getQueryRunner(), session, sqlServerCreateAndInsert("test_sqlserver_datetimeoffset"));
    }

+    @Test
+    public void testSqlServerDatetimeOffsetOldDates()


nit: Instead of old we could use historicalDates as old is pretty relative

Praveen2112 · 2023-12-13T08:31:11Z

plugin/trino-sqlserver/src/test/java/io/trino/plugin/sqlserver/BaseSqlServerTypeMapping.java

+                "'1900-01-01 00:00:00.1234567+00:00'");
+
+        try (TestTable table = new TestTable(onRemoteDatabase(), "test_sqlserver_datetimeoffset_old_date_range_query", "(col0 datetimeoffset(7))", dateTimeOffsetValues)) {
+            assertQuery("SELECT count(*) FROM " + table.getName(), "SELECT 9");


Can we also add an assertion to check if the filter is being pushed down or not ?

Added checks asserting that the filter is pushed down or not

Thanks for addiing additional assertion - Can we merge both of them ? like

assertThat(query(SELECT count(*) FROM " + table.getName())) .matches("SELECT 9") .isFullyPushedDown();

Praveen2112 · 2023-12-13T08:32:41Z

plugin/trino-sqlserver/src/test/java/io/trino/plugin/sqlserver/BaseSqlServerTypeMapping.java

+            assertQuery("SELECT count(*) FROM " + table.getName() + " WHERE col0 <= TIMESTAMP '1582-12-31 23:59:59.9999999+00:00'", "SELECT 3");
+            assertQuery("SELECT count(*) FROM " + table.getName() + " WHERE col0 >= TIMESTAMP '1583-01-01 00:00:00+00:00'", "SELECT 6");
+            assertQuery("SELECT count(*) FROM " + table.getName() + " WHERE col0 IN (TIMESTAMP '1582-12-31 23:59:59.9999999+00:00', TIMESTAMP '1583-01-01 00:00:00+00:00')", "SELECT 2");
+            assertQuery("SELECT count(*) FROM " + table.getName() + " WHERE col0 IN (TIMESTAMP '1583-01-01 00:00:00+00:00', TIMESTAMP '1600-01-01 00:00:00.1234567+00:00')", "SELECT 2");


How about additional test cases for NOT IN and BETWEEN

Added additional test cases

Praveen2112

Overall LGTM

Praveen2112 · 2023-12-20T06:22:41Z

plugin/trino-sqlserver/src/test/java/io/trino/plugin/sqlserver/BaseSqlServerTypeMapping.java

+                "'1900-01-01 00:00:00.1234567+00:00'");
+
+        try (TestTable table = new TestTable(onRemoteDatabase(), "test_sqlserver_datetimeoffset_old_date_range_query", "(col0 datetimeoffset(7))", dateTimeOffsetValues)) {
+            assertQuery("SELECT count(*) FROM " + table.getName(), "SELECT 9");


Thanks for addiing additional assertion - Can we merge both of them ? like

assertThat(query(SELECT count(*) FROM " + table.getName())) .matches("SELECT 9") .isFullyPushedDown();

Praveen2112 · 2023-12-20T08:46:14Z

plugin/trino-sqlserver/src/main/java/io/trino/plugin/sqlserver/SqlServerClient.java

            return DISABLE_PUSHDOWN.apply(session, domain);
        }
        return FULL_PUSHDOWN.apply(session, simplifiedDomain);
    };

+    // Dates prior to the Gregorian calendar switch in 1582 can cause incorrect results when pushed down,
+    // so we disable predicate push down when the domain contains values prior to 1583
+    private static final Instant DISABLE_DATETIMEOFFSET_PUSHDOWN_IF_BEFORE = Instant.parse("1583-01-01T00:00:00Z");


The variable name gives an assumption to be like a boolean field or some function which returns boolean

Changed them to GREGORIAN_SWITCH_INSTANT and GREGORIAN_SWITCH_DATETIMEOFFSET, open to other suggestions.

Praveen2112 · 2023-12-20T08:49:55Z

plugin/trino-sqlserver/src/test/java/io/trino/plugin/sqlserver/BaseSqlServerTypeMapping.java

+            assertThat(query("SELECT count(*) FROM " + table.getName()))
+                    .isFullyPushedDown();
+
+            assertThat(query("SELECT * FROM " + table.getName() + " WHERE col0 <= TIMESTAMP '1582-12-31 23:59:59.9999999+00:00'"))


Can we also have an additional test case for WHERE col0 <= TIMESTAMP '1990-01-01 00:00:00+00:00

The value returned via the microsoft.sql.DateTimeOffset when converted to an OffsetDateTime is changed for old dates due to an issue in the JDBC driver. This changes retrieving datetimeoffset types from SQL Server to use getString instead of getObject and OffsetDateTime. Predicate pushdown is now disabled for this type due to test failures with IS NOT DISTINCT FROM predicates if the value is before the year 1583.

adamjshook · 2023-12-20T16:09:14Z

@Praveen2112 Updated per your last batch of comments. Please take another look when you can, thank you!

cla-bot bot added the cla-signed label Nov 3, 2023

adamjshook requested review from ebyhr and hashhar November 3, 2023 17:39

adamjshook force-pushed the adamjshook/sqlserver-datetime-offset branch from 094fc62 to 289b893 Compare November 3, 2023 19:30

ebyhr requested review from wendigo and removed request for ebyhr November 6, 2023 02:18

kokosing reviewed Nov 15, 2023

View reviewed changes

adamjshook force-pushed the adamjshook/sqlserver-datetime-offset branch 2 times, most recently from ab1450a to 77a137b Compare December 4, 2023 16:58

adamjshook requested a review from kokosing December 4, 2023 18:26

hashhar requested a review from Praveen2112 December 11, 2023 09:10

nineinchnick approved these changes Dec 11, 2023

View reviewed changes

hashhar reviewed Dec 12, 2023

View reviewed changes

adamjshook force-pushed the adamjshook/sqlserver-datetime-offset branch 2 times, most recently from 5667923 to 1913373 Compare December 12, 2023 23:23

Praveen2112 reviewed Dec 13, 2023

View reviewed changes

adamjshook force-pushed the adamjshook/sqlserver-datetime-offset branch from 1913373 to 1e9fe12 Compare December 18, 2023 16:10

adamjshook requested review from Praveen2112, hashhar and nineinchnick December 18, 2023 16:11

nineinchnick approved these changes Dec 19, 2023

View reviewed changes

Praveen2112 reviewed Dec 20, 2023

View reviewed changes

adamjshook force-pushed the adamjshook/sqlserver-datetime-offset branch from 1e9fe12 to 72376f5 Compare December 20, 2023 14:06

Praveen2112 approved these changes Dec 22, 2023

View reviewed changes

hashhar merged commit 457df6e into trinodb:master Dec 26, 2023

github-actions bot added this to the 436 milestone Dec 26, 2023

colebow mentioned this pull request Jan 9, 2024

Add Trino 436 release notes #20166

Merged

mosabua mentioned this pull request Jan 18, 2024

SQL Server connector returns incorrect results for old datetimeoffset values #16559

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix SQL Server DATETIMEOFFSET for old dates #19627

Fix SQL Server DATETIMEOFFSET for old dates #19627

adamjshook commented Nov 3, 2023

adamjshook commented Nov 3, 2023

kokosing Nov 15, 2023

Praveen2112 Dec 13, 2023

adamjshook Dec 13, 2023

adamjshook Dec 18, 2023

hashhar commented Nov 16, 2023

adamjshook commented Dec 4, 2023

nineinchnick Dec 11, 2023

adamjshook Dec 12, 2023

hashhar left a comment

hashhar Dec 12, 2023

hashhar Dec 12, 2023

adamjshook Dec 12, 2023

Praveen2112 commented Dec 12, 2023

Praveen2112 Dec 13, 2023

Praveen2112 Dec 13, 2023

Praveen2112 Dec 13, 2023

adamjshook Dec 18, 2023

Praveen2112 Dec 20, 2023

Praveen2112 Dec 13, 2023

adamjshook Dec 18, 2023 •

edited

Loading

Praveen2112 left a comment

Praveen2112 Dec 20, 2023

Praveen2112 Dec 20, 2023

adamjshook Dec 20, 2023

Praveen2112 Dec 20, 2023

adamjshook commented Dec 20, 2023

	.appendFraction(NANO_OF_SECOND, 0, 7, true)
	.appendFraction(NANO_OF_SECOND, 0, MAX_SUPPORTED_TEMPORAL_PRECISION, true)

		String stringValue = resultSet.getString(columnIndex);
		OffsetDateTime offsetDateTime = ZonedDateTime.from(DATE_TIME_OFFSET_FORMATTER.parse(stringValue)).toOffsetDateTime();

Fix SQL Server DATETIMEOFFSET for old dates #19627

Fix SQL Server DATETIMEOFFSET for old dates #19627

Conversation

adamjshook commented Nov 3, 2023

Description

Additional context and related issues

Release notes

adamjshook commented Nov 3, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hashhar commented Nov 16, 2023

adamjshook commented Dec 4, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hashhar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Praveen2112 commented Dec 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamjshook Dec 18, 2023 • edited Loading

Choose a reason for hiding this comment

Praveen2112 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamjshook commented Dec 20, 2023

adamjshook Dec 18, 2023 •

edited

Loading