Support from_unixtime via Gpu for non-UTC time zone #9814

res-life · 2023-11-21T09:51:18Z

closes #9605

Support from_unixtime via Gpu

First convert long to Timestamp Seconds then to Timestamp Microseconds, then shift to the time to the expected time zone.
Add is Daylight Saving Time utils.

Signed-off-by: Chong Gao [email protected]

res-life · 2023-11-21T09:52:42Z

build

res-life · 2023-11-21T09:54:06Z

Please only review the last commit: Support from_unixtime via CPU POC

res-life · 2023-11-21T09:57:16Z

build

res-life · 2023-11-21T10:07:13Z

build

winningsix · 2023-11-21T13:36:42Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsMeta.scala

@@ -1098,6 +1098,7 @@ abstract class BaseExprMeta[INPUT <: Expression](
  //+------------------------+-------------------+-----------------------------------------+
  lazy val needTimezoneCheck: Boolean = {
    wrapped match {
+      case _: FromUnixTime => false


why do we still need FromUnixTime given it's a sub-class of TimeZoneAwareExpression?

Done, now use new APIs.

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala

winningsix · 2023-11-21T13:40:18Z

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/datetimeExpressions.scala

-      tsVector.asStrings(strfFormat)
+    withResource(lhs.getBase.asTimestampSeconds) { secondsVector =>
+      withResource(secondsVector.asTimestampMicroseconds) { tsVector =>
+        if (zoneId.normalized().equals(ZoneId.of("UTC").normalized())) {


Use TimeZoneDB utils instead of hard coding UTC here.

res-life · 2023-11-21T15:39:11Z

Premerge failed, case is: convert large InternalRow iterator to cached batch single col *** FAILED ***

[2023-11-21T10:38:24.433Z] CachedBatchWriterSuite:

[2023-11-21T10:38:26.315Z] - convert columnar batch to cached batch on single col table with 0 rows in a batch

[2023-11-21T10:38:26.883Z] - convert large columnar batch to cached batch on single col table

[2023-11-21T10:38:26.883Z] - convert large columnar batch to cached batch on multi-col table

[2023-11-21T10:38:28.766Z] - convert large InternalRow iterator to cached batch single col *** FAILED ***

[2023-11-21T10:38:28.767Z]   java.lang.RuntimeException: ai.rapids.cudf.CudfException: RMM failure at:/home/jenkins/agent/workspace/jenkins-spark-rapids-jni_nightly-pre_release-211-cuda11/thirdparty/cudf/cpp/build/_deps/rmm-src/include/rmm/mr/device/arena_memory_resource.hpp:238: allocation not found

[2023-11-21T10:38:28.767Z]   at ai.rapids.cudf.ColumnVector$OffHeapState.cleanImpl(ColumnVector.java:1131)

[2023-11-21T10:38:28.767Z]   at ai.rapids.cudf.MemoryCleaner$Cleaner.clean(MemoryCleaner.java:117)

[2023-11-21T10:38:28.767Z]   at ai.rapids.cudf.ColumnVector.close(ColumnVector.java:268)

[2023-11-21T10:38:28.767Z]   at com.nvidia.spark.rapids.CachedBatchWriterSuite$TestResources.close(CachedBatchWriterSuite.scala:56)

[2023-11-21T10:38:28.767Z]   at com.nvidia.spark.rapids.RapidsPluginImplicits$AutoCloseableColumn.safeClose(implicits.scala:61)

[2023-11-21T10:38:28.767Z]   at com.nvidia.spark.rapids.Arm$.withResource(Arm.scala:31)

[2023-11-21T10:38:28.767Z]   at com.nvidia.spark.rapids.CachedBatchWriterSuite.$anonfun$new$7(CachedBatchWriterSuite.scala:96)

[2023-11-21T10:38:28.767Z]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

[2023-11-21T10:38:28.767Z]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)

[2023-11-21T10:38:28.767Z]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)

[2023-11-21T10:38:28.767Z]   ...

[2023-11-21T10:38:28.767Z]   Cause: ai.rapids.cudf.CudfException: RMM failure at:/home/jenkins/agent/workspace/jenkins-spark-rapids-jni_nightly-pre_release-211-cuda11/thirdparty/cudf/cpp/build/_deps/rmm-src/include/rmm/mr/device/arena_memory_resource.hpp:238: allocation not found

[2023-11-21T10:38:28.767Z]   at ai.rapids.cudf.Rmm.free(Native Method)

[2023-11-21T10:38:28.767Z]   at ai.rapids.cudf.DeviceMemoryBuffer$DeviceBufferCleaner.cleanImpl(DeviceMemoryBuffer.java:50)

[2023-11-21T10:38:28.767Z]   at ai.rapids.cudf.MemoryCleaner$Cleaner.clean(MemoryCleaner.java:117)

[2023-11-21T10:38:28.767Z]   at ai.rapids.cudf.MemoryBuffer.close(MemoryBuffer.java:247)

[2023-11-21T10:38:28.767Z]   at ai.rapids.cudf.ColumnVector$OffHeapState.cleanImpl(ColumnVector.java:1115)

[2023-11-21T10:38:28.767Z]   at ai.rapids.cudf.MemoryCleaner$Cleaner.clean(MemoryCleaner.java:117)

[2023-11-21T10:38:28.767Z]   at ai.rapids.cudf.ColumnVector.close(ColumnVector.java:268)

[2023-11-21T10:38:28.767Z]   at com.nvidia.spark.rapids.CachedBatchWriterSuite$TestResources.close(CachedBatchWriterSuite.scala:56)

[2023-11-21T10:38:28.767Z]   at com.nvidia.spark.rapids.RapidsPluginImplicits$AutoCloseableColumn.safeClose(implicits.scala:61)

[2023-11-21T10:38:28.767Z]   at com.nvidia.spark.rapids.Arm$.withResource(Arm.scala:31)

[2023-11-21T10:38:28.767Z]   ...

res-life · 2023-11-27T01:05:07Z

build

Signed-off-by: Chong Gao <[email protected]>

NVnavkumar · 2023-12-04T18:08:25Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsMeta.scala

@@ -1130,7 +1130,7 @@ abstract class BaseExprMeta[INPUT <: Expression](
    if (!isTimeZoneSupported) return checkUTCTimezone(this)

    // Level 4 check
-    if (TimeZoneDB.isSupportedTimezone(getZoneId())) {
+    if (!TimeZoneDB.isSupportedTimezone(getZoneId())) {


Switch to GpuTimeZoneDB.isSupportedTimezone. It's the same logic but in a class that will not be removed in the future.

res-life · 2023-12-05T01:59:21Z

build

res-life · 2023-12-05T05:27:06Z

build

res-life · 2023-12-05T07:58:31Z

integration_tests/src/main/python/time_zone_utils.py

@@ -0,0 +1,630 @@
+# Copyright (c) 2021-2022, NVIDIA CORPORATION.


should be 2023.
will update.

res-life · 2023-12-05T08:56:39Z

build

revans2 · 2023-12-05T18:06:31Z

integration_tests/src/main/python/time_zone_utils.py

+# limitations under the License.
+
+# get from Java:
+# ZoneId.getAvailableZoneIds


nit: The problem is that the backend on java is pluggable. This can change from one version of java to another. The TimeZoneProvider can also change this within a single version of java. Also this does not actually reflect all of the possible time zones. These are the normalized ones. There are deprecated ones that java still kind of supports, but get mapped to these. There are also fixed time zone offsets that are not covered by this and are technically any time range up to 24 hours at second granularity. This is minor because it is not likely to change, but if we could take the time zone and send it to java to check if it is valid/etc and then memorize the result instead that would make me feel better about this.

I think we can potentially handle as part of #9747

Actually this issue: #9633

Now forward to Java to check:

jvm.org.apache.spark.sql.rapids.TimeZoneDB.isSupportedTimeZone(tz)

I tested, it works in the Pytest marker.

integration_tests/src/main/python/date_time_test.py

Didn't see newer comments

res-life · 2023-12-06T06:11:52Z

build

res-life · 2023-12-06T08:07:13Z

Tested Iran and America/Los_Angeles time zones.
It falls back when TZ is Los_Angeles

gerashegalov · 2023-12-06T08:57:57Z

integration_tests/src/main/python/conftest.py

+    tz = get_test_tz()
+    jvm = spark_jvm()
+    ret = jvm.org.apache.spark.sql.rapids.TimeZoneDB.isSupportedTimeZone(tz)
+    print("my debug: is_supported_time_zone " + str(ret))


if this print is useful it should not be "my debug"

Done, remove the debug print.

revans2 · 2023-12-06T14:52:00Z

integration_tests/src/main/python/conftest.py

+    """
+    tz = get_test_tz()
+    jvm = spark_jvm()
+    ret = jvm.org.apache.spark.sql.rapids.TimeZoneDB.isSupportedTimeZone(tz)


Could we memorize this? Calling back into the JVM can be expensive. And if this is never going to change I would much rather have us keep the result cached.

Can we also use the GpuTimeZoneDB version of this?

Now cached the support info; And updated to GpuTimeZoneDB.

integration_tests/src/main/python/date_time_test.py

res-life · 2023-12-07T02:35:45Z

build

NVnavkumar

LGTM

integration_tests/src/main/python/conftest.py

res-life · 2023-12-07T07:21:30Z

build

res-life · 2023-12-07T07:22:30Z

build

res-life requested review from NVnavkumar and winningsix November 21, 2023 09:51

winningsix reviewed Nov 21, 2023

View reviewed changes

res-life changed the title ~~Support from_unixtime via CPU POC for non-UTC time zone~~ Support from_unixtime via Gpu for non-UTC time zone Nov 27, 2023

sameerz added the feature request New feature or request label Nov 28, 2023

res-life changed the base branch from branch-23.12 to branch-24.02 November 28, 2023 01:58

This was linked to issues Dec 1, 2023

[FEA] Support FromUnixTime for non UTC timezones #6836

Closed

[FEA] Support from_unixtime with CST(China Time Zone) support #9605

Closed

res-life force-pushed the FromUnixTime branch from afb6ac8 to 172de1d Compare December 4, 2023 10:38

Support from_unixtime via Gpu for non-UTC time zone

7625e16

Signed-off-by: Chong Gao <[email protected]>

res-life force-pushed the FromUnixTime branch from 172de1d to 7625e16 Compare December 4, 2023 10:41

Add is_dst_tz; Fix

6c961fb

NVnavkumar reviewed Dec 4, 2023

View reviewed changes

res-life marked this pull request as ready for review December 5, 2023 00:52

Update comments; Refactor; Use Gpu path

4c6133e

Chong Gao added 2 commits December 4, 2023 19:00

Add licence info; re-order imports

92741c6

xfail case instead of skipif

877130a

res-life mentioned this pull request Dec 5, 2023

Support date_format via Gpu for non-UTC time zone [databricks] #9721

Merged

res-life commented Dec 5, 2023

View reviewed changes

Update license; code format

abafbc1

revans2 reviewed Dec 5, 2023

View reviewed changes

NVnavkumar previously approved these changes Dec 5, 2023

View reviewed changes

Chong Gao added 2 commits December 6, 2023 09:32

Add a fall back case; Forward to Java to check whether TZ is supported

fff57fa

Fix Python import

afbaa8a

res-life mentioned this pull request Dec 6, 2023

[BUG] HostColumnVector memory leak at GpuTimeZoneDB.doLoadData #9971

Closed

gerashegalov reviewed Dec 6, 2023

View reviewed changes

revans2 reviewed Dec 6, 2023

View reviewed changes

Cache time zone support info; use GpuTimeZoneDB instead of TimeZoneDB

0ba4a2f

NVnavkumar previously approved these changes Dec 7, 2023

View reviewed changes

winningsix reviewed Dec 7, 2023

View reviewed changes

integration_tests/src/main/python/conftest.py Outdated Show resolved Hide resolved

winningsix previously approved these changes Dec 7, 2023

View reviewed changes

Format code

e7b1e7e

res-life dismissed stale reviews from winningsix and NVnavkumar via e7b1e7e December 7, 2023 07:19

res-life changed the title ~~Support from_unixtime via Gpu for non-UTC time zone~~ Support from_unixtime via Gpu for non-UTC time zone [databricks] Dec 7, 2023

res-life changed the title ~~Support from_unixtime via Gpu for non-UTC time zone [databricks]~~ Support from_unixtime via Gpu for non-UTC time zone Dec 7, 2023

winningsix approved these changes Dec 7, 2023

View reviewed changes

res-life merged commit 2805b95 into NVIDIA:branch-24.02 Dec 7, 2023
37 checks passed

res-life deleted the FromUnixTime branch December 7, 2023 10:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support from_unixtime via Gpu for non-UTC time zone #9814

Support from_unixtime via Gpu for non-UTC time zone #9814

res-life commented Nov 21, 2023 •

edited

Loading

res-life commented Nov 21, 2023

res-life commented Nov 21, 2023

res-life commented Nov 21, 2023

res-life commented Nov 21, 2023

winningsix Nov 21, 2023

res-life Dec 5, 2023

winningsix Nov 21, 2023

res-life Dec 5, 2023

res-life commented Nov 21, 2023

res-life commented Nov 27, 2023

NVnavkumar Dec 4, 2023

res-life Dec 5, 2023

res-life commented Dec 5, 2023

res-life commented Dec 5, 2023

res-life Dec 5, 2023

res-life Dec 6, 2023

res-life commented Dec 5, 2023

revans2 Dec 5, 2023

NVnavkumar Dec 5, 2023

NVnavkumar Dec 6, 2023

res-life Dec 6, 2023

res-life commented Dec 6, 2023

res-life commented Dec 6, 2023

gerashegalov Dec 6, 2023

res-life Dec 7, 2023

revans2 Dec 6, 2023

NVnavkumar Dec 6, 2023

res-life Dec 7, 2023

res-life commented Dec 7, 2023

NVnavkumar left a comment

res-life commented Dec 7, 2023

res-life commented Dec 7, 2023

		@@ -0,0 +1,630 @@
		# Copyright (c) 2021-2022, NVIDIA CORPORATION.

Support from_unixtime via Gpu for non-UTC time zone #9814

Support from_unixtime via Gpu for non-UTC time zone #9814

Conversation

res-life commented Nov 21, 2023 • edited Loading

res-life commented Nov 21, 2023

res-life commented Nov 21, 2023

res-life commented Nov 21, 2023

res-life commented Nov 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

res-life commented Nov 21, 2023

res-life commented Nov 27, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

res-life commented Dec 5, 2023

res-life commented Dec 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

res-life commented Dec 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

res-life commented Dec 6, 2023

res-life commented Dec 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

res-life commented Dec 7, 2023

NVnavkumar left a comment

Choose a reason for hiding this comment

res-life commented Dec 7, 2023

res-life commented Dec 7, 2023

res-life commented Nov 21, 2023 •

edited

Loading