Add `TIME_FORMAT()` Function To SQL Plugin #1301

GabeFernandez310 · 2023-01-27T20:52:22Z

Signed-off-by: GabeFernandez310 [email protected]

Description

Adds the time_format function to the SQL plugin. The function takes a DATE/DATETIME/TIME/TIMESTAMP/STRING in a valid format, and a formatting string (e.g., "%h %p") and outputs a formatted string. If a DATE is passed in, it treats the argument as if it is taken at midnight (i.e., 00:00:00).

Example:

opensearchsql> SELECT time_format("1998-01-31 13:14:15.012345", "%f %H %h %I %i %p %r %S %s %T");
fetched rows / total rows = 1/1
+------------------------------------------------------------------------------+
| time_format("1998-01-31 13:14:15.012345", "%f %H %h %I %i %p %r %S %s %T")   |
|------------------------------------------------------------------------------|
| 012345 13 01 01 14 PM 01:14:15 PM 15 15 13:14:15                             |
+------------------------------------------------------------------------------+

Issues Resolved

#722

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov-commenter · 2023-01-27T21:01:38Z

Codecov Report

Merging #1301 (efe4407) into main (72547f4) will increase coverage by 0.00%.
The diff coverage is 100.00%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@            Coverage Diff            @@
##               main    #1301   +/-   ##
=========================================
  Coverage     98.36%   98.37%           
- Complexity     3643     3680   +37     
=========================================
  Files           343      343           
  Lines          9010     9065   +55     
  Branches        585      585           
=========================================
+ Hits           8863     8918   +55     
  Misses          142      142           
  Partials          5        5

Flag	Coverage Δ
sql-engine	`98.37% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...c/main/java/org/opensearch/sql/expression/DSL.java	`100.00% <100.00%> (ø)`
...sql/expression/datetime/DateTimeFormatterUtil.java	`100.00% <100.00%> (ø)`
...arch/sql/expression/datetime/DateTimeFunction.java	`100.00% <100.00%> (ø)`
...h/sql/expression/function/BuiltinFunctionName.java	`100.00% <100.00%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

penghuo · 2023-01-30T16:46:57Z

integ-test/src/test/java/org/opensearch/sql/ppl/DateTimeFunctionIT.java

@@ -636,7 +636,7 @@ public void testDateFormat() throws IOException {
    String timestamp = "1998-01-31 13:14:15.012345";
    String timestampFormat = "%a %b %c %D %d %e %f %H %h %I %i %j %k %l %M "
        + "%m %p %r %S %s %T %% %P";
-    String timestampFormatted = "Sat Jan 01 31st 31 31 12345 13 01 01 14 031 13 1 "
+    String timestampFormatted = "Sat Jan 01 31st 31 31 012345 13 01 01 14 031 13 1 "


what is the change? 12345 -> 012345

For DATE_FORMAT and TIME_FORMAT functions, microseconds should be padded with 0s on the left side according to both MySQL and our documentation (6 digits of precision should always be returned when using %f). This was fixed and the test was changed.

penghuo · 2023-01-30T16:51:11Z

core/src/main/java/org/opensearch/sql/expression/datetime/DateTimeFunction.java

@@ -846,6 +847,12 @@ private DefaultFunctionResolver date_format() {
            STRING, DATE, STRING),
        impl(nullMissingHandling(DateTimeFormatterUtil::getFormattedDate),
            STRING, DATETIME, STRING),
+        implWithProperties(


update javadoc also.

Updated in 1af5666

Signed-off-by: GabeFernandez310 <[email protected]>

acarbonetto · 2023-02-07T17:45:23Z

core/src/main/java/org/opensearch/sql/expression/datetime/DateTimeFormatterUtil.java

@@ -61,7 +66,7 @@ interface DateTimeFormatHandler {
      .put("%D", (date) -> // %w - Day of month with English suffix
          String.format("'%d%s'", date.getDayOfMonth(), getSuffix(date.getDayOfMonth())))
      .put("%f", (date) -> // %f - Microseconds
-          String.format("'%d'", (date.getNano() / 1000)))
+          String.format(NANO_SEC_FORMAT, (date.getNano() / 1000)))


Love it - but I feel like we should use a constant for all of the surrounding format strings.

acarbonetto

minor requests

acarbonetto · 2023-02-07T18:59:08Z

core/src/test/java/org/opensearch/sql/expression/datetime/DateTimeFunctionTest.java

+            DSL.literal("%h")),
+        Arguments.of(
+            DSL.literal("61:11:12"),
+            DSL.literal("%h"))


nanoseconds test would be good

acarbonetto · 2023-02-07T19:00:39Z

core/src/test/java/org/opensearch/sql/expression/datetime/DateTimeFunctionTest.java

+        Arguments.of("%V"),
+        Arguments.of("%v"),
+        Arguments.of("%X"),
+        Arguments.of("%x")


you could also test formats without % or with extra characters like %%h or %hh or wrong order or extra spaces like h% or %h

…ction

MaxKsyunz

LGTM @GabeFernandez310.

GabeFernandez310 requested a review from a team as a code owner January 27, 2023 20:52

Yury-Fridlyand added enhancement New feature or request backport 2.x labels Jan 27, 2023

Yury-Fridlyand requested review from dai-chen and penghuo January 27, 2023 20:54

penghuo reviewed Jan 30, 2023

View reviewed changes

penghuo previously approved these changes Jan 31, 2023

View reviewed changes

acarbonetto mentioned this pull request Jan 31, 2023

SQL ingestion and ops - standup #791

Open

dai-chen previously approved these changes Feb 1, 2023

View reviewed changes

GabeFernandez310 dismissed stale reviews from dai-chen and penghuo via 31fdec6 February 2, 2023 16:40

Added Implementation And Testing (Fix After Rebase)

406ce81

Signed-off-by: GabeFernandez310 <[email protected]>

GabeFernandez310 force-pushed the integ-add-time_format-function branch from 31fdec6 to 406ce81 Compare February 2, 2023 16:51

GabeFernandez310 requested review from penghuo and dai-chen and removed request for penghuo February 2, 2023 17:33

Yury-Fridlyand previously approved these changes Feb 2, 2023

View reviewed changes

GabeFernandez310 marked this pull request as draft February 2, 2023 18:13

Yury-Fridlyand mentioned this pull request Feb 6, 2023

[FEATURE] Support date and time functions #722

Closed

87 tasks

acarbonetto reviewed Feb 7, 2023

View reviewed changes

acarbonetto previously approved these changes Feb 7, 2023

View reviewed changes

Merge branch 'main' into integ-add-time_format-function

84756c7

GabeFernandez310 dismissed stale reviews from acarbonetto and Yury-Fridlyand via 84756c7 February 7, 2023 21:58

Merge branch 'opensearch-project:main' into integ-add-time_format-fun…

67f213c

…ction

GabeFernandez310 marked this pull request as ready for review February 9, 2023 18:21

MaxKsyunz approved these changes Feb 14, 2023

View reviewed changes

dai-chen approved these changes Feb 15, 2023

View reviewed changes

Merge branch 'main' into integ-add-time_format-function

efe4407

MaxKsyunz merged commit bae0f6b into opensearch-project:main Feb 15, 2023

opensearch-trigger-bot bot mentioned this pull request Feb 15, 2023

[Backport 2.x] Add TIME_FORMAT() Function To SQL Plugin #1346

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `TIME_FORMAT()` Function To SQL Plugin #1301

Add `TIME_FORMAT()` Function To SQL Plugin #1301

GabeFernandez310 commented Jan 27, 2023 •

edited

Loading

codecov-commenter commented Jan 27, 2023 •

edited

Loading

penghuo Jan 30, 2023

GabeFernandez310 Jan 30, 2023

penghuo Jan 30, 2023

GabeFernandez310 Jan 30, 2023

acarbonetto Feb 7, 2023

acarbonetto left a comment

acarbonetto Feb 7, 2023

acarbonetto Feb 7, 2023

MaxKsyunz left a comment

Add TIME_FORMAT() Function To SQL Plugin #1301

Add TIME_FORMAT() Function To SQL Plugin #1301

Conversation

GabeFernandez310 commented Jan 27, 2023 • edited Loading

Description

Issues Resolved

Check List

codecov-commenter commented Jan 27, 2023 • edited Loading

Codecov Report

penghuo Jan 30, 2023

Choose a reason for hiding this comment

GabeFernandez310 Jan 30, 2023

Choose a reason for hiding this comment

penghuo Jan 30, 2023

Choose a reason for hiding this comment

GabeFernandez310 Jan 30, 2023

Choose a reason for hiding this comment

acarbonetto Feb 7, 2023

Choose a reason for hiding this comment

acarbonetto left a comment

Choose a reason for hiding this comment

acarbonetto Feb 7, 2023

Choose a reason for hiding this comment

acarbonetto Feb 7, 2023

Choose a reason for hiding this comment

MaxKsyunz left a comment

Choose a reason for hiding this comment

Add `TIME_FORMAT()` Function To SQL Plugin #1301

Add `TIME_FORMAT()` Function To SQL Plugin #1301

GabeFernandez310 commented Jan 27, 2023 •

edited

Loading

codecov-commenter commented Jan 27, 2023 •

edited

Loading