Add support for XPath scalar functions - xpath,xpath_string,xpath_boolean,xpath_double #9219

sajjoseph · 2021-09-12T14:02:28Z

Supersedes #4024 Fixes #3888

Added support for the following scalar functions.

xpath - returns an array of strings that matched the criteria
xpath_string - returns a string that matched the criteria
xpath_boolean - returns the boolean value based on the xpath expression
xpath_double - returns a double-precision value after evaluating the xpath expression

Example:

trino> select xpath ('<a><b id="1"><c/></b><b id="2"><c/></b></a>','/descendant::c/ancestor::b/@id');
 _col0
--------
 [1, 2]
 
 trino> SELECT xpath_string ('<a><b>bb</b><c>cc</c></a>', 'a/b');
 _col0
-------
 bb


trino> SELECT xpath_double('<a><b class="odd">1</b><b class="even">2</b><b class="odd">4</b><c>8</c></a>', 'sum(a/*)') ;
 _col0
-------
  15.0

Few more sample queries can be found here - https://cwiki.apache.org/confluence/display/Hive/LanguageManual+XPathUDF

…ath_string,xpath_boolean,xpath_double

ebyhr

Left initial comments. I'm looking into other database behavior.

Please add the documentation.

The below comment about the argument order isn't yet resolved, so let me remind here.
#4024 (comment)

ebyhr · 2021-09-17T12:30:54Z

core/trino-main/src/main/java/io/trino/util/UDFXPathUtil.java

+ * Utility class for all XPath UDFs. Each UDF instance should keep an instance
+ * of this class.
+ */
+public class UDFXPathUtil


We don't use "UDF" to represent functions.

I guess you copied this class from Hive repository. Please add the reference to javadoc.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/xml/UDFXPathUtil.java

ebyhr · 2021-09-17T12:37:58Z

core/trino-main/src/main/java/io/trino/operator/scalar/XPathFunctions.java

+    @ScalarFunction("xpath_boolean")
+    @LiteralParameters({"x", "y"})
+    @SqlType(StandardTypes.BOOLEAN)
+    public static Boolean xpath_boolean(@SqlType("varchar(y)") Slice xml, @SqlType("varchar(x)") Slice path)


Rename to xpathBoolean

You don't need to change the value in @ScalarFunction annotation, but this method should be renamed.

ebyhr · 2021-09-17T12:38:09Z

core/trino-main/src/main/java/io/trino/operator/scalar/XPathFunctions.java

+    @ScalarFunction("xpath_double")
+    @LiteralParameters({"x", "y"})
+    @SqlType(StandardTypes.DOUBLE)
+    public static Double xpath_double(@SqlType("varchar(y)") Slice xml, @SqlType("varchar(x)") Slice path)


Rename to xpathDouble

ebyhr · 2021-09-17T12:39:56Z

core/trino-main/src/test/java/io/trino/operator/scalar/TestXPathFunction.java

+    @Test
+    public void testXpathString()
+    {
+        assertFunction("xpath_string('<a><b>bb</b><c>cc</c></a>','a/b')", VarcharType.createVarcharType(25), "bb");


Static import createVarcharType().

ebyhr · 2021-09-17T12:41:18Z

core/trino-main/src/main/java/io/trino/operator/scalar/XPathFunctions.java

+
+    @SqlNullable
+    @Description("Returns the text contents of the first xml node that matches the xpath expression")
+    @ScalarFunction("xpath_string")


The value of the annotation is redundant.

ebyhr · 2021-09-17T12:48:11Z

core/trino-main/src/main/java/io/trino/operator/scalar/XPathFunctions.java

+                .map(nodeList::item)
+                .filter(field -> field.getNodeValue() != null)
+                .map(field -> field.getNodeValue())
+                .collect(Collectors.toList());


Use toImmutableList().

ebyhr · 2021-09-17T12:49:23Z

core/trino-main/src/main/java/io/trino/operator/scalar/XPathFunctions.java

+        UDFXPathUtil xpath = new UDFXPathUtil();
+        NodeList nodeList = xpath.evalNodeList(xml, path);
+        if (nodeList == null) {
+            return Collections.<String>emptyList();


Return ImmutableList.of().

ebyhr · 2021-09-17T13:56:50Z

core/trino-main/src/main/java/io/trino/operator/scalar/XPathFunctions.java

+    public static Slice xpathString(@SqlType("varchar(x)") Slice xml, @SqlType("varchar(y)") Slice path)
+    {
+        UDFXPathUtil xpath = new UDFXPathUtil();
+        String s = xpath.evalString(xml.toStringUtf8(), path.toStringUtf8());


Please don't use abbreviation for variable name.

ebyhr · 2021-09-17T14:02:39Z

core/trino-main/src/test/java/io/trino/operator/scalar/TestXPathFunction.java

+    {
+        assertFunction("xpath('<a><b id=\"foo\">b1</b><b id=\"bar\">b2</b></a>','//@id')",
+                new ArrayType(VARCHAR),
+                Arrays.asList("foo", "bar"));


Use ImmutableList.of().

ebyhr · 2021-09-17T14:05:22Z

core/trino-main/src/main/java/io/trino/operator/scalar/XPathFunctions.java

+    @ScalarFunction("xpath")
+    @LiteralParameters({"x", "y"})
+    @SqlType("array(varchar)")
+    public static Block xpath(@SqlType("varchar(y)") Slice xml, @SqlType("varchar(x)") Slice path)


Why x and y are flipped? Subsequent functions also.

wgzhao · 2021-11-24T12:18:31Z

@sajjoseph Are you still continuing the development of this feature? I need this feature in my work and I migrated the related functions in Hive using the same approach, can you continue to develop this PR and have it merged into the master branch

mosabua · 2022-10-28T03:11:57Z

👋 @sajjoseph - this PR has become inactive. If you're still interested in working on it, please let us know. @ebyhr might have some insight how/if to proceed.

We're working on closing out old and inactive PRs, so if you're too busy or this has too many merge conflicts to be worth picking back up, we'll be making another pass to close it out in a few weeks.

tooptoop4 · 2022-11-19T01:37:15Z

this lit 🔥

colebow · 2022-11-30T22:32:48Z

Closing this one out due to inactivity, but please reopen if you would like to pick this back up. It seems like there's a decent amount of interest here, so if anyone else would like to take this on and get it updated, go for it.

cla-bot bot added the cla-signed label Sep 12, 2021

ebyhr added needs-docs This pull request requires changes to the documentation syntax-needs-review labels Sep 12, 2021

sajjoseph force-pushed the XPath-functions branch from f67aeea to 0236dd6 Compare September 12, 2021 16:09

Add support for XPath scalar functions to process XML data - xpath,xp…

f539da3

…ath_string,xpath_boolean,xpath_double

sajjoseph force-pushed the XPath-functions branch from 8fa3f44 to f539da3 Compare September 12, 2021 16:24

findepi added the enhancement New feature or request label Sep 13, 2021

ebyhr reviewed Sep 17, 2021

View reviewed changes

ebyhr assigned sajjoseph Oct 27, 2021

Merge branch 'trinodb:master' into XPath-functions

7fc17f6

ebyhr mentioned this pull request Nov 24, 2021

Add xml-related functions for parsing XML data using XPath expressions #10057

Closed

tooptoop4 approved these changes Jul 2, 2022

View reviewed changes

nineinchnick mentioned this pull request Nov 18, 2022

Some product tests are executed in multiple test suites #15096

Closed

colebow closed this Nov 30, 2022

weijiii mentioned this pull request Jun 4, 2024

Add support for XPath functions #22254

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for XPath scalar functions - xpath,xpath_string,xpath_boolean,xpath_double #9219

Add support for XPath scalar functions - xpath,xpath_string,xpath_boolean,xpath_double #9219

sajjoseph commented Sep 12, 2021 •

edited

Loading

ebyhr left a comment

ebyhr Sep 17, 2021

ebyhr Sep 17, 2021

ebyhr Oct 29, 2021

sajjoseph Oct 29, 2021

ebyhr Sep 17, 2021

sajjoseph Oct 29, 2021

ebyhr Sep 17, 2021

sajjoseph Oct 29, 2021

ebyhr Sep 17, 2021

ebyhr Sep 17, 2021

ebyhr Sep 17, 2021

ebyhr Sep 17, 2021

ebyhr Sep 17, 2021

ebyhr Sep 17, 2021

wgzhao commented Nov 24, 2021

mosabua commented Oct 28, 2022

tooptoop4 commented Nov 19, 2022

colebow commented Nov 30, 2022

Add support for XPath scalar functions - xpath,xpath_string,xpath_boolean,xpath_double #9219

Add support for XPath scalar functions - xpath,xpath_string,xpath_boolean,xpath_double #9219

Conversation

sajjoseph commented Sep 12, 2021 • edited Loading

ebyhr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wgzhao commented Nov 24, 2021

mosabua commented Oct 28, 2022

tooptoop4 commented Nov 19, 2022

colebow commented Nov 30, 2022

sajjoseph commented Sep 12, 2021 •

edited

Loading