get_json_object() implementation

[nvdbaranec]

An implementation of get_json_object().

Reference: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-get_json_object

The fundamental functionality here is running a JSONPath query on each row in an input column of json strings.

JSONPath spec: https://tools.ietf.org/id/draft-goessner-dispatch-jsonpath-00.html

For review purposes, the key entry point is parse_json_path(). Each thread of the kernel processes 1 row via this function. The behavior is recursive in nature but we maintain our own context stack to do it in loop fashion.

parse_json_path is just the high level controlling logic, with most of the heavy lifting happening in the json_state parser class. Though the "heavy lifting" is pretty much just traditional string parsing code.

The path to optimization here (I'll open a separate cudf issue for this) is

• Change parse_json_path to work on a warp basis. So each row in the column would be processed by one warp.
• Make the json_state parser class thread/warp aware (the class would just store its tid and operate accordingly). I think this is reasonably straightforward to do as most of the cuIO decoding kernels behave like this.

[harrism]

@nvdbaranec the docs you linked for get_json_object do not say what the function does. Can you please provide a description of what the code in this PR does?

[kkraus14]

From here: https://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=udf#pyspark.sql.functions.get_json_object

Extracts json object from a json string based on json path specified, and returns json string of the extracted json object. It will return null if the input json string is invalid.

[harrism]

Let's discuss whether libcudf is the right home for this functionality.

[harrism]

Let's discuss whether libcudf is the right home for this functionality.

Criteria:

1. Is it broadly applicable for libcudf users?
   A: I don't know. So far I know it's important to one Spark use case.
2. Is it uniquely enabled by libcudf features and internals? (would it be a lot harder to support outside of libcudf)
   A: It seems to be fairly independent, custom CUDA code that is not heavily dependent on other libcudf features or data structures. So I think the answer is no.

(I'm sure there are other criteria, but those two come quickly to mind.)

[kkraus14]

1. Is it broadly applicable for libcudf users?

@randerzander just recently pinged me inquiring supporting a function like this as well, so I think we can assume some type of functionality like this is broadly applicable. @randerzander do you need it purely in the case of parsing input JSONLines or do you have use cases where you end up with a String column of JSON formatted strings?

[randerzander]

1. Is it broadly applicable for libcudf users?

@randerzander just recently pinged me inquiring supporting a function like this as well, so I think we can assume some type of functionality like this is broadly applicable. @randerzander do you need it purely in the case of parsing input JSONLines or do you have use cases where you end up with a String column of JSON formatted strings?

Yes, there are use-cases where you need to extract the string of whatever existed (could be more nested JSON, could be a numeric, a string, a list) at the specified JSON path.

[nvdbaranec]

rerun tests

[davidwendt]

Just some headers that I think can be removed.

[jrhemstad]

@gpucibot

[nvdbaranec]

@gpucibot merge

[nvdbaranec]

@gpucibot merge