diff --git a/docs/compatibility.md b/docs/compatibility.md index 31d084a9a46..b8dcca65f53 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -296,38 +296,21 @@ The JSON format read is a very experimental feature which is expected to have so it by default. If you would like to test it, you need to enable `spark.rapids.sql.format.json.enabled` and `spark.rapids.sql.format.json.read.enabled`. -Currently, the GPU accelerated JSON reader doesn't support column pruning, which will likely make -this difficult to use or even test. The user must specify the full schema or just let Spark infer -the schema from the JSON file. eg, - -We have a `people.json` file with below content - +Reading input containing invalid JSON format (in any row) will throw runtime exception. +An example of valid input is as following: ``` console -{"name":"Michael"} {"name":"Andy", "age":30} {"name":"Justin", "age":19} ``` -Both below ways will work - -- Inferring the schema - - ``` scala - val df = spark.read.json("people.json") - ``` - -- Specifying the full schema - - ``` scala - val schema = StructType(Seq(StructField("name", StringType), StructField("age", IntegerType))) - val df = spark.read.schema(schema).json("people.json") - ``` - -While the below code will not work in the current version, +The following input is invalid and will cause error: +```console +{"name":"Andy", "age":30} ,,,, +{"name":"Justin", "age":19} +``` -``` scala -val schema = StructType(Seq(StructField("name", StringType))) -val df = spark.read.schema(schema).json("people.json") +```console +{"name": Justin", "age":19} ``` ### JSON supporting types