[FEA] Expose json_recovery_mode in cudf.read_json Python API, and add a column with filenames index

[miguelusque]

Is your feature request related to a problem? Please describe.
Hi!

When passing a list of files to cudf.read_json, it might fail because any of the json objects cannot be parsed because of any reason.

With the current Python API, we cannot distinguish which file failed, so it makes hard to debug those failures when dealing with thousands of jsonl files (for instance, when curating a corpus for LLM training).

Therefore, if the following functionalities would be available:

• Create a row with nulls when one of the json objects of a file canot be read, and
• Optionally, adding a column containing the index to the files passed to cudf.read_json method that indicates to which file corresponds each row in the dataset.

With those two functionalities, reading thousands of jsonl files in parallel while managing exceptions while reading them would be much efficient (reading files in a list is several times faster than reading them one after the other).

Already discussed with @vuule.

Thanks!

P.S.: NeMo Curator may benefit of this FR when performing Exact Deduplication, Fuzzy Deduplication and Download and Extract corpus features.

[GregoryKimball]

Hello @shrshi, would you please add python bindings for the JSON reader option json_recovery_mode?

As far as the data source indexes.. we can leave that for a follow-on PR.

[GregoryKimball]

Hello @galipremsagar, it looks like @shrshi is getting pulled deeper into the JSON reader tree algorithms, would you please help us add the json_recovery_mode python binding for NeMo Curator?

[galipremsagar]

Sure, I'll take care of it.

[shrshi]

Thank you @galipremsagar

[miguelusque]

Thank you all! Much appreciated!