[FEA] Implement cuIO fuzz tests

[vuule]

Depends on #6000

Python tests:

• Interop for data generators from [FEA] Implement data generators for different data profiles #6000
• Parquet tests;
• ORC tests;
• Save input on failure (params XML, seed);
• AVRO tests
• CSV tests
• JSON tests

To get started, tests can only vary the data profile, and keep the reader/writer parameters to defaults.

[vuule]

In this design we have the data generators implemented only in C++, including the parameter serialization to/from an XML file. The tests implemented in Python/Java are supposed to use the generator API through interop. The idea is to avoid duplicating the serialization code and get the best performance when generating inputs. @kkraus14 does this sound viable, at least for Python?

[vuule]

@nvdbaranec you suggested that there needs to be a way to replay the failing tests in C++ environment to facilitate debugging. The design here should allow that since the data generators are deterministic and the data profile and seed can be saved and passed to a C++ API to regenerate the same input.
Do you have any feedback/concerns about the approach?

[nvdbaranec]

That seems fine to me. It would be particularly useful if any failure message also included a quick blurb about how to reproduce. Something like:

"Test XYZ failed :
[test output]

Test was run with these parameters [5236783, 34217, etc]. To reproduce, call ABC() with these values"

[vuule]

Yeah, I neglected the call parameters in the diagram and the description here because we can get started with random data + defaults. You're right, the API and its parameters should definitely be included.
In the ideal case we would save the call parameters in a machine readable format and have a utility that executes the test based on all of this input.

[nvdbaranec]

Random comment on the kinds of things that seem to dig up bugs:

• Nulls and empties, especially in combination.

• Tweakable format meta-parameters such as page size and row group size in Parquet. Things that don't necessarily affect the formal format itself but squeeze the data into interestingly small or large blobs. The classic example from the parquet reader was page sizes sufficiently small to cause entire nested rows to neither begin, nor end within a single row-group page. I also strongly suspect we will see some (fixable) performance issues with very large page sizes.

• Very large and very small files.

• Tiny or otherwise fiendishly constructed reads. Reading 1 row out of millions in the middle of a file. Reading 2 rows that deliberately span units of parallelization (eg, 1 block per page in Parquet).

• Super-compressible vs terribly-compressible data. Billions of rows all containing the same value from a dictionary. That kind of thing.

• Things that deliberately break the reader, like files that will cause out-of-memory issues. The idea is to figure out how gracefully things fail, and how we might produce more useful output when it does.

• Files with huge numbers of columns vs huge numbers of rows.

[nvdbaranec]

"In the ideal case we would save the call parameters in a machine readable format and have a utility that executes the test based on all of this input."

I am a big fan of being able to reproduce the testing outside of the testing framework itself. I like working in my own little (debuggable) seperate test app. So consideration for providing simple code hooks that can be called would be great.

[vuule]

Out of the cases you listed, I see a potential problem with one:

Reading 1 row out of millions in the middle of a file. Reading 2 rows that deliberately span units of parallelization (eg, 1 block per page in Parquet).

For this type of testing (en masse) we have to rely on the reference implementation to validate the output. And I don't think the other Parquet implementations support row selection (outside of row group selection). I guess in this case we can crop the reference output before comparison.

I think we will hit all of the listed corner cases eventually with random parameter sampling as long as we set up the data profile parameters' ranges correctly.

[nvdbaranec]

True. If this really saturates the parameter space (implying long scale run times as you mentioned) it'll probably catch the interesting stuff. I would reiterate the point to make sure that we parameterize things that aren't specifically the data itself (eg page sizes).

[nvdbaranec]

Thought of another thing. This is probably not super relevant right now, but if we ever did overlapped reads with decoding (that is, read/decompress blob N+1 while decoding blob N) having tests that use both memory mapped files and true spinning-disk files might yield some race condition stuff.

[nvdbaranec]

Item : One of the tests I added to the python code for the parquet reader went long enough that it occasionally was causing the build machines to time out. The conclusion was that in the future we should be doing this kind of exhaustive/stress testing in the cpp tests instead of python, which makes sense.

As such, it would be pretty handy if whatever reference reader/writer code you use for this new test framework was also readily available in the normal cudf test framework.

[kkraus14]

Item : One of the tests I added to the python code for the parquet reader went long enough that it occasionally was causing the build machines to time out. The conclusion was that in the future we should be doing this kind of exhaustive/stress testing in the cpp tests instead of python, which makes sense.

Just a note that the 5 minute timeout is specifically defined for the Python unit tests and isn't a limitation of the CI framework / machine.