[AG-1386] Implement Error Catching for GX Data Validation

[BWMac]

Problem:

Previously, failures in GX data validation were not causing pipeline failures. As a result, visibility for these failures is poor and our data validation pipeline is relatively ineffective.

Solution:

Leverage our existing error-catching strategy to capture data validation failures by introducing a new error type which is triggered during GX data validation if validation for a dataset fails.

Notes:

• ADTDataValidationError is raised when data validation fails. This is caught by the try/except logic in process_all_files and a formatted string is printed when the pipeline concludes. Example.
• Needed tests are added for GreatExpectationsRunner.run and the new_function GreatExpectationsRunner.get_failed_expectations which is used to parse out which expectations failed and produce the error message.

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
3.1% Duplication on New Code

See analysis details on SonarCloud

[jaclynbeck-sage]

Looks good!

[JessterB]

This is a more general issue, but I can't align the example CI run you linked with a specific file or manifest version in Synapse, at least not without doing a ton of manual digging around that no one has time for.

The Synapse UI doesn't display full timestamps (just the date portion), there were 6 different "Agora Testing Data" processing runs on 3/5/23, and there is nothing logged in the CI job's output about what version the uploaded files or manifest (and also GE reports, once we collapse the reports into versioned files) end up with.

Would it be an easy update to log the version of each file after it's uploaded?

[BWMac]

@JessterB There wouldn't be a manifest uploaded when that run failed. Nor would there be an output data file for neuropath_corr uploaded because that particular iteration of process_dataset would not have made it to the upload step. Therefore, I think the versioning strategy would only really be useful for GX reports.