feature: Added Cerberus as a new option for item validation

[vipulgupta2048]

🐣
This brings an end to a great and fulfilling period of contributing to Spidermon and the Scrapy Project as part of Google Summer of Code 2019.

Google Summer of Code 2019 with The Scrapy Project

Project - Integrate Cerberus, solves #182,
Project Description- Google Archive

Co-Org Admin - Cathal
Mentors - @rennerocha , @ejulio
Personal GSoC Blog - Mixster x GSoC
PSF Blog - https://blogs.python-gsoc.org/en/blogs/vipulgupta2048s-blog/

Description

• Spidermon is a recommended tool for monitoring spiders created using Scrapy. The user at the time can choose between two libraries for item validation rules: jsonschema and schematics. We want to provide a third option that being Cerberus.
• Cerberus provides powerful yet simple and lightweight data validation functionality out of the box and is designed to be easily extensible, allowing for custom validation. It has no dependencies and is thoroughly tested on several Python versions.
• The goal of this project was to integrate, test and enable Cerberus as a new option for item validation available for the user.

Deliverables & Work Done

1. All Code of highest quality standards having detailed documentation, black​ styling and well tested (Pull request – feature: Added Cerberus as a new option for item validation #201)
   This Pull Request Includes:

• CerberusValidator() class for Item validation through Cerberus. (Cerberus Validator : Validate() Method vipulgupta2048/spidermon#2)
• Translator for translating errors for a better, unified system working with other validation methods. (Implement Translator  vipulgupta2048/spidermon#4)
• Complete integration with Scrapy pipelines, working with raw schema, URL’s, and paths. (Integration Cerberus  vipulgupta2048/spidermon#5)

2. Unit + integration tests for each component in place.
3. Documentation for Cerberus Validation method. (Spidermon x Cerberus Documentation vipulgupta2048/spidermon#6)
4. A ​detailed, well-documented tutorial​ will be developed during the course of the summers implementing almost every feature of Spidermon to help developers as a reference and blogs will be written.
5. One blog ​each week​ regarding Spidermon and my experience, learning through the project on ​Mixster x GSoC.
6. For the community to track progress, a ​tracker was maintained with my latest developments containing week-to-week updates​, and MoM of mentor meetings​. This helps to maintain accountability​, transparency and keeping track.​

For system testing, one could go ahead and use the pre-configured Quotes spider https://github.com/vipulgupta2048/testing_quotes and installing Spidermon from the master branch of my fork.

Looking forward to improving the source code even further, through all your valued opinions, reviews, and comments. Would love to clarify and help understand the work done.

This project has been completed with long nights of reading and writing the code, learning new concepts on the fly and asking hundreds of pop-questions on Slack, that were answered duly by my mentors @ejulio @rennerocha as without their constant help, motivation, and guidance completing this uphill task wouldn't be ever possible.

[Gallaecio]

The documentation does not mention either that the schema may be specified as a string.

I wonder if we should remove support for this, and instead allow paths to JSON files regardless of their file extension, just as we don’t require a file extension or Content-Type header when using a URL.

[vipulgupta2048]

From my perspective, I feel like adding raw schemas directly to the settings.py file as strings for validation is something users wouldn't feel like doing. Also, only my code (Cerberus Validation) has tests and docs for adding raw schemas because I thought that it would be a good feature to work with.

Bit of context behind this change:
The code is from the original file is /jsonschema/tools.py, and it was moved, tweaked, and was made available for both Cerberus and JSONSchema to use. Hence, these lines are from the original file only.

[Gallaecio]

@rennerocha What are your thoughts on this?

[vipulgupta2048]

@rennerocha How should we proceed?

[vipulgupta2048]

@rennerocha pinging again. 😅

[vipulgupta2048]

@rennerocha Pinging again 🙈

[vipulgupta2048]

Hi @rennerocha, we need your comments in some reviews 😅, have you checked them out yet.

[rennerocha]

We will not add a new item validation library as built-in in Spidermon now, so we will close this PR. If needed, it can be created as a separated package.