Workflow

The harvesting is composed as the following :

1. For each publisher, we fetch all new articles from their associated source. Some publishers store them in FTP servers, whereas other do it with a REST API.
2. Everything is then pushed to our S3 server in the publisher's bucket.
3. The publisher-specific parsing gets all information from the article to have it in our internal format.
4. The generic parsing restructures the internal format and cleans some data.
5. The enhancement adds additional fields based on existing fields. For example adding the creation date of parsed article.
6. The enrichment adds additional fields based on external sources. For example adding Arxiv categories.
7. The final article is then verified to ensure that is is compliant with the JSON Schema. The JSON article is then pushed to PSQL.