Adds: mechanism for crawling only once

[spirosdelviniotis]

Description

• Adds: scrapy-crawl-once plug-in to setup.py.
• Adds: extends scrapy-crawl-once plug-in for supporting custom data-fields in DB per spider.
• Adds: enables scrapy-crawl-once plug-in.
• Adds: tests about scrapy-clawl-once for FTP and FILE.
• Re-factored: WSP spider and utils module in order not to check the crawled records.
• Removes: unused function for arxiv tests.
• Adds: create temporary folder to unzip the crawled files and in InspireCeleryPushPipeline.close_spider methods deletes this temporary folder.
• Adds: new default argument to clean_dir method is the generated DB folder from scrapy-crawl-once plug-in.
• Adds: to tests clean_dir to tests that using the DB.
• Adds: makes staticmethod _get_collections class method.
• Adds: minor indentation fix.
• Removes: unused import.
• Adds: make InspireAPIPushPipeline._cleanup staticmethod.
• Adds: variable in settings.py to specify where to store scrapy-crawl-once DB.

Signed-off-by: Spyridon Delviniotis [email protected]

Related Issue

Closes #161

Motivation and Context

Checklist:

• I have all the information that I need (if not, move to RFC and look for it).
• I linked the related issue(s) in the corresponding commit logs.
• I wrote good commit log messages.
• My code follows the code style of this project.
• I've added any new docs if API/utils methods were added.
• I have updated the existing documentation accordingly.
• I have added tests to cover my changes.
• All new and existing tests passed.

[david-caro]

@staticmethod
def _is_newer(this_timestamp, than_this_timestamp):
    return this_timestamp < than_this_timestamp

def _has_to_be_crawled(self, reques, spider):    
        request_db_key = self._get_key(request)

        if not request_db_key in self.db:
            return True

        new_request_timestamp = self._get_timestamp(request, spider)
        parsed_url = urlparse(request.url)
        if self._is_newer(
            new_request_timestamp,
            self.db.get(key=request_db_key),
        ):
                    return True

    return False

def process_request(self, request, spider):
   # ... populate crawl_once_value and key
    if not request.meta.get('crawl_once', self.default):
        return

    if not self._has_to_be_crawled(request, spider):
        self.stats.inc_value('crawl_once/ignored')
        raise IgnoreRequest()

[david-caro]

if (
    conference or
    current_journal_title == (
        "basbalbalbal"
        "ntshnsthsnthsnth"
    )
):
    ...

[david-caro]

delete

[david-caro]

move this before line 125

[david-caro]

You can move this to a method called get_relative_path or similar so it's clearer.

[david-caro]

You can move all this if block to get_ftp_timestamp

[david-caro]

instead of this you should use a configuration override to set it to somewhere inside the test directory.