Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add mechanism for crawling only once #161

Closed
spirosdelviniotis opened this issue Aug 17, 2017 · 0 comments · Fixed by #162
Closed

Add mechanism for crawling only once #161

spirosdelviniotis opened this issue Aug 17, 2017 · 0 comments · Fixed by #162
Assignees
@spirosdelviniotis
Milestone
Desy spider & PoS...

Comments

@spirosdelviniotis
Copy link
Contributor

We have to find a way not to crawl many times the same records.

Expected Behavior

We are going to extend the scrapy-crawl-once plug-in.

Current Behavior

Hepcrawl re-crawls records generated from previous executions.

Steps to Reproduce (for bugs)

  1. Adapt scrapy-crawl-once plug-in to Hepcrawl.
  2. Extend the scrapy-crawl-once plug-in in a way that stores in the DB a key-value record for every request. As key we have the unique file name (FTP-FILE requests) or the unique id in the parameters (HTTP-HTTPS requests). As value we store the last-modified time stamp (FTP-FILE requests) or the crawling time stamp (HTTP-HTTPS requests).

Context

We are trying to crawl only once every record.

Screenshots (if appropriate):
@spirosdelviniotis spirosdelviniotis self-assigned this Aug 17, 2017
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 17, 2017
@spirosdelviniotis
setup: add scrapy-crawl-once plug-in
0532e78 
Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 17, 2017
@spirosdelviniotis
middlewares: add support for crawling only once
199ad7b 
* Adds: extends `scrapy-crawl-once` plug-in for supporting custom data-fields in DB per spider.
* Adds: enables `scrapy-crawl-once` plug-in.

Closes inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 17, 2017
@spirosdelviniotis
middlewares: add support for crawling only once
62da410 
* Adds: extends `scrapy-crawl-once` plug-in for supporting custom data-fields in DB per spider.
* Adds: enables `scrapy-crawl-once` plug-in.

Closes inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 17, 2017
@spirosdelviniotis
tests: support for scrapy-crawl-once
5930c13 
Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 17, 2017
@spirosdelviniotis
tests: add tests for scrapy-clawl-once
eae96e4 
* Adds: tests about `scrapy-clawl-once` for FTP and FILE.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 18, 2017
@spirosdelviniotis
wsp: refactored to support scrapy-crawl-once
b258ce7 
* Re-factored: `WSP` spider and `utils` module in order not to check the crawled records.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 18, 2017
@spirosdelviniotis
wsp: add temporary folder to the crawlings
da7a3b3 
* Adds: create temporary folder to unzip the crawled files and in `InspireCeleryPushPipeline.close_spider`
	methods deletes this temporary folder.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 18, 2017
@spirosdelviniotis
testlib: refactored clean_dir default arguments
adf4d7b 
* Adds: new default argument to `clean_dir` method is the generated DB folder from `scrapy-crawl-once` plugin.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 18, 2017
@spirosdelviniotis
tests: add clean_dir to tests that using the DB
b014690 
Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 18, 2017
@spirosdelviniotis
wsp: add temporary folder to the crawlings
063663b 
* Adds: create temporary folder to unzip the crawled files and in `InspireCeleryPushPipeline.close_spider`
	methods deletes this temporary folder.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 18, 2017
@spirosdelviniotis
testlib: refactored clean_dir default arguments
f50d983 
* Adds: new default argument to `clean_dir` method is the generated DB folder from `scrapy-crawl-once` plugin.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 18, 2017
@spirosdelviniotis
tests: add clean_dir to tests that using the DB
5de385f 
Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 18, 2017
@spirosdelviniotis
global: add CRAWL_ONCE_PATH to settings
f694856 
* Adds: variable in `settings.py` to specify where to store `scrapy-crawl-once` DB.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
@spirosdelviniotis spirosdelviniotis mentioned this issue Aug 18, 2017
Merged
8 tasks
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 21, 2017
@spirosdelviniotis
middlewares: add support for crawling only once
dafcd3b 
* Adds: extends `scrapy-crawl-once` plug-in for supporting custom data-fields in DB per spider.
* Adds: enables `scrapy-crawl-once` plug-in.

Closes inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 21, 2017
@spirosdelviniotis
tests: support for scrapy-crawl-once
06e106f 
Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 21, 2017
@spirosdelviniotis
tests: add tests for scrapy-clawl-once
5758194 
* Adds: tests about `scrapy-clawl-once` for FTP and FILE.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 21, 2017
@spirosdelviniotis
wsp: refactored to support scrapy-crawl-once
0a8920d 
* Re-factored: `WSP` spider and `utils` module in order not to check the crawled records.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 21, 2017
@spirosdelviniotis
wsp: add temporary folder to the crawlings
e9fb27a 
* Adds: create temporary folder to unzip the crawled files and in `InspireCeleryPushPipeline.close_spider`
	methods deletes this temporary folder.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 21, 2017
@spirosdelviniotis
testlib: refactored clean_dir default arguments
0c0a66b 
* Adds: new default argument to `clean_dir` method is the generated DB folder from `scrapy-crawl-once` plugin.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 21, 2017
@spirosdelviniotis
tests: add clean_dir to tests that using the DB
3bdf93c 
Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 21, 2017
@spirosdelviniotis
global: add CRAWL_ONCE_PATH to settings
abf2c97 
* Adds: variable in `settings.py` to specify where to store `scrapy-crawl-once` DB.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
spirosdelviniotis added a commit to spirosdelviniotis/hepcrawl that referenced this issue Aug 21, 2017
@spirosdelviniotis
global: add CRAWL_ONCE_PATH to settings
9d28bfd 
* Adds: variable in `settings.py` to specify where to store `scrapy-crawl-once` DB.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to david-caro/hepcrawl that referenced this issue Aug 25, 2017
@spirosdelviniotis @david-caro
setup: add scrapy-crawl-once plug-in
7adc38e 
Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to david-caro/hepcrawl that referenced this issue Aug 25, 2017
@spirosdelviniotis @david-caro
middlewares: add support for crawling only once
85f4f36 
* Adds: extends `scrapy-crawl-once` plug-in for supporting custom data-fields in DB per spider.
* Adds: enables `scrapy-crawl-once` plug-in.

Closes inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to david-caro/hepcrawl that referenced this issue Aug 25, 2017
@spirosdelviniotis @david-caro
tests: support for scrapy-crawl-once
dab61e1 
Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to david-caro/hepcrawl that referenced this issue Aug 25, 2017
@spirosdelviniotis @david-caro
tests: add tests for scrapy-clawl-once
92b0efa 
* Adds: tests about `scrapy-clawl-once` for FTP and FILE.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to david-caro/hepcrawl that referenced this issue Aug 25, 2017
@spirosdelviniotis @david-caro
wsp: refactored to support scrapy-crawl-once
4054da8 
* Re-factored: `WSP` spider and `utils` module in order not to check the crawled records.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to david-caro/hepcrawl that referenced this issue Aug 25, 2017
@spirosdelviniotis @david-caro
wsp: add temporary folder to the crawlings
5f6f873 
* Adds: create temporary folder to unzip the crawled files and in `InspireCeleryPushPipeline.close_spider`
	methods deletes this temporary folder.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to david-caro/hepcrawl that referenced this issue Aug 25, 2017
@spirosdelviniotis @david-caro
testlib: refactored clean_dir default arguments
5404136 
* Adds: new default argument to `clean_dir` method is the generated DB folder from `scrapy-crawl-once` plugin.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to david-caro/hepcrawl that referenced this issue Aug 25, 2017
@spirosdelviniotis @david-caro
tests: add clean_dir to tests that using the DB
45d1693 
Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to david-caro/hepcrawl that referenced this issue Aug 25, 2017
@spirosdelviniotis @david-caro
global: add CRAWL_ONCE_PATH to settings
b6c2821 
* Adds: variable in `settings.py` to specify where to store `scrapy-crawl-once` DB.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to spirosdelviniotis/hepcrawl that referenced this issue Sep 20, 2017
@spirosdelviniotis @david-caro
wsp: add temporary folder to the crawlings
6897d81 
* Adds: create temporary folder to unzip the crawled files and in `InspireCeleryPushPipeline.close_spider`
	methods deletes this temporary folder.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to spirosdelviniotis/hepcrawl that referenced this issue Sep 20, 2017
@spirosdelviniotis @david-caro
testlib: refactored clean_dir default arguments
1a95836 
* Adds: new default argument to `clean_dir` method is the generated DB folder from `scrapy-crawl-once` plugin.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to spirosdelviniotis/hepcrawl that referenced this issue Sep 20, 2017
@spirosdelviniotis @david-caro
tests: add clean_dir to tests that using the DB
b4d1009 
Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to spirosdelviniotis/hepcrawl that referenced this issue Sep 20, 2017
@spirosdelviniotis @david-caro
global: add CRAWL_ONCE_PATH to settings
245ed9b 
* Adds: variable in `settings.py` to specify where to store `scrapy-crawl-once` DB.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to spirosdelviniotis/hepcrawl that referenced this issue Sep 20, 2017
@spirosdelviniotis @david-caro
setup: add scrapy-crawl-once plug-in
15ab37d 
Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to spirosdelviniotis/hepcrawl that referenced this issue Sep 20, 2017
@spirosdelviniotis @david-caro
middlewares: add support for crawling only once
7694fe3 
* Adds: extends `scrapy-crawl-once` plug-in for supporting custom data-fields in DB per spider.
* Adds: enables `scrapy-crawl-once` plug-in.

Closes inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to spirosdelviniotis/hepcrawl that referenced this issue Sep 20, 2017
@spirosdelviniotis @david-caro
tests: support for scrapy-crawl-once
a283b21 
Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to spirosdelviniotis/hepcrawl that referenced this issue Sep 20, 2017
@spirosdelviniotis @david-caro
tests: add tests for scrapy-clawl-once
1301d25 
* Adds: tests about `scrapy-clawl-once` for FTP and FILE.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to spirosdelviniotis/hepcrawl that referenced this issue Sep 20, 2017
@spirosdelviniotis @david-caro
wsp: add temporary folder to the crawlings
a758819 
* Adds: create temporary folder to unzip the crawled files and in `InspireCeleryPushPipeline.close_spider`
	methods deletes this temporary folder.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to spirosdelviniotis/hepcrawl that referenced this issue Sep 20, 2017
@spirosdelviniotis @david-caro
testlib: refactored clean_dir default arguments
28a2603 
* Adds: new default argument to `clean_dir` method is the generated DB folder from `scrapy-crawl-once` plugin.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to spirosdelviniotis/hepcrawl that referenced this issue Sep 20, 2017
@spirosdelviniotis @david-caro
tests: add clean_dir to tests that using the DB
026bdfb 
Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
david-caro pushed a commit to spirosdelviniotis/hepcrawl that referenced this issue Sep 20, 2017
@spirosdelviniotis @david-caro
global: add CRAWL_ONCE_PATH to settings
a82a158 
* Adds: variable in `settings.py` to specify where to store `scrapy-crawl-once` DB.

Addresses inspirehep#161

Signed-off-by: Spiros Delviniotis <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@spirosdelviniotis spirosdelviniotis

Labels
None yet
Projects
None yet
Milestone
Desy spider & PoS specification
Development

Successfully merging a pull request may close this issue.

Adds: mechanism for crawling only once spirosdelviniotis/hepcrawl
1 participant
@spirosdelviniotis