changed: Updated HcfMiddleware. Fixes #56
#57
Conversation
| def _save_new_links_count(self): | ||
| """ Save the new extracted links into the HCF.""" | ||
| for slot, new_links in self.new_links.items(): | ||
| self._msg('Stored %d new links in slot(%s)' % (len(new_links), slot)) |
There was a problem hiding this comment.
Using len is the same as storing the counts yourself.
There was a problem hiding this comment.
The purpose of storing the counts myself (instead of len) is to avoid storing 100 million strings in new_links in my case. Now with hcf_dont_filter set to True in request.meta, new_links could be empty and space saved. :)
There was a problem hiding this comment.
But where are you cleaning new_links? I'm confused :P
There was a problem hiding this comment.
Previously new_links is cleaned in close_spider. However I do not think that is necessary since the spider is closing, everything here would be cleaned once the instance of.HcfMiddleware is recycled.
|
What's the verdict on this one? |
|
@starrify , is the issue and fix still valid? |
No description provided.