Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix cache: Don't check one url multiple times #687

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

PetrDlouhy
Copy link

@PetrDlouhy PetrDlouhy commented Nov 9, 2016

As mentioned in #608, Linkchecker does multiple checks for one URL. It is caused by inconsistent use of cache. The URL is added to the queue when checking page, but it doesn't get to cache until the moment of checking the URL itself. In the meantime, it can be added to queue multiple times.

This PR should fix it.

@PetrDlouhy
Copy link
Author

Travis tests are working, so only tests are waiting.

@PetrDlouhy PetrDlouhy force-pushed the master branch 2 times, most recently from 3e7fc19 to 2b9dafa Compare January 26, 2017 04:46
@PetrDlouhy PetrDlouhy changed the title (not complete) Fix cache: Don't check one url multiple times Fix cache: Don't check one url multiple times Jan 26, 2017
@PetrDlouhy
Copy link
Author

Now it should be finished. It includes tests for urlqueue.

@anarcat
Copy link

anarcat commented Jan 31, 2017

please reroll this PR in the new organisation here: https://github.com/linkcheck/linkchecker/pulls see #686 for details

@dpalic
Copy link

dpalic commented Jun 9, 2017

would be a great feature to merge into mainstream. Is there any timeplan for this?
The current checking is too much load generating to the target page/server for checking over and over the same URL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants