Linkchecker

[dylanPowers]

I found a tool in python that checks for dead or invalid links. I went ahead and wrote up some directions on how to use it while also setting up a small configuration file.

This fixes #13

[jbenet]

oof. how complicated is it? could we have a go replacement?

also, ipfs is reliable enough to start "vendoring" deps to ipfs itself.

(having to manually run pip is frustrating and complicates usage and CI)

[harlantwood]

Hm, how bout a nodejs solution, as the website is already built by node tools?

Two options from a quick google:

• https://www.npmjs.com/package/broken-link-checker
• https://www.npmjs.com/package/grunt-link-checker

[jbenet]

@harlantwood those sounds good to me. thoughts @dylanPowers ?

[dylanPowers]

I checked out the node implementations first and for the most part they weren't very feature complete and buggy.

• https://www.npmjs.com/package/broken-link-checker -> NPM barfs on this one and it won't install
• https://www.npmjs.com/package/metalsmith-broken-link-checker -> Decided every single link was broken
• ...looked at others and at that point I got discouraged on the node solutions
• https://www.npmjs.com/package/grunt-link-checker -> Since you pointed it out I'll set up a grunt file and look into it. It may even be able to be pulled out of having a dependency on grunt https://github.com/ChrisWren/grunt-link-checker/blob/master/tasks/link-checker.js

This python implementation is the most mature and feature complete at this point in time. It's also a non-trivial app https://github.com/wummel/linkchecker/tree/master/linkcheck

Using go would still introduce another dependency so I'm not sure that's a great solution either, but there are some options there as well that could be looked at.

[whyrusleeping]

python is installed by default on pretty much every linux machine. I dont really think having it as a dependency is all that bad

[harlantwood]

FWIW this is the patch to get broken-link-checker installing -- just posted to his repo --

For now you can put this in your package.json to get the head from github:

"broken-link-checker": "stevenvachon/broken-link-checker"

[harlantwood]

But admittedly its not mature. I agree that python dependency is not so bad, tho it would be elegant to have a node solution.

[jbenet]

I don't want to depend on pip. it can be a nightmare. can we check the deps we need into ipfs and seed them? (assume python, just the modules)

[dylanPowers]

Update on the node options:

• https://www.npmjs.com/package/broken-link-checker -> Got it to install. It doesn't recursively check through all the links in a site. It appears to only check the links on the page you give it.
• https://www.npmjs.com/package/grunt-link-checker -> Relies upon https://github.com/cgiffard/node-simplecrawler. Currently there's a bug where it will stall out for a few minutes after a number of requests and it also doesn't check external links.

[harlantwood]

Maybe this will help? From the simplecrawler docs:

crawler.filterByDomain - Specifies whether the crawler will restrict queued requests to a given domain/domains.

[dylanPowers]

From my interpretation of the docs, filterByDomain means that the crawler will crawl the entire external website. Also mentioned in simplecrawler/simplecrawler#114. A pull request has been made to enable the intended functionality we want in simplecrawler/simplecrawler#74 (not quite what is wanted). I'm going to go ahead and fork + merge that pull request and see if it works.
I'll also look into that bug I'm running into. At the moment, it looks like it will be easier to do that than try to get rid of pip in the python app.

Update:
https://www.npmjs.com/package/grunt-link-checker simply isn't going to work. node-simple crawler isn't a good use for a link checker because it's messy in the way it finds links. It's using regexs rather than actually parsing the html so it creates a lot of false positives and doesn't understand things like base tags (simplecrawler/simplecrawler#57). So back to the python app.