Detecting missing links and 404s

[asitemade4u]

The developer of the website I intend to scrape information from is sloppy and has left a lot of broken links.
When I execute an otherwise effective Ferret script on a list of pages, it stops altogether at every 404.
Is there a DOCUMENT_EXISTS or anything that would help the script go on?

[ziflex]

Nope, there is no such a function. But we can come up with something like that.

[ksdme]

@ziflex I would like to pick this issue.

[ziflex]

@asitemade4u could you provide some links Ferret is failing on?

404 are handled if a target website handles them too:

LET p = DOCUMENT('http://google.com/fdsfsd', { driver: "cdp" })

RETURN p.response.statusCode