README.md

crawler-helper

A simplified crawler that yields visited urls and responses Results are returned immediately.

Example usage

start_url = 'https://example.com'
crawler = Crawler(start_url)
for url, response in crawler.start():
    print(f"URL: {url}")
    print(f"Response: {response.status_code}")
    print('-' * 50)