is there alternative project like this project?

[socialpercon]

• pyspider version:
• Operating system:
• Start up command:

Expected behavior

Actual behavior

How to reproduce

is there alternative project like this project?

i don't understand why this project no longer maintainece. i think alternative project more powerful... but i don't know...

[Chaffy-0]

Scrapy

[JermellB]

This project isn't maintained any more because their javascript rendering capability is done by phantomjs which is no longer maintained.

Like @Chaffy-0 said, Scrapy is likely the best option if you wanted to do a spider like this.

These days, elasticsearch comes paired with one if you were doing something simple and didn't need to collect and process your own data from the wild.

Most places I've done stuff @ will use things like selenium + chrome or firefox, paired with beautiful soup for the rendered html parsing. Then you could keep track of where you'd spider with simple things like a bloom filter implemented on top of redis or something.

But yeah, Scrapy if you don't feel like getting too dirty.

[milahu]

some active python web scraper projects
https://github.com/Gerapy/Gerapy
https://github.com/howie6879/ruia

[roniemartinez]

Just in case people will be interested in my project 🙇 : https://github.com/roniemartinez/dude