Program crawl whole data from thingiverse.com
Whole data is saved in postgreSQL volume. Moreover every html page of website is stored in html directory.
In /thingiverse_pro/last_nb/num_thing.txt user should add the latest id of thing in Thingiverse. Crawler start scrapy from the latest one to the oldest one.
docker volume create thingiverse_database
docker-compose up
- Scrapy
- Docker-compose
- PostgreSQL
- Psycopg2