Automated Image Crawler

Documentation

It is a simple image crawler with Selenium, but its differentiation is that it can detect objects in an image with a customized pre-trained model (Yolov5) and improve the model's performance using crawled images.

# I used it to crawl anus diseases images.

Used Skills

- Cloud: AWS S3, EC2, RDS (PostgreSQL)
- Container: Docker, Docekr Compose
- CI/CD: GitHub Action, AWS CodeDeploy
- Automation: Airflow

Install

Clone repo and install requirements.txt

git clone https://github.com/farmboy-dev/Anus  # clone
pip install -r requirements.txt  # install

Collector

If the object is detected -> Crawl the image

# at Anus
python ./yolov5/image_collector.py

Verifier

The customized pre-trained model is not perfect, so sometime it detects a wrong object. That is reason why I made the verifier to feed good quality images to the model.

# at Anus
python ./yolov5/image_verifier.py

Cropper

My purpose is to detect anus disease (object) and crop the object not whole image.

# at Anus
python ./yolov5/image_cropper.py

Further Tasks

1. Data Version Control
2. Tracking a status of models (performace, reproducibility, etc)