PeC Scraper

A simple web scraper demo using Crystal and Chart.js

Some resources I used in building this:

• Crystal Documentation
• Crystal db package
• Xpather for testing Xpath
• Chart.js

Running the project:

• depending on your platform you may have to install libsqlite3, using Ubuntu this can be done with "sudo apt-get install libsqlite3-dev"
• You will need crystal and shards. I happened to use Crystal 1.11.2 as that is what was provided by my package manager.
• Install the dependencies using shards install
• Run the application using crystal pec_scraper.cr
• You should now be able to use the web interface at http://localhost:8080
• The scraper will run every 60s while the page is open, or when you click the "Scrape" button directly below the chart