Crawling the Homepage of Gatech CoC Faculty

A simple crawler to crawl the home page url of Gatech CoC faculty and download their homepage. (https://www.cc.gatech.edu/people/faculty)

node version v8.12.0
npm version 6.4.1

Installation

npm install
node app.js

Crawl the index page from Gatech CoC

• Sample Output

(Num Faculty: 183)
--------------------
https://www.cc.gatech.edu/people/jacob-abernethy
https://www.cc.gatech.edu/people/gregory-abowd
https://www.cc.gatech.edu/people/mustaque-ahamad
https://www.cc.gatech.edu/people/srinivas-aluru
https://www.cc.gatech.edu/people/mostafa-ammar
https://www.cc.gatech.edu/people/annie-anton
https://www.cc.gatech.edu/people/ronald-arkin
https://www.cc.gatech.edu/people/rosa-arriaga
https://www.cc.gatech.edu/people/joy-arulraj
--------------------
https://www.cc.gatech.edu/people/david-bader
https://www.cc.gatech.edu/people/albert-badre
https://www.cc.gatech.edu/people/tucker-balch
https://www.cc.gatech.edu/people/rahul-basole
https://www.cc.gatech.edu/people/robert-bates
https://www.cc.gatech.edu/people/dhruv-batra
https://www.cc.gatech.edu/people/michael-best
https://www.cc.gatech.edu/people/ketan-bhardwaj
https://www.cc.gatech.edu/people/douglas-blough
https://www.cc.gatech.edu/people/gregory-bodwin
https://www.cc.gatech.edu/people/ian-bogost
https://www.cc.gatech.edu/people/alexandra-boldyreva
--------------------
https://www.cc.gatech.edu/people/elijah-cameron
https://www.cc.gatech.edu/people/umit-v-catalyurek
https://www.cc.gatech.edu/people/richard-catrambone
https://www.cc.gatech.edu/people/polo-chau
https://www.cc.gatech.edu/people/sonia-chernova
https://www.cc.gatech.edu/people/lucio-chiaraviglio
https://www.cc.gatech.edu/people/edmond-chow
https://www.cc.gatech.edu/people/henrik-i-christensen
https://www.cc.gatech.edu/people/xu-chu
https://www.cc.gatech.edu/people/pak-ho-simon-chung
https://www.cc.gatech.edu/people/russell-clark
https://www.cc.gatech.edu/people/thomas-conte
--------------------

Web Scrape to get Homepage URL

• Sample Output

Jacob Abernethy | College of Computing
https://www.cc.gatech.edu/~jabernethy9/
Gregory Abowd | College of Computing
http://www.cc.gatech.edu/~abowd
Mustaque Ahamad | College of Computing
https://iisp.gatech.edu/mustaque-ahamad
Srinivas Aluru | College of Computing
http://www.cc.gatech.edu/~saluru
Mostafa Ammar | College of Computing
http://www.cc.gatech.edu/~ammar
Annie Antón | College of Computing
http://www.cc.gatech.edu/~aianton/
Ronald Arkin | College of Computing
https://www.cc.gatech.edu/aimosaic/faculty/arkin/
Rosa Arriaga | College of Computing
https://sites.google.com/site/riarriaga/
Joy Arulraj | College of Computing
https://www.cc.gatech.edu/~jarulraj/
David A. Bader | College of Computing
http://www.cc.gatech.edu/~bader

Download the Homepage

• Sample Output