Skip to content
/ crawl Public

The application consists of a command line client and a local service (daemon) which performs the actual web crawling. The communication between client and server should use a form of IPC mechanism. For each URL, the Web Crawler creates a tree of links with the root of the tree being the root URL.

Notifications You must be signed in to change notification settings

rowanfr/crawl

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

crawl

The application consists of a command line client and a local service (daemon) which performs the actual web crawling. The communication between client and server uses a form of IPC mechanism. For each URL, the Web Crawler creates a tree of links with the root of the tree being the root URL.

The commands for the application are as follows:

  • -start
    This starts the daemon if one doesn't exist
  • -start url
    This starts the application and tasks the daemon with scraping a url if both that url and daemon exist
  • -stop url
    This stops the url from being scraped
  • -list
    This lists all scraped urls to the terminal
  • -clear
    This clears all files related to the daemon
  • -kill
    This kills the daemon and then clears all files related to the daemon
  • -print
    This prints out the scraped urls to output.txt

About

The application consists of a command line client and a local service (daemon) which performs the actual web crawling. The communication between client and server should use a form of IPC mechanism. For each URL, the Web Crawler creates a tree of links with the root of the tree being the root URL.

Resources

Stars

Watchers

Forks

Packages

No packages published

Languages