Gogo Crawler

Gogo Crawler is a web crawling tool designed to crawl the web and store the data in a MongoDB database.

Table of Contents

• Prerequisites
• Installation
• Setup
• Usage
• Credits

Prerequisites

Before you begin, ensure you have met the following requirements:

• You have installed Node.js and Bun
• You have a running instance of MongoDB

Installation

1. Clone the repository:

   git clone https://github.com/DeveloperJosh/gogo-crawler.git
   cd gogo-crawler

2. Install the dependencies:

   bun install

Setup

1. Create a .env file in the root of the project and add your MongoDB connection details:

   MONGO_URL=your_mongo_url

2. Ensure your MongoDB instance is running and accessible.

Usage

To start the Gogo Crawler, run the following command:

bun crawl

And that's it! The crawler will start crawling the web and storing the data in your MongoDB database.

Credits

This project was inspired by the anime-crawler project by riimuru.