Skip to content

Latest commit

 

History

History
148 lines (110 loc) · 5.58 KB

README.md

File metadata and controls

148 lines (110 loc) · 5.58 KB
Raw

Gateway Proxy Server

Overview

The Gateway Proxy Server is a robust and scalable solution designed to manage and distribute traffic through a pool of proxy servers. It includes functionalities for dynamically adding and removing proxies, monitoring their availability, and intelligently routing traffic based on proxy status and host-specific rules. This server is particularly useful for scenarios where reliable proxy management and distribution are critical, such as web scraping, SEO monitoring, or accessing geo-restricted content.

Key Features

  1. Dynamic HTTP Proxy Management:

    • Add and remove proxies on the fly using RESTful API endpoints.
    • Each proxy is stored with detailed metadata, including its status (alive), response times, and usage statistics per host.

  2. Host-Specific Proxy Status:

    • Track the availability of proxies not just globally but also on a per-host basis.
    • Proxies can be marked as unavailable for specific hosts if they are detected as blocked or non-functional.

  3. Intelligent Proxy Selection:

    • Proxies are selected based on their availability and usage statistics, ensuring a balanced distribution of traffic.
    • If a proxy fails for a specific host, the system automatically retries with a different proxy.

  4. Monitoring and Health Checks:

    • Periodic checks on all proxies to ensure they are functional.
    • Logs detailed information about the status and response times of each proxy.

  5. Integration with Redis:

    • All proxy information is stored and managed using Redis, ensuring high performance and easy scalability.

Installation

  1. Clone the repository:
git clone https://github.com/n0a/gateway-proxy.git
cd gateway-proxy
  1. Install dependencies: Ensure you have Python installed. Then install the required packages:
pip install -r requirements.txt
  1. Run: Don't forget to set up the .env file before running the application. Alternatively, you can set environment variables directly in the docker-compose.yml file.
python server.py:
  1. Configure Redis: Make sure Redis is running on your local machine or adjust the get_redis_client function in the code to point to your Redis instance.

Environment Variables

The following environment variables can be set to configure the server:

  • FLASK_PORT: The port on which the Flask API will run. Default is 5550.
  • PROXY_PORT: The port on which the Proxy server will run. Default is 8181.
  • HOSTNAME: The hostname or IP address the server will bind to. Default is 0.0.0.0.
  • BASIC_AUTH: Basic authentication credentials in the format user:pass for gateway proxy access.
  • INITIAL_PROXIES: Comma-separated list of initial proxy URLs to add to the server on startup.

These variables can be set in a .env file or passed directly to Docker.

Example .env file

FLASK_PORT=5550         # Flask port
FLASK_USER=admin        # Flask basic auth username
FLASK_PASS=secret       # Flask basic auth password
PROXY_PORT=8181         # Proxy gateway port
HOSTNAME=0.0.0.0        # Interface for listening port
NUM_WORKERS=5           # Number of worker threads for the proxy server
BASIC_AUTH=user:pass    # Basic auth credentials for the gateway proxy server
INITIAL_PROXIES=http://user1:[email protected]:8000,http://user2:[email protected]:8000,another_n_servers

Usage

  1. Start the Proxy Server::

    docker-compose --env-file .env up -d --build

    This command will start the services in the background (detached mode).

  2. Start the Proxy Server::

    docker-compose down

    This will stop and remove all containers, networks, and volumes created by docker-compose up.

  3. Restart Policy:: The docker-compose.yml file includes a restart policy restart: unless-stopped, ensuring that the containers are automatically restarted if they fail, but they won't be restarted if stopped manually.

API Endpoints

  • Add a Proxy:

    • Endpoint: /add_proxy
    • Method: POST
    • Payload: {"proxy": "http://username:password@proxy:port"}
    • Description: Adds a new proxy to the pool. Checks its availability before adding.

  • Remove a Proxy:

    • Endpoint: /remove_proxy
    • Method: DELETE
    • Payload: {"proxy": "http://username:password@proxy:port"}
    • Description: Removes a proxy from the pool.

  • Get Proxy Info:

    • Endpoint: /proxy_info
    • Method: GET
    • Description: Retrieves detailed information about all proxies in the pool.
  1. Proxy Management:
  • The server periodically checks the availability of each proxy using predefined test URLs.
  • Proxies are marked as dead for specific hosts if they fail to connect, ensuring that traffic is not routed through non-functional proxies.

Example Code Usage

Adding a Proxy

import requests

response = requests.post('http://localhost:5550/add_proxy', json={'proxy': 'http://username:password@proxy:port'})
print(response.json())

Removing a Proxy

import requests

response = requests.delete('http://localhost:5550/remove_proxy', json={'proxy': 'http://username:password@proxy:port'})
print(response.json())

Fetching Proxy Info

import requests

response = requests.get('http://localhost:5550/proxy_info')
print(response.json())

Future Enhancements

  • Advanced Proxy Selection: Integrate machine learning algorithms to predict the best proxy based on historical data.
  • Detailed Analytics: Provide more in-depth analytics and reporting on proxy performance and usage.

Contributing

Contributions are welcome! Please submit a pull request or open an issue for any bugs or feature requests.