BeautifulSoup/BeautifulSoup_Scrape_emails_from_URL.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0754f440-b4a3-4958-9919-e30e9302adde",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "<img width=\"8%\" alt=\"BeautifulSoup.png\" src=\"https://raw.githubusercontent.com/jupyter-naas/awesome-notebooks/master/.github/assets/logos/BeautifulSoup.png\" style=\"border-radius: 15%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3edeaf6-cf15-4876-8903-61232e581781",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "# BeautifulSoup - Scrape emails from URL\n",
    "<a href=\"https://bit.ly/3JyWIk6\">Give Feedback</a> | <a href=\"https://github.com/jupyter-naas/awesome-notebooks/issues/new?assignees=&labels=bug&template=bug_report.md&title=BeautifulSoup+-+Scrape+emails+from+URL:+Error+short+description\">Bug report</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "165000f2-9bc9-430b-80aa-666fc178529d",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Tags:** #beautifulsoup #python #scraping #emails #url #webscraping #html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11467202-eecd-47c6-b353-afd4a4c474cd",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Author:** [Florent Ravenel](https://www.linkedin.com/in/florent-ravenel/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6adfa6e0-0ca9-46ee-8258-8c3b24fcb16e",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Last update:** 2023-04-12 (Created: 2023-02-16)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5bdadc0-3346-4222-a74d-9a9237a2075f",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Description:** This notebook will show how to scrape emails stored in HTML webpage using BeautifulSoup."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14ecfcca-6d08-45fe-9806-a71fd35b5269",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "<u>References:</u>\n",
    "- [Beautiful Soup Documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)\n",
    "- [Regular Expression Documentation](https://docs.python.org/3/library/re.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9da6f607-cac6-436d-b59e-00aa4370b25f",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Input"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5b2693c-e417-4a03-ae5c-3c777de5c7db",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22494cd6-648e-4107-88ef-f446282e9c50",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "import re\n",
    "import requests\n",
    "from urllib.parse import urlsplit\n",
    "from collections import deque\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2128459d-e6e6-4c46-8e52-0c5bbeeb68c3",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Setup Variables\n",
    "- `url`: URL of the webpage to scrape\n",
    "- `limit`: number of emails found to stop scraping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc07a613-0018-403e-a1ca-0678f4e274c5",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "url = \"https://www.naas.ai/\"\n",
    "limit = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b779ddd5-5914-4768-8f94-bce081793b95",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68493b82-d57b-4509-a71c-45c198da4d77",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Scrape emails from URL"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8893303c-85d5-4784-bc07-cf72ac4d655d",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "We will use the `requests` library to get the HTML content of the webpage and the `BeautifulSoup` library to parse the HTML content. We will use a regular expression to extract the emails from the HTML content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15f1e143-fbb8-4fac-98f0-afc3ad659343",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "unscraped = deque([url])  \n",
    "\n",
    "scraped = set()  \n",
    "\n",
    "emails = set()  \n",
    "\n",
    "while len(unscraped):\n",
    "    url = unscraped.popleft()  \n",
    "    scraped.add(url)\n",
    "\n",
    "    parts = urlsplit(url)\n",
    "        \n",
    "    base_url = \"{0.scheme}://{0.netloc}\".format(parts)\n",
    "    if '/' in parts.path:\n",
    "        path = url[:url.rfind('/')+1]\n",
    "    else:\n",
    "        path = url\n",
    "\n",
    "    print(\"Crawling URL: %s\" % url)\n",
    "    try:\n",
    "        response = requests.get(url)\n",
    "    except (requests.exceptions.MissingSchema, requests.exceptions.ConnectionError):\n",
    "        continue\n",
    "        \n",
    "    exclude = [\"google.com\", \"gmail.com\", \"example.com\"]    \n",
    "    # Get emails from URL\n",
    "    new_emails = re.findall(r\"[a-z0-9\\.\\-+_]+@[a-z0-9\\.\\-+_]+\\.+[a-z]{1,3}\", url)\n",
    "    for email in new_emails:\n",
    "        for e in exclude:\n",
    "            if not email.endswith(e):\n",
    "                emails.update([email])\n",
    "                \n",
    "    # Get emails from content\n",
    "    new_emails = set(re.findall(r\"[a-z0-9\\.\\-+_]+@[a-z0-9\\.\\-+_]+\\.+[a-z]{1,3}\", response.text, re.I))\n",
    "    for email in new_emails:\n",
    "        for e in exclude:\n",
    "            if not email.endswith(e):\n",
    "                emails.update([email])\n",
    "                \n",
    "    if len(emails) >= limit:\n",
    "        break\n",
    "\n",
    "    soup = BeautifulSoup(response.text, 'lxml')\n",
    "    for anchor in soup.find_all(\"a\"):\n",
    "        if \"href\" in anchor.attrs:\n",
    "            link = anchor.attrs[\"href\"]\n",
    "        else:\n",
    "            link = ''\n",
    "\n",
    "        if link.startswith('/'):\n",
    "            link = base_url + link\n",
    "        \n",
    "        elif not link.startswith('http'):\n",
    "            link = path + link\n",
    "\n",
    "        if not link.endswith(\".gz\"):\n",
    "            if not link in unscraped and not link in scraped:\n",
    "                unscraped.append(link)\n",
    "\n",
    "print(emails)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37508c40-89b9-4c0f-b05f-aba4d0a041f5",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c37e870-430c-4f33-8980-8da03df22cdf",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Display result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a788c4a5-1559-49ce-8b97-589dc4ef1b58",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(f\"🚀 {len(emails)} founded on {url}\")\n",
    "print(emails)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ec95b4b-a905-4a9c-9ab9-e0824c582302",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    " "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "naas": {
   "notebook_id": "12ef03c9788da13b430b0d23171c6c0949ff23c86a70281d36e19d8a6237b135",
   "notebook_path": "BeautifulSoup/BeautifulSoup_Scrape_emails_from_URL.ipynb"
  },
  "papermill": {
   "default_parameters": {},
   "environment_variables": {},
   "parameters": {},
   "version": "2.4.0"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}