Advertools/Advertools_Analyze_website_content_using_XML_sitemap.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5da81fdb-1607-4dcd-8945-78492d41c20c",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "<img width=\"8%\" alt=\"Advertools.png\" src=\"https://raw.githubusercontent.com/jupyter-naas/awesome-notebooks/master/.github/assets/logos/Advertools.png\" style=\"border-radius: 15%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96049d01-11e9-4538-9012-57992898281e",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "# Advertools - Analyze website content using XML sitemap\n",
    "<a href=\"https://bit.ly/3JyWIk6\">Give Feedback</a> | <a href=\"https://github.com/jupyter-naas/awesome-notebooks/issues/new?assignees=&labels=bug&template=bug_report.md&title=Advertools+-+Analyze+website+content+using+XML+sitemap:+Error+short+description\">Bug report</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76b0bd6e-4a5c-472f-a473-a315e3738936",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Tags:** #advertools #xml #sitemap #website #analyze #seo"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1089c7e4-b7a9-48ab-8320-e17847ad3b30",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Author:** [Elias Dabbas](https://www.linkedin.com/in/eliasdabbas/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "747ff26a-034d-4047-b573-e15ceabbf62d",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Last update:** 2023-05-23 (Created: 2023-05-09)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d93cd7a5-1d4f-4bc6-94a2-b74461c7f538",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Description:** This notebook helps you get an overview of a website's content by analyzing and visualizing its XML sitemap. It's also an important SEO audit process that can uncover some potential issues that might affect the website."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d84a0cf-6ba8-456a-8b42-1bf63e3a17f2",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**References:**\n",
    "- [advertools Sitemaps](https://advertools.readthedocs.io/en/master/advertools.sitemaps.html)\n",
    "- [XML Sitemap](https://www.xml-sitemaps.com/)\n",
    "- [Sitemaps Protocol](https://www.sitemaps.org/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a75ef3a2-b261-45a8-bd64-52cb78b5f502",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Input"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "984b2abc-b888-45fe-a48c-a868c21cf3e9",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Install libraries\n",
    "If running it on naas, run the code below to install the libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1fc5466-e7ac-4dad-80e0-98db9735024e",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "#!pip install advertools adviz pandas==1.5.3 --user"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f530c66d-1a5b-47d1-8d80-01f21a83ac3e",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e8d108a-44de-44b4-941f-f12795838c5c",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "import advertools as adv\n",
    "import adviz\n",
    "from urllib.parse import urlsplit"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d35a9e6f-71c4-4cb2-b9bc-32c23b18c4ba",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Setup Variables\n",
    "- `sitemap_url`: URL of the sitemap to analyze, which can be\n",
    "    * The URL of an XML sitemap\n",
    "    * The URL of an XML sitemapindex\n",
    "    * The URL of a robots.txt file\n",
    "    * Normal and zipped formats are supported\n",
    "- `recursive`: If this is a sitemapindex, should all the sub-sitemaps also be  downloaded, parsed and combined into one DataFrame?\n",
    "- `max_workers`: Number of concurrent workers to fetch the sitemaps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cebcf1a5-23ce-49ca-aee9-aac7ff28a2e4",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "sitemap_url = \"https://blog.sriniketh.design/sitemap.xml\"\n",
    "recursive = True\n",
    "max_workers = 8"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ad6e918-ae61-4a78-9b71-a6ad5f530f53",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dafe9992-9fcb-4849-a69e-ced13c995aff",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Analyze website content using XML sitemap\n",
    "Getting the sitemap(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4949b9c4-f852-47f5-9889-3ccce1a0100a",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "sitemap = adv.sitemap_to_df(\n",
    "    sitemap_url=sitemap_url,\n",
    "    max_workers=max_workers,\n",
    "    recursive=recursive\n",
    ")\n",
    "sitemap"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b907e816",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "Split URLs into their components for further analysis/understanding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e43820cb",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "urldf = adv.url_to_df(sitemap['loc'])\n",
    "urldf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec5491f8-3773-40e3-9e71-3058d4b7b870",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd817594-1b7f-4d02-a245-9e1dcd6ad2b6",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Display results\n",
    "\n",
    "#### Errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc89d554-375c-4a10-a028-371d5b30781b",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "if 'errors' in sitemap:\n",
    "    from IPython.display import display\n",
    "    display(sitemap[sitemap['errors'].notnull()])\n",
    "else:\n",
    "    print('No errors found')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b34af8e",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "#### Duplicated URLs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ecab9fe-071a-47d8-a172-ae20ad3f4224",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "duplicated = sitemap[sitemap['loc'].duplicated()]\n",
    "if not duplicated.empty:\n",
    "    display(duplicated)\n",
    "else:\n",
    "    print('No duplicated URLs found')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26ec0200",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "#### URL counts per sitemap and sitemap sizes\n",
    "\n",
    "Each sitemap should have a maximumof 50,000 URLs, and its size should not exceek 50MB\n",
    "\n",
    "URL counts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff6e436c",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "adviz.value_counts_plus(sitemap['sitemap'], name='Sitemap URLs')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f80fbcc",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "URL Sizes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0716b1c8",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "sitemap['sitemap_size_mb'].describe().to_frame().T.style.format('{:,.2f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0b356e7",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "#### Count unique values of URL components"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "469a91d7",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "for col in ['scheme', 'netloc', 'dir_1', 'dir_2', 'dir_3']:\n",
    "    try:\n",
    "        display(adviz.value_counts_plus(urldf[col], name=col))\n",
    "    except Exception as e:\n",
    "        continue"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07a1255a",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "#### Visualize the structure of the URLs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c44e36e",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "domain = urlsplit(sitemap_url).netloc\n",
    "try:\n",
    "    adviz.url_structure(\n",
    "        urldf['url'].fillna(''),\n",
    "        items_per_level=30,\n",
    "        domain=domain,\n",
    "        height=750,\n",
    "        title=f'URL Structure: {domain} XML sitemap'\n",
    "    )\n",
    "except Exception as e:\n",
    "    print(str(e))\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "376c5fe7-1c06-4c31-a61c-1e684dabff23",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "naas": {
   "notebook_id": "1fc79d1caf15ef3a1adf0874fa85f6a8b05e172f5e45f0fc6ce7ee40e5c54cae",
   "notebook_path": "Advertools/Advertools_Analyze_website_content_using_XML_sitemap.ipynb"
  },
  "papermill": {
   "default_parameters": {},
   "environment_variables": {},
   "parameters": {},
   "version": "2.4.0"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}