A journey over APIs for handling geographical data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A journey over APIs for handling geographical data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Table of contents__<br>\n",
    "[OpenStreetMap](#OpenStreetMap)<br>\n",
    "[GeoPy](#GeoPy)<br>\n",
    "[Overpass API](#Overpass-API)<br>\n",
    "[Overpass API Python Wrapper](#Overpass-API-Python-Wrapper)<br>\n",
    "[Geographical Distance](#Geographical-Distance)<br>\n",
    "[Routing](#Routing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## OpenStreetMap\n",
    "\n",
    "http://openstreetmap.org"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## GeoPy\n",
    "\n",
    "https://geopy.readthedocs.io/en/stable/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Geocoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"place_id\": \"45157935\",\n",
      "  \"licence\": \"Data \\u00a9 OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright\",\n",
      "  \"osm_type\": \"node\",\n",
      "  \"osm_id\": \"3213617576\",\n",
      "  \"boundingbox\": [\n",
      "    \"52.5141169\",\n",
      "    \"52.5142169\",\n",
      "    \"13.3963086\",\n",
      "    \"13.3964086\"\n",
      "  ],\n",
      "  \"lat\": \"52.5141669\",\n",
      "  \"lon\": \"13.3963586\",\n",
      "  \"display_name\": \"32, J\\u00e4gerstra\\u00dfe, Spandauer Vorstadt, Mitte, Berlin, 10117, Germany\",\n",
      "  \"class\": \"place\",\n",
      "  \"type\": \"house\",\n",
      "  \"importance\": 0.44025\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "from geopy.geocoders import Nominatim\n",
    "import json \n",
    "\n",
    "geolocator = Nominatim(user_agent=\"pyladies-berlin\")\n",
    "location = geolocator.geocode(u\"Jägerstraße 32, 10117 Berlin\", language='en')\n",
    "print(json.dumps(location.raw, indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "32, Jägerstraße, Spandauer Vorstadt, Mitte, Berlin, 10117, Germany\n"
     ]
    }
   ],
   "source": [
    "print(location.address)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "52.5141669\n"
     ]
    }
   ],
   "source": [
    "king_latitude = location.latitude\n",
    "print(king_latitude)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13.3963586\n"
     ]
    }
   ],
   "source": [
    "king_longitude = location.longitude\n",
    "print(king_longitude)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Overpass API\n",
    "\n",
    "Overpass API is \"a database over the web: the client sends a query to the API and gets back the data set that corresponds to the query.\"\n",
    "\n",
    "Below is the ice-cream search query you can run on the public Overpass server. \n",
    "\n",
    "Go to https://overpass-turbo.eu\n",
    "1. Search for `Berlin` (or any other place you wish)\n",
    "2. Copy the query below, paste it into the query box on the left and press `Run`. \n",
    "\n",
    "![Overpass API how-to](images/overpass-turbo.png \"Overpass API how-to\")\n",
    "\n",
    "\n",
    "\n",
    "__Query:__\n",
    "\n",
    "```\n",
    "(\n",
    "  \t// our location\n",
    "\tnode(52.5141669, 13.3963586, 52.5141669, 13.3963586);\n",
    "  \t// ice cream places in the radius of 1km\n",
    "\tnode(around:1000.0, 52.5141669, 13.3963586)[\"cuisine\"=\"ice_cream\"];\n",
    "    node(around:1000.0, 52.5141669, 13.3963586)[\"amenity\"=\"ice_cream\"];\n",
    ");\n",
    "// print results\n",
    "out body;\n",
    "\n",
    "// style the ice cream places for better visibility\n",
    "{{style:\n",
    "\tnode[cuisine=ice_cream],\n",
    "\tnode[amenity=ice_cream]{\n",
    "  \t\ticon-image: url('icons/maki/ice-cream-24@2x.png');\n",
    "  \t\ticon-width: 56;\n",
    "    }\n",
    "}}\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/icecream.png\" alt=\"Ice cream search\" width=\"512\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining some utility classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class IceCream:\n",
    "    def __init__(self, feature):\n",
    "        self.lat = feature['geometry']['coordinates'][1]\n",
    "        self.lon = feature['geometry']['coordinates'][0]\n",
    "        self.name = feature['properties'].get('name', 'N/A')\n",
    "        self.coords = (self.lat, self.lon)\n",
    "        self.geo_dist = float('nan')\n",
    "        self.road_dist = float('nan')\n",
    "        self.route = None\n",
    "        \n",
    "    def __str__(self):\n",
    "        return 'Geodesic dist: {: 4.2f} m\\tRoad dist: {: 4.2f} m\\t{}'.format(self.geo_dist, self.road_dist, self.name)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Overpass API Python Wrapper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 5 ice cream places\n",
      "{\n",
      "  \"type\": \"Feature\",\n",
      "  \"id\": 4283150546,\n",
      "  \"geometry\": {\n",
      "    \"type\": \"Point\",\n",
      "    \"coordinates\": [\n",
      "      13.3984353,\n",
      "      52.5069067\n",
      "    ]\n",
      "  },\n",
      "  \"properties\": {\n",
      "    \"amenity\": \"ice_cream\",\n",
      "    \"name\": \"Fedora Eismanufaktur\",\n",
      "    \"toilets:wheelchair\": \"no\",\n",
      "    \"wheelchair\": \"yes\"\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import overpass\n",
    "import json\n",
    "\n",
    "radius = 1000.0\n",
    "query = '( node(around:{r},{lat},{lon})[\"amenity\"=\"ice_cream\"]; ' \\\n",
    "          'node(around:{r},{lat},{lon})[\"cuisine\"=\"ice_cream\"]; ' \\\n",
    "        ')'.format(r=radius, lat=king_latitude, lon=king_longitude)\n",
    "        \n",
    "api = overpass.API()\n",
    "response = api.get(query)\n",
    "\n",
    "features = response['features']\n",
    "print('Found {} ice cream places'.format(len(features)))\n",
    "\n",
    "# Uncomment to view the full response\n",
    "print(json.dumps(features[-1], indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Geographical Distance\n",
    "\n",
    "What is the straight line distance from King to the ice cream place?\n",
    "\n",
    "Use the [geodesic distance](https://en.wikipedia.org/wiki/Geodesics_on_an_ellipsoid) which is an ellipsoidal-surface formulae\n",
    "Alternatively one can use the [great circle distance](https://en.wikipedia.org/wiki/Great-circle_distance) which is a spherical surface formulae of the geographic distance\n",
    "\n",
    "More on geographical distances [Geographical distance](https://en.wikipedia.org/wiki/Geographical_distance)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Geodesic dist:  704.39 m\tRoad dist:  nan m\tBandy Brooks\n",
      "Geodesic dist:  751.77 m\tRoad dist:  nan m\tWonderpots Frozen Yogurt\n",
      "Geodesic dist:  810.71 m\tRoad dist:  nan m\tBandy Brooks\n",
      "Geodesic dist:  820.10 m\tRoad dist:  nan m\tFedora Eismanufaktur\n",
      "Geodesic dist:  858.96 m\tRoad dist:  nan m\tkalter Krieg\n"
     ]
    }
   ],
   "source": [
    "from geopy import distance\n",
    "\n",
    "king = (king_latitude, king_longitude)\n",
    "ice_creams = []\n",
    "\n",
    "# calculate the geographical distance for each ice cream place\n",
    "for feature in features:\n",
    "    ice_cream = IceCream(feature)\n",
    "    ice_cream.geo_dist = distance.distance(king, ice_cream.coords).m\n",
    "    ice_creams.append(ice_cream)\n",
    "    \n",
    "# sort ice cream according to the geographical distance\n",
    "ice_creams_geo_dist = sorted(ice_creams, key=lambda x: x.geo_dist)\n",
    "for ice_cream in ice_creams_geo_dist:\n",
    "    print(ice_cream)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bandy Brooks (52.5197446,13.3886757)\n",
      "kalter Krieg (52.5075334,13.3898891)\n",
      "Bandy Brooks (52.519203,13.402645)\n",
      "Wonderpots Frozen Yogurt (52.5200489,13.3909113)\n",
      "Fedora Eismanufaktur (52.5069067,13.3984353)\n"
     ]
    }
   ],
   "source": [
    "for feature in features:\n",
    "    ice_lat = feature['geometry']['coordinates'][1]\n",
    "    ice_lon = feature['geometry']['coordinates'][0]\n",
    "    ice_name = feature['properties'].get('name', 'N/A')\n",
    "    print('{} ({},{})'.format(ice_name, ice_lat, ice_lon))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Routing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(52.5141669, 13.3963586)\n"
     ]
    }
   ],
   "source": [
    "print(king)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pyroutelib3 import Router # Import the router\n",
    "router = Router('foot') # Initialise it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start node id in osm is  29207837\n"
     ]
    }
   ],
   "source": [
    "start = router.data.findNode(king_latitude, king_longitude) # Find start and end nodes\n",
    "print('Start node id in osm is ', start)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Geodesic dist:  704.39 m\tRoad dist:  1026.64 m\tBandy Brooks\n",
      "Geodesic dist:  751.77 m\tRoad dist:  1057.29 m\tWonderpots Frozen Yogurt\n",
      "Geodesic dist:  820.10 m\tRoad dist:  1079.49 m\tFedora Eismanufaktur\n",
      "Geodesic dist:  858.96 m\tRoad dist:  1168.23 m\tkalter Krieg\n",
      "Geodesic dist:  810.71 m\tRoad dist:  1169.20 m\tBandy Brooks\n"
     ]
    }
   ],
   "source": [
    "def route_length(route):\n",
    "    d = 0\n",
    "    for i in range(len(route)-1):\n",
    "        d += router.distance(route[i], route[i+1])\n",
    "    # distance is in km, multiplying by 1000 to get meters\n",
    "    return d*1000\n",
    "\n",
    "for ice_cream in ice_creams:\n",
    "    end = router.data.findNode(ice_cream.lat, ice_cream.lon)\n",
    "    status, route = router.doRoute(start, end) # Find the route - a list of OSM nodes\n",
    "\n",
    "    if status == 'success':\n",
    "        ice_cream.road_dist = route_length(route)\n",
    "        routeLatLons = list(map(router.nodeLatLon, route)) # Get actual route coordinates\n",
    "        ice_cream.route = routeLatLons\n",
    "\n",
    "# order ice_creams according to the road distance\n",
    "ice_creams_road_dist = sorted(ice_creams, key=lambda x: x.road_dist)\n",
    "for ice_cream in ice_creams_road_dist:\n",
    "    print(ice_cream)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[29207837, 25663722, 2435495987, 29207418, 2435495986]\n",
      "[[52.5143491, 13.3969553], [52.515233, 13.3967476], [52.51517, 13.395714], [52.5151553, 13.3955002], [52.5151448, 13.3953108]]\n"
     ]
    }
   ],
   "source": [
    "from pyroutelib3 import Router \n",
    "\n",
    "lat = 52.5141669\n",
    "lon = 13.3963586\n",
    "router = Router('foot') \n",
    "\n",
    "# Starting point\n",
    "start = router.data.findNode(lat, lon) \n",
    "\n",
    "# Ending point\n",
    "ice_cream = ice_creams[0]\n",
    "end = router.data.findNode(ice_cream.lat, ice_cream.lon)\n",
    "\n",
    "status, route = router.doRoute(start, end) \n",
    "\n",
    "if status == 'success':\n",
    "    # print 5 first nodes in the route\n",
    "    print(route[:5])\n",
    "    # remap nodes to coordinates\n",
    "    routeLatLons = list(map(router.nodeLatLon, route)) \n",
    "    print(routeLatLons[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Look at the result nb 3, Bandy Brooks. Although it is the 3rd in terms of the geographical distance, it is the furthest in terms of the road distance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "var end=[13.402645, 52.519203];\n",
      "var start=[13.3963586, 52.5141669];\n",
      "var path=[[52.5143491, 13.3969553], [52.5144598, 13.3973412], [52.5145683, 13.3977015], [52.5147075, 13.3981526], [52.5148244, 13.3982565], [52.5152812, 13.3979998], [52.5153282, 13.3979735], [52.515411, 13.3979269], [52.5154356, 13.398027], [52.5154572, 13.3981267], [52.5154764, 13.3982154], [52.5155891, 13.3987352], [52.5157998, 13.3995934], [52.515815, 13.3996587], [52.5158835, 13.39993], [52.515901, 13.3999959], [52.5161622, 13.4006294], [52.5161877, 13.4007192], [52.5161916, 13.400797], [52.5161715, 13.4009083], [52.5159727, 13.4011562], [52.5159587, 13.4012545], [52.5159711, 13.4013603], [52.5163583, 13.402266], [52.5164427, 13.4024634], [52.5167195, 13.4031448], [52.5169649, 13.4037369], [52.5169879, 13.4037966], [52.5171978, 13.4043066], [52.5172145, 13.4043458], [52.5172524, 13.4044333], [52.5174453, 13.4043692], [52.5175835, 13.4043357], [52.5179444, 13.4042224], [52.5180278, 13.4041776], [52.5180906, 13.4041367], [52.5181973, 13.4040257], [52.5183099, 13.4039049], [52.5183967, 13.4037753], [52.5183109, 13.4036129], [52.518133, 13.4033205], [52.5185022, 13.4029662], [52.518568, 13.4029075], [52.5185948, 13.4028833], [52.518584, 13.402854], [52.5185699, 13.4028106], [52.5185908, 13.4027871], [52.5186998, 13.4026722], [52.5187138, 13.4026679], [52.5187275, 13.4026656], [52.5187718, 13.402661], [52.5188317, 13.4026431], [52.5189124, 13.4025547], [52.5189558, 13.4024912], [52.5189746, 13.4024366], [52.5189796, 13.4024086], [52.5189874, 13.4023865], [52.5190653, 13.402326], [52.5190935, 13.4023026], [52.5191068, 13.4023477], [52.5191309, 13.4024113], [52.5191447, 13.4024379], [52.5191181, 13.4024663], [52.5190652, 13.4025249]];\n"
     ]
    }
   ],
   "source": [
    "# Copy that to HTML file in this folder to see the map\n",
    "print('var end=[{}, {}];'.format(ice_creams_road_dist[0].lon, ice_creams_road_dist[0].lat))\n",
    "print('var start=[{}, {}];'.format(king_longitude, king_latitude))\n",
    "print('var path={};'.format(ice_creams_road_dist[0].route))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}