4-data_management/Topic2-Online_Data_Access.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Website-Data\" data-toc-modified-id=\"Website-Data-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Website Data</a></span><ul class=\"toc-item\"><li><span><a href=\"#Requests-Package\" data-toc-modified-id=\"Requests-Package-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Requests Package</a></span></li><li><span><a href=\"#JSON-files\" data-toc-modified-id=\"JSON-files-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>JSON files</a></span></li><li><span><a href=\"#Discussion:-What-could-go-wrong-if-this-approach-were-applied-to-a-new-compound?\" data-toc-modified-id=\"Discussion:-What-could-go-wrong-if-this-approach-were-applied-to-a-new-compound?-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>Discussion: What could go wrong if this approach were applied to a new compound?</a></span></li><li><span><a href=\"#Exercise:-Write-a-function-that-counts-the-number-of-C-H-bonds-in-ethanol\" data-toc-modified-id=\"Exercise:-Write-a-function-that-counts-the-number-of-C-H-bonds-in-ethanol-1.4\"><span class=\"toc-item-num\">1.4&nbsp;&nbsp;</span>Exercise: Write a function that counts the number of C-H bonds in ethanol</a></span></li></ul></li><li><span><a href=\"#Application-Programming-Interfaces-(APIs)\" data-toc-modified-id=\"Application-Programming-Interfaces-(APIs)-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Application Programming Interfaces (APIs)</a></span><ul class=\"toc-item\"><li><span><a href=\"#RESTful-API's\" data-toc-modified-id=\"RESTful-API's-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>RESTful API's</a></span></li><li><span><a href=\"#Exercise:-Write-a-function-that-returns-the-CID-given-a-compound-name\" data-toc-modified-id=\"Exercise:-Write-a-function-that-returns-the-CID-given-a-compound-name-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>Exercise: Write a function that returns the CID given a compound name</a></span></li><li><span><a href=\"#Exercise:-Write-a-function-that-returns-the-SMILES-string-for-any-compound-based-on-CAS-number\" data-toc-modified-id=\"Exercise:-Write-a-function-that-returns-the-SMILES-string-for-any-compound-based-on-CAS-number-2.3\"><span class=\"toc-item-num\">2.3&nbsp;&nbsp;</span>Exercise: Write a function that returns the SMILES string for any compound based on CAS number</a></span></li><li><span><a href=\"#Python-API's\" data-toc-modified-id=\"Python-API's-2.4\"><span class=\"toc-item-num\">2.4&nbsp;&nbsp;</span>Python API's</a></span></li><li><span><a href=\"#Exercise:-Write-a-function-that-takes-an-arbitrary-chemical-name-or-CAS-number-and-returns-the-number-of-C-H-bonds.\" data-toc-modified-id=\"Exercise:-Write-a-function-that-takes-an-arbitrary-chemical-name-or-CAS-number-and-returns-the-number-of-C-H-bonds.-2.5\"><span class=\"toc-item-num\">2.5&nbsp;&nbsp;</span>Exercise: Write a function that takes an arbitrary chemical name or CAS number and returns the number of C-H bonds.</a></span></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Online Data Access\n",
    "\n",
    "A large amount of data is only available through the internet. There are many ways to access the data, but some are more convenient than others. In this lecture we will work with the PubChem website and database.\n",
    "\n",
    "## Website Data\n",
    "\n",
    "### Requests Package\n",
    "\n",
    "Let's start by looking at the PubChem page for ethanol: https://pubchem.ncbi.nlm.nih.gov/compound/Ethanol\n",
    "\n",
    "The main Python package for accessing online data is `requests`, which essentially makes HTTP requests for data. We can \"request\" data from this URL:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "page = requests.get('https://pubchem.ncbi.nlm.nih.gov/compound/Ethanol')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#page.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the raw text that describes the website, in this case it is HTML. It is possible to extract data directly from HTML, but it is challenging and tedious. Packages such as [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) can make the process much easier, but we won't cover it in this course."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JSON files\n",
    "\n",
    "An alternate approach is to use a more structured representation. Note that there is a \"Download\" button in the top right of the page. If you select \"JSON\" under \"Data used to display this page\" the result will be the `ethanol.json` file. We can load this with the Python `json` package:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "with open('data/ethanol.json') as f:\n",
    "    etoh = json.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The JSON file acts like a Python dictionary, and can contain other dictionaries/lists within it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#etoh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is still pretty messy, but it's a little more organized. Working with JSON data can be challenging if there are many nested structures, headers, etc. It is very useful to use a visualization tool:\n",
    "\n",
    "* [Code Beautify](https://codebeautify.org/jsonviewer)\n",
    "* [Chrome Extension](https://chrome.google.com/webstore/detail/json-viewer/gbmdgpbipfallnflgajpaliibnhdgobh?hl=en-US)\n",
    "\n",
    "From the visualizer we can see how to extract the information we need. Note the \"search\" feature!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try to extract some information:\n",
    "\n",
    "* SMILES string\n",
    "* Molecular weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SMILES: CCO\n",
      "Molecular Weight: 46.07\n"
     ]
    }
   ],
   "source": [
    "SMILES = etoh['Record']['Section'][2]['Section'][1]['Section'][3]['Information'][0]['Value']['StringWithMarkup'][0]['String']#['StringValue']\n",
    "MW = etoh['Record']['Section'][3]['Section'][0]['Section'][0]['Information'][0]['Value']['Number'][0]\n",
    "print('SMILES: {}'.format(SMILES))\n",
    "print('Molecular Weight: {}'.format(MW))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion: What could go wrong if this approach were applied to a new compound?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is somewhat structured, but it is still very time consuming to find spedific data. This is mainly due to the complex structure of the JSON file. If there were fewer nested loops it would have been a lot easier to find specific info. For example, consider the simpler JSON file that also contains data on ethanol (we will see how to get this later):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'PC_Compounds': [{'id': {'id': {'cid': 702}}, 'atoms': {'aid': [1, 2, 3, 4, 5, 6, 7, 8, 9], 'element': [8, 6, 6, 1, 1, 1, 1, 1, 1]}, 'bonds': {'aid1': [1, 1, 2, 2, 2, 3, 3, 3], 'aid2': [2, 9, 3, 4, 5, 6, 7, 8], 'order': [1, 1, 1, 1, 1, 1, 1, 1]}, 'coords': [{'type': [1, 5, 255], 'aid': [1, 2, 3, 4, 5, 6, 7, 8, 9], 'conformers': [{'x': [3.732, 2.866, 2, 2.4675, 3.2646, 2.31, 1.4631, 1.69, 4.269], 'y': [0.25, -0.25, 0.25, -0.7249, -0.7249, 0.7869, 0.56, -0.2869, -0.06]}]}], 'charge': 0, 'props': [{'urn': {'label': 'Compound', 'name': 'Canonicalized', 'datatype': 5, 'release': '2019.01.04'}, 'value': {'ival': 1}}, {'urn': {'label': 'Compound Complexity', 'datatype': 7, 'implementation': 'E_COMPLEXITY', 'version': '3.4.6.11', 'software': 'Cactvs', 'source': 'xemistry.com', 'release': '2019.06.18'}, 'value': {'fval': 2.8}}, {'urn': {'label': 'Count', 'name': 'Hydrogen Bond Acceptor', 'datatype': 5, 'implementation': 'E_NHACCEPTORS', 'version': '3.4.6.11', 'software': 'Cactvs', 'source': 'xemistry.com', 'release': '2019.06.18'}, 'value': {'ival': 1}}, {'urn': {'label': 'Count', 'name': 'Hydrogen Bond Donor', 'datatype': 5, 'implementation': 'E_NHDONORS', 'version': '3.4.6.11', 'software': 'Cactvs', 'source': 'xemistry.com', 'release': '2019.06.18'}, 'value': {'ival': 1}}, {'urn': {'label': 'Count', 'name': 'Rotatable Bond', 'datatype': 5, 'implementation': 'E_NROTBONDS', 'version': '3.4.6.11', 'software': 'Cactvs', 'source': 'xemistry.com', 'release': '2019.06.18'}, 'value': {'ival': 0}}, {'urn': {'label': 'Fingerprint', 'name': 'SubStructure Keys', 'datatype': 16, 'parameters': 'extended 2', 'implementation': 'E_SCREEN', 'version': '3.4.6.11', 'software': 'Cactvs', 'source': 'xemistry.com', 'release': '2019.06.18'}, 'value': {'binary': '0000037180402000000000000000000000000000000000000000000000000000000000000000001A00000800000000A080020200000002000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000'}}, {'urn': {'label': 'IUPAC Name', 'name': 'Allowed', 'datatype': 1, 'version': '2.6.6', 'software': 'LexiChem', 'source': 'openeye.com', 'release': '2019.06.18'}, 'value': {'sval': 'ethanol'}}, {'urn': {'label': 'IUPAC Name', 'name': 'CAS-like Style', 'datatype': 1, 'version': '2.6.6', 'software': 'LexiChem', 'source': 'openeye.com', 'release': '2019.06.18'}, 'value': {'sval': 'ethanol'}}, {'urn': {'label': 'IUPAC Name', 'name': 'Markup', 'datatype': 1, 'version': '2.6.6', 'software': 'LexiChem', 'source': 'openeye.com', 'release': '2019.06.18'}, 'value': {'sval': 'ethanol'}}, {'urn': {'label': 'IUPAC Name', 'name': 'Preferred', 'datatype': 1, 'version': '2.6.6', 'software': 'LexiChem', 'source': 'openeye.com', 'release': '2019.06.18'}, 'value': {'sval': 'ethanol'}}, {'urn': {'label': 'IUPAC Name', 'name': 'Systematic', 'datatype': 1, 'version': '2.6.6', 'software': 'LexiChem', 'source': 'openeye.com', 'release': '2019.06.18'}, 'value': {'sval': 'ethanol'}}, {'urn': {'label': 'IUPAC Name', 'name': 'Traditional', 'datatype': 1, 'version': '2.6.6', 'software': 'LexiChem', 'source': 'openeye.com', 'release': '2019.06.18'}, 'value': {'sval': 'ethanol'}}, {'urn': {'label': 'InChI', 'name': 'Standard', 'datatype': 1, 'version': '1.0.5', 'software': 'InChI', 'source': 'iupac.org', 'release': '2019.06.18'}, 'value': {'sval': 'InChI=1S/C2H6O/c1-2-3/h3H,2H2,1H3'}}, {'urn': {'label': 'InChIKey', 'name': 'Standard', 'datatype': 1, 'version': '1.0.5', 'software': 'InChI', 'source': 'iupac.org', 'release': '2019.06.18'}, 'value': {'sval': 'LFQSCWFLJHTTHZ-UHFFFAOYSA-N'}}, {'urn': {'label': 'Log P', 'name': 'XLogP3-AA', 'datatype': 7, 'version': '3.0', 'source': 'sioc-ccbg.ac.cn', 'release': '2019.06.18'}, 'value': {'fval': -0.1}}, {'urn': {'label': 'Mass', 'name': 'Exact', 'datatype': 7, 'version': '2.1', 'software': 'PubChem', 'source': 'ncbi.nlm.nih.gov', 'release': '2019.06.18'}, 'value': {'fval': 46.041864813}}, {'urn': {'label': 'Molecular Formula', 'datatype': 1, 'version': '2.1', 'software': 'PubChem', 'source': 'ncbi.nlm.nih.gov', 'release': '2019.06.18'}, 'value': {'sval': 'C2H6O'}}, {'urn': {'label': 'Molecular Weight', 'datatype': 7, 'version': '2.1', 'software': 'PubChem', 'source': 'ncbi.nlm.nih.gov', 'release': '2019.06.18'}, 'value': {'fval': 46.07}}, {'urn': {'label': 'SMILES', 'name': 'Canonical', 'datatype': 1, 'version': '2.1.5', 'software': 'OEChem', 'source': 'openeye.com', 'release': '2019.06.18'}, 'value': {'sval': 'CCO'}}, {'urn': {'label': 'SMILES', 'name': 'Isomeric', 'datatype': 1, 'version': '2.1.5', 'software': 'OEChem', 'source': 'openeye.com', 'release': '2019.06.18'}, 'value': {'sval': 'CCO'}}, {'urn': {'label': 'Topological', 'name': 'Polar Surface Area', 'datatype': 7, 'implementation': 'E_TPSA', 'version': '3.4.6.11', 'software': 'Cactvs', 'source': 'xemistry.com', 'release': '2019.06.18'}, 'value': {'fval': 20.2}}, {'urn': {'label': 'Weight', 'name': 'MonoIsotopic', 'datatype': 7, 'version': '2.1', 'software': 'PubChem', 'source': 'ncbi.nlm.nih.gov', 'release': '2019.06.18'}, 'value': {'fval': 46.041864813}}], 'count': {'heavy_atom': 3, 'atom_chiral': 0, 'atom_chiral_def': 0, 'atom_chiral_undef': 0, 'bond_chiral': 0, 'bond_chiral_def': 0, 'bond_chiral_undef': 0, 'isotope_atom': 0, 'covalent_unit': 1, 'tautomers': -1}}]}\n"
     ]
    }
   ],
   "source": [
    "with open('data/ethanol_simple.json') as f:\n",
    "    etoh_simple = json.load(f)\n",
    "    \n",
    "print(etoh_simple)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can extract the same information with significantly less effort:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SMILES: CCO\n",
      "Molecular Weight: 46.07\n"
     ]
    }
   ],
   "source": [
    "SMILES = etoh_simple['PC_Compounds'][0]['props'][18]['value']['sval']\n",
    "MW = etoh_simple['PC_Compounds'][0]['props'][17]['value']['fval']\n",
    "print('SMILES: {}'.format(SMILES))\n",
    "print('Molecular Weight: {}'.format(MW))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Write a function that counts the number of C-H bonds in ethanol\n",
    "\n",
    "Use the `ethanol_simple.json` file as input. You will need both `bonds` and `atoms` information. Note that `element` refers to the atomic number (e.g. hydrogen is `1`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'aid1': [1, 1, 2, 2, 2, 3, 3, 3], 'aid2': [2, 9, 3, 4, 5, 6, 7, 8], 'order': [1, 1, 1, 1, 1, 1, 1, 1]}\n",
      "{'aid': [1, 2, 3, 4, 5, 6, 7, 8, 9], 'element': [8, 6, 6, 1, 1, 1, 1, 1, 1]}\n"
     ]
    }
   ],
   "source": [
    "print(etoh_simple['PC_Compounds'][0]['bonds'])\n",
    "print(etoh_simple['PC_Compounds'][0]['atoms'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The number of C-H bonds will be the number of bonds between (2 or 3) and (4, 5, 6, 7, 8, or 9)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Application Programming Interfaces (APIs)\n",
    "\n",
    "API's are like GUI's for experts. They are not limited to online data, or even data in general. API is a term for any programmatic structure that makes it easier to interact with a more complex underlying code or data structure. However, they are particularly prevalent in data science because accessing data is much less painful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RESTful API's\n",
    "\n",
    "REST stands for \"representational state transfer\", and is a protocol that enables accessing data directly through a URL. This is a very common and very powerful approach because it allows the data provider to abstract the database back-end from the API. In other words, data providers can provide a uniform interface to data in relational (schema-driven) databases, schema-free databases, file servers, or services in any programming language. All the user needs to know is how to \"query\" from a URL. If you pay attention to URL's as you browse the web you will see that you use RESTful API's all the time without knowing it!\n",
    "\n",
    "<center>\n",
    "<img src=\"images/RESTful.png\" width=\"500\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RESTful API's are designed to return data in specific structures, and respond to specific queries that are embedded in the URL. A few notes:\n",
    "\n",
    "* Many API's require a \"key\" or \"token\". This is to avoid spammers overloading their servers.\n",
    "* Most API's also limit the amount of data per request, and the rate of requests.\n",
    "* It is still necessary to understand the underlying structure of the data you are querying.\n",
    "\n",
    "You should always start by reading the documentation of an API to learn what you can/can't do.\n",
    "\n",
    "In this lecture we will work with the PubChem API:\n",
    "\n",
    "[PubChem API tutorial documentation](http://pubchemdocs.ncbi.nlm.nih.gov/pug-rest-tutorial$_Toc458584421)\n",
    "\n",
    "[PubChem API full documentation](http://pubchemdocs.ncbi.nlm.nih.gov/pug-rest)\n",
    "\n",
    "The nice thing about RESTful API's is that they can be accessed directly through HTTP requests. Let's try to find the compound identifier (CID) for ethanol.\n",
    "\n",
    "First, we need to understand the structure of the query to decide how to search. From the documentation:\n",
    "\n",
    "* prolog: `https://pubchem.ncbi.nlm.nih.gov/rest/pug`\n",
    "\n",
    "* input: `/compound/name/ethanol`\n",
    "\n",
    "* operation: `/cids`\n",
    "\n",
    "* output: `/TXT`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = requests.get('https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/name/ethanol/cids/TXT')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "702\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"name\" search is pretty flexible, and we can even search by CAS number. For example, the CAS number for ethanol is 64-17-5:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "702\n",
      "\n"
     ]
    }
   ],
   "source": [
    "r = requests.get('https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/name/64-17-5/cids/TXT')\n",
    "print(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that if we ask for something that isn't there we get a 404 error that gives some insight into what went wrong:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Status: 404\n",
      "Code: PUGREST.NotFound\n",
      "Message: No CID found\n",
      "Detail: No CID found that matches the given name\n",
      "\n"
     ]
    }
   ],
   "source": [
    "r = requests.get('https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/name/whiskey/cids/TXT')\n",
    "print(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Write a function that returns the CID given a compound name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "702\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def returnCID(compound):\n",
    "    r = requests.get('https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/name/{}/cids/TXT'.format(compound))\n",
    "    return r.text\n",
    "\n",
    "print(returnCID('ethanol'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is much easier, less memory intensive, and more robust, than trying to extract the property from the JSON of the webpage (or the HTML).\n",
    "\n",
    "An alternative intermediate strategy is to pull the full record using the API, then work with the resulting JSON for a single compound. The following function was written using the documentation for the PubChem RESTful interface:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_full(chemical):\n",
    "    r = requests.get('https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/name/{}/record/json'.format(chemical))\n",
    "    chem_json = json.loads(r.text)\n",
    "    return chem_json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use it to get information for ethanol:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'PC_Compounds': [{'id': {'id': {'cid': 702}}, 'atoms': {'aid': [1, 2, 3, 4, 5, 6, 7, 8, 9], 'element': [8, 6, 6, 1, 1, 1, 1, 1, 1]}, 'bonds': {'aid1': [1, 1, 2, 2, 2, 3, 3, 3], 'aid2': [2, 9, 3, 4, 5, 6, 7, 8], 'order': [1, 1, 1, 1, 1, 1, 1, 1]}, 'coords': [{'type': [1, 5, 255], 'aid': [1, 2, 3, 4, 5, 6, 7, 8, 9], 'conformers': [{'x': [3.732, 2.866, 2, 2.4675, 3.2646, 2.31, 1.4631, 1.69, 4.269], 'y': [0.25, -0.25, 0.25, -0.7249, -0.7249, 0.7869, 0.56, -0.2869, -0.06]}]}], 'charge': 0, 'props': [{'urn': {'label': 'Compound', 'name': 'Canonicalized', 'datatype': 5, 'release': '2019.01.04'}, 'value': {'ival': 1}}, {'urn': {'label': 'Compound Complexity', 'datatype': 7, 'implementation': 'E_COMPLEXITY', 'version': '3.4.6.11', 'software': 'Cactvs', 'source': 'xemistry.com', 'release': '2019.06.18'}, 'value': {'fval': 2.8}}, {'urn': {'label': 'Count', 'name': 'Hydrogen Bond Acceptor', 'datatype': 5, 'implementation': 'E_NHACCEPTORS', 'version': '3.4.6.11', 'software': 'Cactvs', 'source': 'xemistry.com', 'release': '2019.06.18'}, 'value': {'ival': 1}}, {'urn': {'label': 'Count', 'name': 'Hydrogen Bond Donor', 'datatype': 5, 'implementation': 'E_NHDONORS', 'version': '3.4.6.11', 'software': 'Cactvs', 'source': 'xemistry.com', 'release': '2019.06.18'}, 'value': {'ival': 1}}, {'urn': {'label': 'Count', 'name': 'Rotatable Bond', 'datatype': 5, 'implementation': 'E_NROTBONDS', 'version': '3.4.6.11', 'software': 'Cactvs', 'source': 'xemistry.com', 'release': '2019.06.18'}, 'value': {'ival': 0}}, {'urn': {'label': 'Fingerprint', 'name': 'SubStructure Keys', 'datatype': 16, 'parameters': 'extended 2', 'implementation': 'E_SCREEN', 'version': '3.4.6.11', 'software': 'Cactvs', 'source': 'xemistry.com', 'release': '2019.06.18'}, 'value': {'binary': '0000037180402000000000000000000000000000000000000000000000000000000000000000001A00000800000000A080020200000002000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000'}}, {'urn': {'label': 'IUPAC Name', 'name': 'Allowed', 'datatype': 1, 'version': '2.6.6', 'software': 'LexiChem', 'source': 'openeye.com', 'release': '2019.06.18'}, 'value': {'sval': 'ethanol'}}, {'urn': {'label': 'IUPAC Name', 'name': 'CAS-like Style', 'datatype': 1, 'version': '2.6.6', 'software': 'LexiChem', 'source': 'openeye.com', 'release': '2019.06.18'}, 'value': {'sval': 'ethanol'}}, {'urn': {'label': 'IUPAC Name', 'name': 'Markup', 'datatype': 1, 'version': '2.6.6', 'software': 'LexiChem', 'source': 'openeye.com', 'release': '2019.06.18'}, 'value': {'sval': 'ethanol'}}, {'urn': {'label': 'IUPAC Name', 'name': 'Preferred', 'datatype': 1, 'version': '2.6.6', 'software': 'LexiChem', 'source': 'openeye.com', 'release': '2019.06.18'}, 'value': {'sval': 'ethanol'}}, {'urn': {'label': 'IUPAC Name', 'name': 'Systematic', 'datatype': 1, 'version': '2.6.6', 'software': 'LexiChem', 'source': 'openeye.com', 'release': '2019.06.18'}, 'value': {'sval': 'ethanol'}}, {'urn': {'label': 'IUPAC Name', 'name': 'Traditional', 'datatype': 1, 'version': '2.6.6', 'software': 'LexiChem', 'source': 'openeye.com', 'release': '2019.06.18'}, 'value': {'sval': 'ethanol'}}, {'urn': {'label': 'InChI', 'name': 'Standard', 'datatype': 1, 'version': '1.0.5', 'software': 'InChI', 'source': 'iupac.org', 'release': '2019.06.18'}, 'value': {'sval': 'InChI=1S/C2H6O/c1-2-3/h3H,2H2,1H3'}}, {'urn': {'label': 'InChIKey', 'name': 'Standard', 'datatype': 1, 'version': '1.0.5', 'software': 'InChI', 'source': 'iupac.org', 'release': '2019.06.18'}, 'value': {'sval': 'LFQSCWFLJHTTHZ-UHFFFAOYSA-N'}}, {'urn': {'label': 'Log P', 'name': 'XLogP3-AA', 'datatype': 7, 'version': '3.0', 'source': 'sioc-ccbg.ac.cn', 'release': '2019.06.18'}, 'value': {'fval': -0.1}}, {'urn': {'label': 'Mass', 'name': 'Exact', 'datatype': 7, 'version': '2.1', 'software': 'PubChem', 'source': 'ncbi.nlm.nih.gov', 'release': '2019.06.18'}, 'value': {'fval': 46.041864813}}, {'urn': {'label': 'Molecular Formula', 'datatype': 1, 'version': '2.1', 'software': 'PubChem', 'source': 'ncbi.nlm.nih.gov', 'release': '2019.06.18'}, 'value': {'sval': 'C2H6O'}}, {'urn': {'label': 'Molecular Weight', 'datatype': 7, 'version': '2.1', 'software': 'PubChem', 'source': 'ncbi.nlm.nih.gov', 'release': '2019.06.18'}, 'value': {'fval': 46.07}}, {'urn': {'label': 'SMILES', 'name': 'Canonical', 'datatype': 1, 'version': '2.1.5', 'software': 'OEChem', 'source': 'openeye.com', 'release': '2019.06.18'}, 'value': {'sval': 'CCO'}}, {'urn': {'label': 'SMILES', 'name': 'Isomeric', 'datatype': 1, 'version': '2.1.5', 'software': 'OEChem', 'source': 'openeye.com', 'release': '2019.06.18'}, 'value': {'sval': 'CCO'}}, {'urn': {'label': 'Topological', 'name': 'Polar Surface Area', 'datatype': 7, 'implementation': 'E_TPSA', 'version': '3.4.6.11', 'software': 'Cactvs', 'source': 'xemistry.com', 'release': '2019.06.18'}, 'value': {'fval': 20.2}}, {'urn': {'label': 'Weight', 'name': 'MonoIsotopic', 'datatype': 7, 'version': '2.1', 'software': 'PubChem', 'source': 'ncbi.nlm.nih.gov', 'release': '2019.06.18'}, 'value': {'fval': 46.041864813}}], 'count': {'heavy_atom': 3, 'atom_chiral': 0, 'atom_chiral_def': 0, 'atom_chiral_undef': 0, 'bond_chiral': 0, 'bond_chiral_def': 0, 'bond_chiral_undef': 0, 'isotope_atom': 0, 'covalent_unit': 1, 'tautomers': -1}}]}\n"
     ]
    }
   ],
   "source": [
    "etoh_json = get_full('ethanol')\n",
    "print(etoh_json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this is the same JSON file we saw earlier. Now we could apply our bond counting function or SMILES/molecular weight extraction to get this information for any compound if we know its name or CAS number."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Write a function that returns the SMILES string for any compound based on CAS number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CCO\n"
     ]
    }
   ],
   "source": [
    "def returnSMILES(CAS):\n",
    "    r = requests.get('https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/name/{}/record/json'.format(CAS))\n",
    "    chem_json = json.loads(r.text)\n",
    "    \n",
    "    SMILES = chem_json['PC_Compounds'][0]['props'][18]['value']['sval']\n",
    "    return SMILES\n",
    "\n",
    "print(returnSMILES('64-17-5'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python API's\n",
    "\n",
    "RESTful API's are widely used and easy to interact with. However, reading the documentation and converting more complex queries into the proper URL can be tedious and time consuming, especially because every RESTful API will use a different protocol. Furthermore, not all data sources use RESTful API's.\n",
    "\n",
    "Python is one of the most common languages for API's, and widely-used data sources (e.g. PubChem) will often have a Python \"wrapper\" for their RESTful API.\n",
    "\n",
    "We can use the [PubChemPy](https://pypi.python.org/pypi/PubChemPy/1.0) API to achieve the same goal, but we will need to install it first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# ! pip install PubChemPy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can import the API and will have access to intuitive function names and documentation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pubchempy as pcpy\n",
    "#help(pcpy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python APIs make code more readable, and are more intuitive to learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Compound(702)]\n",
      "2\n",
      "C\n",
      "C\n"
     ]
    }
   ],
   "source": [
    "compounds = pcpy.get_compounds('Ethanol','name')\n",
    "print(compounds)\n",
    "etoh = compounds[0]\n",
    "print(etoh.bonds[0].aid2)\n",
    "print(etoh.atoms[etoh.bonds[0].aid1].element)\n",
    "print(etoh.atoms[etoh.bonds[0].aid2].element)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the full .json output is already parsed into a nice Python data structure that can be accessed by attributes and has element symbols for each atom. We can also inspect this object like other Python objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['__class__',\n",
       " '__delattr__',\n",
       " '__dict__',\n",
       " '__dir__',\n",
       " '__doc__',\n",
       " '__eq__',\n",
       " '__format__',\n",
       " '__ge__',\n",
       " '__getattribute__',\n",
       " '__gt__',\n",
       " '__hash__',\n",
       " '__init__',\n",
       " '__init_subclass__',\n",
       " '__le__',\n",
       " '__lt__',\n",
       " '__module__',\n",
       " '__ne__',\n",
       " '__new__',\n",
       " '__reduce__',\n",
       " '__reduce_ex__',\n",
       " '__repr__',\n",
       " '__setattr__',\n",
       " '__sizeof__',\n",
       " '__str__',\n",
       " '__subclasshook__',\n",
       " '__weakref__',\n",
       " '_atoms',\n",
       " '_bonds',\n",
       " '_record',\n",
       " '_setup_atoms',\n",
       " '_setup_bonds',\n",
       " 'aids',\n",
       " 'atom_stereo_count',\n",
       " 'atoms',\n",
       " 'bond_stereo_count',\n",
       " 'bonds',\n",
       " 'cactvs_fingerprint',\n",
       " 'canonical_smiles',\n",
       " 'charge',\n",
       " 'cid',\n",
       " 'complexity',\n",
       " 'conformer_id_3d',\n",
       " 'conformer_rmsd_3d',\n",
       " 'coordinate_type',\n",
       " 'covalent_unit_count',\n",
       " 'defined_atom_stereo_count',\n",
       " 'defined_bond_stereo_count',\n",
       " 'effective_rotor_count_3d',\n",
       " 'elements',\n",
       " 'exact_mass',\n",
       " 'feature_selfoverlap_3d',\n",
       " 'fingerprint',\n",
       " 'from_cid',\n",
       " 'h_bond_acceptor_count',\n",
       " 'h_bond_donor_count',\n",
       " 'heavy_atom_count',\n",
       " 'inchi',\n",
       " 'inchikey',\n",
       " 'isomeric_smiles',\n",
       " 'isotope_atom_count',\n",
       " 'iupac_name',\n",
       " 'mmff94_energy_3d',\n",
       " 'mmff94_partial_charges_3d',\n",
       " 'molecular_formula',\n",
       " 'molecular_weight',\n",
       " 'monoisotopic_mass',\n",
       " 'multipoles_3d',\n",
       " 'pharmacophore_features_3d',\n",
       " 'record',\n",
       " 'rotatable_bond_count',\n",
       " 'shape_fingerprint_3d',\n",
       " 'shape_selfoverlap_3d',\n",
       " 'sids',\n",
       " 'synonyms',\n",
       " 'to_dict',\n",
       " 'to_series',\n",
       " 'tpsa',\n",
       " 'undefined_atom_stereo_count',\n",
       " 'undefined_bond_stereo_count',\n",
       " 'volume_3d',\n",
       " 'xlogp']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dir(etoh)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the PubChemPy API to ask for specific attributes with the `get_properties` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'CID': 702, 'CanonicalSMILES': 'CCO'}]\n"
     ]
    }
   ],
   "source": [
    "p = pcpy.get_properties('CanonicalSMILES', 'ethanol', 'name')\n",
    "print(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This provides a good tradeoff between the flexibility of the RESTful API and easy to read code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Write a function that takes an arbitrary chemical name or CAS number and returns the number of C-H bonds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def countCH(name):\n",
    "    c = pcpy.get_compounds(name, 'name')\n",
    "    bonds = c[0].bonds\n",
    "    \n",
    "    count = 0\n",
    "    for bond in bonds:\n",
    "        if c[0].atoms[bond.aid1 - 1].element == 'C' and c[0].atoms[bond.aid2 - 1].element == 'H':\n",
    "            count += 1\n",
    "        elif c[0].atoms[bond.aid1 - 1].element == 'H' and c[0].atoms[bond.aid2 - 1].element == 'C':\n",
    "            count += 1\n",
    "            \n",
    "    return count\n",
    "\n",
    "countCH('ethanol')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few notes about accessing data with APIs:\n",
    "\n",
    "* Every data source will have different structures and standards\n",
    "* APIs can sometimes be outdated if they are not maintained properly \n",
    "* Some APIs require \"keys\" to gain access\n",
    "* Many APIs (including PubChem) have limits on data transfer rates\n",
    "* Some APIs have terms of use that should not be violated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, Python API's are the best option for accessing online data with Python, though sometimes they can also be difficult to understand, or may contain bugs if they are not developed by the official maintainers of the dataset. RESTful API's are a good backup option, since they are relatively flexible and easy to access with Python. If this isn't available, then look for JSON or XML versions of the webpage or data source that can be parsed to extract data. Obtaining data by \"scraping\" HTML should only be done as a last resort since it is time consuming and will not work if the website updates its HTML structure."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}