2. Foundations of Machine Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Foundations of Machine Learning\n",
    "### Agenda\n",
    "<hr>\n",
    "* Introduction to Machine Learning\n",
    "* Supervised vs Unsupervised\n",
    "* Types of Supervised Learning - Classification, Regression\n",
    "* Data Ingestion\n",
    "* Data Wrangling\n",
    "* Data Preprocessing\n",
    "* Model Training\n",
    "* Model Validation\n",
    "* Deployment\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Introduction to Machine Learning\n",
    "* A branch of AI, where applications can automatically learn from data or experience is known as Machine Learning.\n",
    "* Traditional system needs explicit programming whereas a machine learning system derives logic from data.\n",
    "* Machine Learning applications can determine if a mail is spam, predict fraud, personalize entertainment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Supervised/Unsupervised Machine Learning \n",
    "* Machine Learning applications can be trained with data-with-labels & also data-without-labels.\n",
    "* Meaning of data-with-labels : Feature Data (House Details like sqft, num_rooms, num_floors etc.), Target Data or labels (Price)\n",
    "* When a model learns using data-with-labels, it's supervised learning.\n",
    "* When model learns patterns from unlabeled data, it's unsupervised learning.\n",
    "* Examples of unsupervised learning : Grouping similar customer together."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Types of Supervised Learning - Regression & Classification\n",
    "* Regression - Target data is continues in nature\n",
    "* Examples \n",
    "  - House Price Prediction\n",
    "  - Rainfall Prediction\n",
    "* Classification - target data is catagorical in nature\n",
    "  - Predicting Spam Mail\n",
    "  - Predicting credit defaulter\n",
    "  \n",
    "<hr>  \n",
    "  \n",
    "<img src=\"https://github.com/awantik/machine-learning-slides/blob/master/ML-Pipeline-business.png?raw=true\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Data Ingestion\n",
    "* Data gathering is one of the most important part of machine learning application development.\n",
    "* Without data there cannot be machine learning.\n",
    "* Data can be **Batched** as well as streamed.\n",
    "* Batched data can be read by spark from csv, json, hive, databases etc.\n",
    "* Streamed data can be recieved from streaming servers like Kafka.\n",
    "* **Spark Streaming** can do processing on data received from Kafka."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Data Wrangling\n",
    "* Transformation of data to a more valuable format suited for downstream applications like Business Intelligence, Analytics etc.\n",
    "* It involves cleaning of data.\n",
    "* Removing incorrect data.\n",
    "* Aggregating information.\n",
    "* Python provides Pandas for Data Wrangling task. \n",
    "* PySpark provides **DataFrames** for data wrangling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Data Preprocessing\n",
    "* PySpark provides Machine learning library for classification,regression,clustering etc.\n",
    "* Before data can be fed to machine learning libraries, it needs to be transformed to a format which is expected by machine leaning libraries\n",
    "* Scaling techniques for bringing numerical data to same scale.\n",
    "* Encoding techniques for converting categorical data to numbers.\n",
    "* Text vectorization techniques.\n",
    "* ML algorithms only understands numbers.\n",
    "* Dimensionality reduction techniques for noise reduction from data & improving computation speed.\n",
    "* All preprocessor are present in **pyspark.ml.feature** module.\n",
    "* We have to create an instance of preprocessor & pass data to it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Model Training\n",
    "* At this stage, data is ready for machine learning.\n",
    "* Based on the problem statement we might have to classification algorithm or regression algorithm.\n",
    "* Learning algorithms are part of **pyspark.ml.classification**, **pyspark.ml.regression** etc.\n",
    "* For supervised learning, pass feature data & target data.\n",
    "* For unsupervised learning, pass the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Model Validation & Selection\n",
    "* There are multiple models to be chosen from & multiple ways to configure models.\n",
    "* Model with perfect balance of bias & variance should be chosen.\n",
    "* For this we need to cross validate our model.\n",
    "* This step also includes hyper-parameter tuning.\n",
    "* Pyspark provides **pyspark.ml.tuning** for cross validation & hyper-parameter tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Deployment\n",
    "* Once the model is validated & selected, it needs to be deployed.\n",
    "* Persisting model for future reuse & shipping to customer.\n",
    "* Persisted model can be re-trained as well.\n",
    "* RESTful interface of model can be exposed for consumption"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}