machine-learning-using-pyspark

1. Understanding PySpark Ecosystem

Big Data
Hadoop
Spark
PySpark
Machine Learning using PySpark

2. Foundations of Machine Learning

Introduction to Machine Learning
Supervised vs Unsupervised
Classification vs Regression
Data Ingestion
Data Wrangling
Data Preprocessing
Model Training
Model Validation
Deployment

3. Internal Details of Spark

Driver
Executors
Partitions
Jobs
Stages
Tasks
Resilient Distributed Datastructure
DataFrames as a High Level Datastructure

4. Low level Understanding using RDD

Creation of RDD
Transformation methods
Aggregation methods
Actions
Caching
Debugging

5. Data Ingestion

Loading CSV, JSON & parquet
Connecting to databases
Getting data from streaming server

5. Data Wrangling using DataFrames

Descriptive Statistics
Accessing subsets of data - Rows, Columns, Filters
Handling Missing Data
Dropping rows & columns
Handling Duplicates
Aggregate functions
Merge, Join & Concatenate

6. Data Preprocessing

Why Preprocessing ?
Scaling Techniques
Encoding Techniques
Text Processing
Dimensionality Reduction
Vectorization of Data

7. Regression Learning Models

Linear Regression
Decision Tree Regressor
Random Forest Regressor
GBT Regressor
Evaluation of Regression Models

8. Classification Learning Models

LogisticRegression
DecisionTreeClassifier
GBT Classifier
RandomForestClassifier
NaiveBayes
MultiLayerPerceptronClassifier
Evaluation of Classification Models

9. Clustering Learning Models

Motivation behind clustering
KMeans
GaussianMixtureModel
Latent Dirichlet Allocation

10. Recommandation Engine

11. Pipeline & Hyper-parameter Tuning

Composite Estimators using Pipelines
Model Selection
Hyper-parameter Tuning
Persisting trained models
Deployment