06.05.data-cleaning-exercises.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"anaconda-cloud":{},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.1"},"colab":{"name":"06.05.data-cleaning-exercises.ipynb","provenance":[],"collapsed_sections":[]}},"cells":[{"cell_type":"markdown","metadata":{"colab_type":"text","id":"Es-ceD1D3ytV"},"source":["<hr>\n","\n","##### Mount Drive - **Google Colab Only Step**\n","\n","When using google colab in order to access files on our google drive we need to mount the drive by running the below python cell, then clicking the link it generates and pasting the code in the cell.\n","\n"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"YKHWrhnJ3ytR","colab":{}},"source":["from google.colab import drive\n","drive.mount('/content/drive')"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"x5AzSk7g3ytO"},"source":["Change Directory To Access The Dependent Files - **Google Colab Only Step**"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"j9rgWUe03ytL","colab":{}},"source":["directory = \"student\"\n","if (directory == \"student\"):\n","  %cd drive/My\\ Drive/Colab\\ Notebooks/data-science-track/\n","else:\n","  %cd drive/Shared\\ drives/Rubrik/Data\\ Science/Course/Data-Science-Track"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"VpxtAxC8MTC4","colab_type":"text"},"source":["<h1 style=\"font-size:42px; text-align:center; margin-bottom:30px;\"> Data Cleaning Exercise</h1>\n","<hr>\n","Welcome to the workbook for <span style=\"color:royalblue\">Exercise 2: Data Cleaning</span>! \n","\n","Remember, **better data beats better algorithms**.\n","\n","\n","<br><hr id=\"toc\">\n","\n","### In this lesson...\n","\n","In this lesson, we'll cover the essential steps for building your analytical base table:\n","1. [Drop unwanted observations](#drop)\n","2. [Fix structural errors](#structural)\n","3. [Handle missing data](#missing-data)\n","\n","Finally, we'll save the data to a new file so we can use it in other lessons.\n","\n","<br><hr>"]},{"cell_type":"markdown","metadata":{"id":"sS6jI8MdMTC_","colab_type":"text"},"source":["### First, let's import libraries and load the dataset.\n","\n","In general, it's good practice to keep all of your library imports at the top of your notebook or program.\n","\n","We've provided comments for guidance."]},{"cell_type":"code","metadata":{"id":"VFI-q0cmMTDC","colab_type":"code","colab":{}},"source":["# NumPy for numerical computing\n","\n","# Pandas for DataFrames\n","\n","\n","# Matplotlib for visualization\n","\n","# Seaborn for easier visualization"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"WlvBgDG8vW8I","colab_type":"text"},"source":["## Import the employee dataset\n","- Use pandas' `read_csv()` function \n","- Provide the following path for the data \n","```python \n","path = './data/employee_data.csv'\n","```"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"N1phkZx45jXW","colab":{}},"source":["# Load employee data from CSV"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"vFZhlu0yMTDX","colab_type":"text"},"source":["<br>"]},{"cell_type":"markdown","metadata":{"id":"CFUEppA7MTDZ","colab_type":"text"},"source":["### **Now we're ready to jump into cleaning the data!**\n","### ..."]},{"cell_type":"markdown","metadata":{"id":"1oPZ-k9CMTDb","colab_type":"text"},"source":["<br>"]},{"cell_type":"markdown","metadata":{"id":"19I6MgnmMTDd","colab_type":"text"},"source":["<span id=\"drop\"></span>\n","# 1. Drop Unwanted observations\n","\n","The first step to data cleaning is removing samples from your dataset that you don't want to include in the model.\n","\n","<br>\n","\n","**First, <span style=\"color:royalblue\">drop duplicates</span> from the dataset.**\n","* Then, print the shape of the new dataframe."]},{"cell_type":"code","metadata":{"id":"29Um3mMGMTDf","colab_type":"code","colab":{}},"source":["# Drop duplicates"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"GWDEHou6MTDk","colab_type":"text"},"source":["<br>"]},{"cell_type":"markdown","metadata":{"id":"yqnOAvSkMTDm","colab_type":"text"},"source":["**Display all of the unique classes of the <code style=\"color:steelblue\">'department'</code> feature**"]},{"cell_type":"code","metadata":{"id":"om9Zfe-fMTDo","colab_type":"code","colab":{}},"source":["# Unique classes of 'department'\n"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"yEHXgm17MTDt","colab_type":"text"},"source":["<br>"]},{"cell_type":"markdown","metadata":{"id":"rKGlhubYMTDu","colab_type":"text"},"source":["**Drop all observations that belong to the <span style=\"color:crimson\">'temp'</span> department.**\n","* **Hint:** This is the same as keeping all that don't belong to that department.\n","* **Hint:** Remember to overwrite your original dataframe.\n","* Then, print the shape of the new dataframe."]},{"cell_type":"code","metadata":{"scrolled":true,"id":"FJEyv6iJMTDw","colab_type":"code","colab":{}},"source":["# Drop temporary workers\n"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"KH75NmcZMTD1","colab_type":"text"},"source":["<p style=\"text-align:center; margin: 40px 0 40px 0; font-weight:bold\">\n","<a href=\"#toc\">Back to Contents</a>\n","</p>"]},{"cell_type":"markdown","metadata":{"id":"4JvtoHuTMTD3","colab_type":"text"},"source":["<span id=\"structural\"></span>\n","# 2. Fix structural errors\n","\n","The next bucket under data cleaning involves fixing structural errors, which arise during measurement, data transfer, or other types of \"poor housekeeping.\"\n","\n","<br>\n","\n","**Print the unique values of <code style=\"color:steelblue\">'filed_complaint'</code> and <code style=\"color:steelblue\">'recently_promoted'</code>.**"]},{"cell_type":"code","metadata":{"id":"qUjkS4BnMTD5","colab_type":"code","colab":{}},"source":["# Print unique values of 'filed_complaint'\n"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"D8FgpJUCMTEA","colab_type":"code","colab":{}},"source":["# Print unique values of 'recently_promoted'\n"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"pvyvAbm-MTEH","colab_type":"text"},"source":["<br>"]},{"cell_type":"markdown","metadata":{"id":"Efose5G6MTEL","colab_type":"text"},"source":["**Fill missing <code style=\"color:steelblue\">'filed_complaint'</code> and <code style=\"color:steelblue\">'recently_promoted'</code> values with <code style=\"color:crimson\">0</code>.**"]},{"cell_type":"code","metadata":{"id":"4UJmZbIQMTEN","colab_type":"code","colab":{}},"source":["# NaN values in filed_complaint should be 0.\n"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"2PT-70IcMTES","colab_type":"code","colab":{}},"source":["# NaN values in recently_promoted should be 0.\n"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"vn8Axu3AMTEX","colab_type":"text"},"source":["<br>"]},{"cell_type":"markdown","metadata":{"id":"yvs-qjLQMTEZ","colab_type":"text"},"source":["**Print the unique values of <code style=\"color:steelblue\">'filed_complaint'</code> and <code style=\"color:steelblue\">'recently_promoted'</code> again, just to confirm.**"]},{"cell_type":"code","metadata":{"id":"mAWBOY63MTEa","colab_type":"code","colab":{}},"source":["# Print unique values of 'filed_complaint'\n"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"Qn0v1_3RMTEf","colab_type":"code","colab":{}},"source":["# Print unique values of 'recently_promoted'\n"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"thYKioQwMTEj","colab_type":"text"},"source":["<br>"]},{"cell_type":"markdown","metadata":{"id":"iIuI9oLKMTEl","colab_type":"text"},"source":["**Replace any instances of <code style=\"color:crimson\">'information_technology'</code> with <code style=\"color:crimson\">'IT'</code> instead.**\n","* Remember to do it **inplace**, OR better yet, save over the column.\n","* Then, plot the **bar chart** for <code style=\"color:steelblue\">'department'</code> to see its new distribution."]},{"cell_type":"code","metadata":{"id":"iIF-7XTPMTEm","colab_type":"code","colab":{}},"source":["# 'information_technology' should be 'IT'\n"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"DXzweUDGMTEt","colab_type":"code","colab":{}},"source":["# Plot class distributions for 'department'\n"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"krxK7rd3MTEx","colab_type":"text"},"source":["<p style=\"text-align:center; margin: 40px 0 40px 0; font-weight:bold\">\n","<a href=\"#toc\">Back to Contents</a>\n","</p>"]},{"cell_type":"markdown","metadata":{"id":"y38lA40TMTEz","colab_type":"text"},"source":["<span id=\"missing-data\"></span>\n","# 3. Handle missing data\n","\n","Next, it's time to handle **missing data**. \n","\n","<br>\n","\n","**Display the <span style=\"color:royalblue\">number of missing values</span> for each feature (both categorical and numeric).**"]},{"cell_type":"code","metadata":{"id":"ojx2-pJvMTE2","colab_type":"code","colab":{}},"source":["# Display number of missing values for all features in the dataset.\n"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"DKt1zTPkMTE8","colab_type":"text"},"source":["<br>"]},{"cell_type":"markdown","metadata":{"id":"vlhAMAwOMTFC","colab_type":"text"},"source":["**Label missing values in <code style=\"color:steelblue\">'department'</code> as <code style=\"color:crimson\">'Missing'</code>.**\n","* By the way, the <code style=\"color:steelblue\">.fillna()</code> function also has an <code style=\"color:steelblue\">inplace=</code> argument, just like the <code style=\"color:steelblue\">.replace()</code> function.\n","* Normally, I recommend we overwrote that column. This time, try using the <code style=\"color:steelblue\">inplace=True</code> argument instead, which does not RETURN anything. Therefore no need to overwrite nay variable. "]},{"cell_type":"code","metadata":{"id":"Y7CCCKq6MTFE","colab_type":"code","colab":{}},"source":["# Fill missing values in department with 'Missing'\n"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"6T-dRIXhMTFI","colab_type":"text"},"source":["<br>"]},{"cell_type":"markdown","metadata":{"id":"kFTe00mNMTFJ","colab_type":"text"},"source":["**First, let's flag <code style=\"color:steelblue\">'last_evaluation'</code> with an indicator variable of missingness.** \n","* <code style=\"color:crimson\">0</code> if not missing.\n","* <code style=\"color:crimson\">1</code> if missing. \n","\n","Let's name the new indicator variable <code style=\"color:steelblue\">'last_evaluation_missing'</code>.\n","* We can use the <code style=\"color:steelblue\">.isnull()</code> function.\n","* Also, remember to convert it with <code style=\"color:steelblue\">.astype(int)</code>"]},{"cell_type":"code","metadata":{"id":"B1zTGm1zMTFL","colab_type":"code","colab":{}},"source":["# Indicator variable for missing last_evaluation\n"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"H7LLDvYHMTFO","colab_type":"text"},"source":["<br>"]},{"cell_type":"markdown","metadata":{"id":"g8Yahfi3MTFQ","colab_type":"text"},"source":["**Then, simply fill in the original missing value with <code style=\"color:crimson\">0</code> just so your algorithms can run properly.**\\\n","\n","<br>\n","<b style=\"color:crimson\">WARNING!</b> This is <b style=\"color:crimson\">NOT</b> standard practice!!! "]},{"cell_type":"code","metadata":{"id":"fgaiANs3MTFR","colab_type":"code","colab":{}},"source":["# Fill missing values in last_evaluation with 0\n"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"s35H1mvVMTFX","colab_type":"text"},"source":["<br>"]},{"cell_type":"markdown","metadata":{"id":"x1BgmPmjMTFZ","colab_type":"text"},"source":["**Display the number of missing values for each feature (both categorical and numeric) again, just to confirm.**"]},{"cell_type":"code","metadata":{"id":"nBovYQ7rMTFa","colab_type":"code","colab":{}},"source":["# Display number of missing values by feature\n"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"cOJGBTayMTFe","colab_type":"text"},"source":["<p style=\"text-align:center; margin: 40px 0 40px 0; font-weight:bold\">\n","<a href=\"#toc\">Back to Contents</a>\n","</p>"]},{"cell_type":"markdown","metadata":{"id":"Dd4c_SelMTFg","colab_type":"text"},"source":["<span id=\"save-abt\"></span>\n","# 5. Save the data\n","\n","Finally, let's save the **cleaned data**. \n","\n","<br>"]},{"cell_type":"code","metadata":{"id":"tArfESdjMTFh","colab_type":"code","colab":{}},"source":[" # Save the dataframe to csv\n"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"yMUOH4CEMTFk","colab_type":"text"},"source":["<p style=\"text-align:center; margin: 40px 0 40px 0; font-weight:bold\">\n","<a href=\"#toc\">Back to Contents</a>\n","</p>"]},{"cell_type":"markdown","metadata":{"id":"g4d4QoP3MTFl","colab_type":"text"},"source":["<br>\n","\n","## Next Steps\n","\n","Congratulations for making through Exercise 2's Employee data cleaning!\n","\n","As a reminder, here are a few things you did in this module:\n","* You cleaned dropped irrelevant observations from the dataset.\n","* You fixed various structural errors, such as wannabe indicator variables.\n","* You handled missing data.\n","\n","<p style=\"text-align:center; margin: 40px 0 40px 0; font-weight:bold\">\n","<a href=\"#toc\">Back to Contents</a>\n","</p>"]},{"cell_type":"code","metadata":{"id":"XxziLaXTMTFm","colab_type":"code","colab":{}},"source":[""],"execution_count":0,"outputs":[]}]}