python_1.1_introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python: Introduction\n",
    "\n",
    "Data science is like painting. Your dataset is your canvas, your statistical techniques are your colors, and your programming allows you to express your technique and artistry. Data scientists with strong programming skills are more able to quickly implement techniques and be more creative problem solvers and scientists. This module will guide you through the basics of programming with Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why use Python?\n",
    "There are many computer languages out there, of which Python is one. The most popular languages are Java and C/C++, known for their platform portability (Java) and speed (C). However, for working with data, the two dominant languages today are Python and R. You'll be exposed to R in the statistics module of this online review, but in this module, Python is the focus. \n",
    "\n",
    "There are numerous strengths of this language, which have helped to propel it to becoming one of the most popular languages for data science:\n",
    "1. **Readability**. Many consider Python to be more natural to read than other programming languages.\n",
    "2. **Scientific programming**. With packages like `NumPy` (efficient numerical computing), `SciPy` (optimization, linear algebra, integration, and much more), `matplotlib` (plotting), `StatsModels` (statistics), and `scikit-learn` (machine learning), Python is well-equipped to handly nearly any scientific programming challenge \n",
    "3. **Object oriented**. From the ground up, Python is \"object oriented\", with all of the primary data types being objects. We'll expore what objects are and their benefits in later units, but their benefits including making the code more modular, testable, extensible, maintainable, and reusable.\n",
    "4. **Free**. Python is free and open source. Enough said.\n",
    "5. **Portable**. Python works on nearly all systems including Macs, Unix, Linux, and Windows.\n",
    "6. **Easy to use and learn**. Due to the language's readbility and clean syntax, you can often get what you want done in many fewer steps than with C, C++, or Java.\n",
    "\n",
    "One of the biggest weaknesses of Python is it's performance. Python is slower than compiled languages like C. However, an increasing number of C-based Python modules are being developed to make Python more efficient. We'll see this in detail when we talk about vectorization with `NumPy`.\n",
    "\n",
    "How does Python relate to the programming language R? For many applications, either Python or R will get the job done, and both are worth knowing for the aspiring data scientist.  R is the primary language written and used by statisticians, so new statistical tecniques are often written in R before any other language.  Python, on the other hand, was developed by computer scientists, and is used in many industries.  Python has risen to the top of rankings of both the data science community and the broader programming community.  The [Kaggle State of Data Science Survey](https://www.kaggle.com/surveys/2017), conducted by the preeminant host of online machine learning competitions, rated Python as the most used programming language of data science professionals. Additionally, those same professionals overwhelmingly recommend Python as the programming language for new data scientists to learn first. Outside of data science, the [Institute of Electrical and Electronics Engineers (IEEE) ranked Python](https://spectrum.ieee.org/computing/software/the-2017-top-programming-languages) as the #1 programming language in 2017. In the [TIOBE software index for 2018 ranks Python](https://www.tiobe.com/tiobe-index/) (shown below), Python is the number 4 programming language just after Java, C, and C++, the dominant languages in industry. Just being in the same league as those industry standards demonstrates that Python is a highly transferrable skill that can help to propel your career in diverse directions.\n",
    "\n",
    "<img src=\"img/tiobe.png\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fun fact: why is it called Python?\n",
    "Contrary to popular belief, the Python programming language was not named after a snake.  The creator of Python, Guido van Rossum, was reading the scripts of the show \"Monty Python's Flying Circus\" (a BBC comedy series from the 1970's with many iconic sketches) when he was creating the tool. The show provided the inspiration for the name of the programming language."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interpreted language vs compiled languages\n",
    "*Section adapted from Charles R. Severance, \"Python for Everybody\" [Chapter 1](https://www.py4e.com/html3/01-intro)*\n",
    "\n",
    "Python is a high-level language intended to be relatively straightforward for humans to read and write, and for computers to read and process. what does it mean to say that a programming language is high-level?  It means that the actual hardware inside the Central Processing Unit (CPU) of the computer does not understand it.  Instead, the CPU understands a language we call machine language. Machine language is very simple and frankly very tiresome to write because it is represented all in zeros and ones:\n",
    "\n",
    "```\n",
    "001010001110100100101010000001111\n",
    "11100110000011101010010101101101\n",
    "...\n",
    "```\n",
    "\n",
    "Machine language seems quite simple on the surface, given that there are only zeros and ones, but its syntax is even more complex and far more intricate than Python. Very few programmers ever write machine language. Instead, we build high-level languages that serve as translators.  Python is one of those translators; other translator languages include Java, C++, PHP, Ruby, Basic, Perl, JavaScript, and many more. When we write programs in these languages, we are essentially writing instructions that will be translated to machine language for actual execution by the CPU.  \n",
    "\n",
    "Since machine language is tied to the computer hardware, machine language is not portable across different types of hardware. Programs written in high-level languages, on the other hand, can be moved between different computers by using a different interpreter on the new machine or recompiling the code to create a machine language version of the program for the new machine.\n",
    "\n",
    "Programming language translators fall into two general categories: (1) interpreters and (2) compilers.\n",
    "\n",
    "An interpreter reads a programmer's code, parses the code, and interprets the instructions written in the code on the fly. Python is an interpreter and when we are running Python interactively, we can type a line of Python and Python processes it immediately and is ready for us to type another line of Python.  \n",
    "\n",
    "Unlike interpreters, compilers need to be handed an entire program in a file.  Compilers translate the instructions within the program into machine language, and then puts the resulting machine language into a separate file for later execution.  You have likely seen these separate files, called \"executables\", before.  If you have a Windows system, executable machine language programs often have a suffix of \".exe\" or \".dll\" which stand for \"executable\" and \"dynamic link library\" respectively. In Linux and Macintosh, there is no suffix that uniquely marks a file as executable.\n",
    "\n",
    "If you were to open an executable file in a text editor, it would look completely crazy and be unreadable:\n",
    "\n",
    "```\n",
    "^?ELF^A^A^A^@^@^@^@^@^@^@^@^@^B^@^C^@^A^@^@^@\\xa0\\x82\n",
    "^D^H4^@^@^@\\x90^]^@^@^@^@^@^@4^@ ^@^G^@(^@$^@!^@^F^@\n",
    "^@^@4^@^@^@4\\x80^D^H4\\x80^D^H\\xe0^@^@^@\\xe0^@^@^@^E\n",
    "^@^@^@^D^@^@^@^C^@^@^@^T^A^@^@^T\\x81^D^H^T\\x81^D^H^S\n",
    "^@^@^@^S^@^@^@^D^@^@^@^A^@^@^@^A\\^D^HQVhT\\x83^D^H\\xe8\n",
    "....\n",
    "```\n",
    "\n",
    "It is not easy to read or write machine language, so it is nice that we have interpreters and compilers that allow us to write in high-level languages like Python or C.\n",
    "\n",
    "Now at this point in our discussion of compilers and interpreters, you should be wondering a bit about the Python interpreter itself. What language is it written in? Is it written in a compiled language? \n",
    "\n",
    "The Python interpreter is written in a high-level language called \"C\". You can look at the actual source code for the Python interpreter by going to www.python.org and working your way to their source code. So Python is a program itself and it is compiled into machine code. When you instal Python on your computer (or the vendor instals it), you copy a machine-code copy of the translated Python program onto your system. In Windows, the executable machine code for Python itself is likely in a file with a name like:\n",
    "\n",
    "```\n",
    "C:\\Python36\\python.exe\n",
    "```\n",
    "\n",
    "That is more than you really need to know to be a Python programmer, but sometimes it pays to answer those little nagging questions right at the beginning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding the Python ecosystem\n",
    "With programming, we can group the core components we need for a project as (1) the code itself (to the left in the figure below) and (2) the tools to use the code (to the right in the figure below). \n",
    "\n",
    "<img src='./img/python_ecosystem.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Terminal, command line, shells, and kernels\n",
    "All of the following terms describe tools we use to interact with an operating system and execute code via commands. In programming, your first step is often to open up a terminal window.\n",
    "\n",
    "The **terminal** (or terminal emulator / console for some systems) is a program that gives us a way to talk to the core underlying computational engine, the kernel. On Mac systems this is called the Terminal, although you can download alternatives. On windows systems, you can access a terminal via the Command Prompt or Windows PowerShell. For this course on Windows we recommend a more user friendly terminal such as [cmder](http://cmder.net/). \n",
    "\n",
    "<img src='./img/terminal.png' style=\"width: 600px;\">\n",
    "\n",
    "When you open the terminal, you will have a **command line** user interface where you can send commands and you can receive output. This is often indicated with a `>`, a `$`, or in cmder it is a $\\lambda$ symbol. The command line is your portal to communicate in the language of code. This interface allows you to communicate with the underlying interpreter of those commands, known as the shell. The **shell** translates what we type into the terminal into a form the computational engine, the **kernel**, can understand. The shell interprets the commands you send via the terminal's command line to the kernel, runs the appropriate code, and outputs any results.  There are multiple kinds of shells you can use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python shell: IDLE\n",
    "Short for \"Integrated DeveLopement Environment\", this shell is the most basic interactive way of working with Python and comes standard with the Python language. It's a Python shell, so it interprets a user's commands to interact with the computational engine, the Python kernel. To get to IDLE, we enter python on the command line and we see the symbol >>> waiting for input. You know you're using IDLE when you see the >>> symbol on the lefthand side of your terminal window. While IDLE will happily interpret your Python commands for you, its limited features makes it less frequently used than the more powerful IPython shell. \n",
    "\n",
    "*This is what opens when you type python on the command line, but I don't recommend using it for day-to-day programming.*\n",
    "\n",
    "<img src='./img/idle_shell.png' style=\"width: 600px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python shell: IPython\n",
    "Short for \"Interactive Python,\" this shell is a much more powerful tool for interactive command line Python programming. It provides a command line with syntax highlighting (which highlights different types of syntax with different colors for clarity), tab completion (hit the tab button and it will try to automatically complete your statement), and the ability to search through  the command history, among many benefits. \n",
    "\n",
    "*If you use an interactive shell - this is the one to use.*\n",
    "\n",
    "\n",
    "<img src='./img/ipython_shell.png' style=\"width: 600px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A text editor\n",
    "Although you can enter commands directly into a Python shell and it will run them, as you will see, it is very difficult to write complicated programs \"on the fly\" at the command line. Therefore, when we want to write a program, we use a text editor to write the Python instructions into a text file, which is called a script. By convention, Python scripts have file names that end with '.py'.\n",
    "  \n",
    "There are many good text editors out there including [Sublime Text 3](https://www.sublimetext.com/), [Emacs](https://www.gnu.org/software/emacs/), [VIM](https://www.vim.org/),  [Notepad++](https://notepad-plus-plus.org/). While each of these offers different features for the the precocious programmer, any of them are capable of editing code that can then be executed on the command line or through an IDE. You can write working code to accomplish your task from any text editor. However, integrated development environments provide additional tools that will be useful to you all together in one environment, as we'll discuss next.\n",
    "\n",
    "*If you haven't chosen a text editor my personal recommendation is [Sublime Text 3](https://www.sublimetext.com/). It's a powerful and extensible text editor with many helpful tools to make coding more efficient.  You will also be learning how to use Emacs when you get to campus.*\n",
    "\n",
    "<img src='./img/sublime.png' style=\"width: 600px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python IDE: Spyder\n",
    "Spyder is one example of an Integrated Development Environment (IDE).  The Spyder IDE combines many pieces that are helpful when you want to code a complex programming project.  To start with, it has a text editor for writing code and an IPython shell for running Python code and viewing its output in one place.  In addition, it also has a variable explorer that allows you to see what's stored in variables you've created (we will learn more about what this means later), debugging tools for identifying and fixing errors in code, and a file explorer for navigating the file system on your computer.\n",
    "\n",
    "Spyder isn't the only IDE; there's also [PyCharm](https://www.jetbrains.com/pycharm/), [Rodeo](https://rodeo.yhat.com/), and Jupyter Notebooks.\n",
    "\n",
    "<mark>**For this course, do all of your coding in the Spyder IDE on your Duke virtual machine (see the \"Tools used in this course\" section of the Sakai website for more details about what virtual machines are and how to connect to them)**.</mark> Spyder has everything you need to successfully program and the flexibility to scale up to larger projects.\n",
    "\n",
    "### Required Video: [Introduction to the Spyder IDE](https://youtu.be/zYNRqVimU3Q)\n",
    "\n",
    "<img src='./img/spyder_interface.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Jupyter Notebook\n",
    "If you are reading this, you are using a Jupyter notebook I created for you right now! Formerly known as IPython notebooks, this tool is built on IPython and enables the user to combine text (formatted through [markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Here-Cheatsheet) language, which you can learn through an interactive tutorial [markdown](http://commonmark.org/help/)), [latex equations](https://en.wikibooks.org/wiki/LaTeX/Mathematics), Python code (and R code), and outputs from running code right in the document including plots, all in one nicely formatted web page that can be easily shared and/or made available online. This is the major benefit of Jupyter notebooks: being able to combine so many pieces in one place seamlessly. There are even tools to convert Jupyter Notebooks directly into static websites and blogs.\n",
    "\n",
    "In Jupyter Notebooks, you can work just as easily with text and equations...\n",
    "\n",
    "<img src='./img/jupyter1.png' style=\"width: 750px;\">\n",
    "\n",
    "...as you can work with code and plots.\n",
    "\n",
    "<img src='./img/jupyter2.png' style=\"width: 750px;\">\n",
    "\n",
    "However, in Jupyter Notebooks, you don't have access to a variable explorer as with the Spyder IDE, so while great for communicating your work, it's not always the most convenient for day-to-day development. For that, a fully-featured IDE like Spyder is recommended. [Jupyterlab](https://jupyterlab.readthedocs.io/en/stable/) has recently been released, and will likely supersede Jupyter Notebooks in upcoming years due to providing both the notebook functionality in addition to enhanced functionality.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Packages\n",
    "Most of this module is focused on the code itself - it's mechanics or syntax and how you can use its conventions to solve problems and answer questions. All of what we do is built on the underlying Python programming language. However, built on that core language are a number of fundamental packages (collections of code) that we'll use for our data analysis including `NumPy` (for numerical array and matrix processing), `SciPy` (for optimization, linear algebra, and numeric integration), and `matplotlib` (for plotting). Built on top of those fundamental packages and the core Python language are higher level packages that include `scikit-learn` (machine learning), `pandas` (data manipulation and analysis), `StatsModels` (statistical programming), and `scikit-image` (image processing), among other (e.g. `TensorFlow`). Together, these components are able to solve a wide array of today's scientific computing challenges."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is a program?\n",
    "\n",
    "The whole point of this module is to learn Python programming.  But what is a program?  The definition of a program at its most basic is a sequence of Python statements that have been crafted to do something. Say you've created a text file called hello.py with the following contents:\n",
    "\n",
    "print('Hello world!')\n",
    "\n",
    "That counts as a program (a one-line program)!  It is not particularly useful, but in the strictest definition, it is a Python program.\n",
    "\n",
    "Most programs are more complicated than that.  Let's think about a problem that a program might be built to solve, and then looking at a program that would solve that problem.\n",
    "\n",
    "Lets say you are doing Social Computing research on Facebook posts and you are interested in the most frequently used word in a series of posts. You could print out the stream of Facebook posts and pore over the text looking for the most common word, but that would take a long time and be very mistake prone. You would be smart to write a Python program to handle the task quickly and accurately so you can spend the weekend doing something fun.\n",
    "\n",
    "For example, look at the following text about a clown and a car. Look at the text and figure out the most common word and how many times it occurs.\n",
    "\n",
    "```\n",
    "the clown ran after the car and the car ran into the tent\n",
    "and the tent fell down on the clown and the car\n",
    "```\n",
    "\n",
    "It would take you a long time to do this if you had to look at millions of texts.  Fortunately, here's a program called \"words.py\" that could do that task for you:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "name = input('Enter file:')\n",
    "handle = open(name, 'r')\n",
    "counts = dict()\n",
    "\n",
    "for line in handle:\n",
    "    words = line.split()\n",
    "    for word in words:\n",
    "        counts[word] = counts.get(word, 0) + 1\n",
    "\n",
    "bigcount = None\n",
    "bigword = None\n",
    "for word, count in list(counts.items()):\n",
    "    if bigcount is None or count > bigcount:\n",
    "        bigword = word\n",
    "        bigcount = count\n",
    "\n",
    "print(bigword, bigcount)\n",
    "\n",
    "# Code: http://www.py4e.com/code3/words.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll learn many of the techniques in this program (and many more besides) throughout this course. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How do I learn to program?\n",
    "One of the most important skills you can develop is learning to learn. While this course provides the pathway to greater understanding, it requires engagement and facing desirable difficulties of not always knowing exactly what to do to learn how to solve problems programmatically. Reading every textbook and watching every video on programming will not, alone, develop your skills. Becoming good at programming is much like becoming good at any spoken language. You will only develop your skills by hands-on experience using the language. This process will be hard at times, and you'll sometimes feel like you're struggling to understand a concept or fix a bug. That's natural and that's often when you learn the most. Here are some tips for success:\n",
    "\n",
    "1. **Embrace effort**. Always work through all the examples until the end - don't stop and assume that you could figure out the rest. Sometimes this struggle is uncomfortable, but know that it's through that struggle that you'll learn the most, and that's a good thing! Numerous research studies have lauded \"desirable difficulties\" in the learning process. \n",
    "2. **When you get stuck, search**. If you get stuck - search or an answer yourself and you'll be surprised how many times there's someone who has already answered your question as the internet is awash with resources for answering targeted questions. Every time I go to code something that I haven't done before, I inevitably end up looking on one of a number of websites / forums such as [Stack Overflow](https://stackoverflow.com/), [Stack Exchange](https://stackexchange.com/), [Quora](https://www.quora.com/), or a host of other sites that are best found through a Google search of your topic. A key skill to develop is knowing when to trust a source and when to keep looking.\n",
    "3. **When in doubt, draw it out**. When you're trying to accomplish something algorithmically, think through and draw out (on paper) the logic that you're trying to implement before typing in code. You'll be extremely surprised how many times you'll realize a flaw in your logic or something you missed when you get it down on a page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "Through this course, we will guide you through the basics of Python programming: using the command line and the core components of the Python ecosystem, implementing core programming concepts, and applying syntax you'll need to operate the mechanics of the system to be an effective Python programmer. Then, we'll explore the key data analysis modules that make Python shine as an exemplary tool for data science.\n",
    "\n",
    "Throughout this course **TRY EVERYTHING YOURSELF**. This is key to learning. If I provide you with code, try running it on the IPython console in Spyder (or in a cell in one of the Jupyter notebooks) and make sure your output matches expectations. This will help you to check your understanding along the way and provide practice working with the syntax. Also, work through the exercises presented at the end of sections along with the explanation. The more you try on your own and put in the effort to trying, the more you will learn."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}