2 - Resource Managers and Schedulers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ch. 2. Resource Managers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## I. Interacting With the Operating System\n",
    "Programming languages are nice for carrying out computations based on input data. They have useful constructs like `for` loops, `if` statements, and module systems so that you can reuse code easily. However, for certain tasks, what's built in to the programming language is simply not enough. For situations like this, programming languages usually allow explicit interaction with the underlying operating system. In Python, this is handled by the `os` module. The following graphic helps to describe how the underlying OS behaves and how we can interact with it:\n",
    "![OS Architecture Diagram](https://docs.mesosphere.com/1.11/img/dcos-architecture-layers.png)\n",
    "\n",
    "The code you have been running in these notebooks fits into the \"services\" section of the software layer, and our OS interaction will allow us to request data and provide instructions to the objects in the \"platform\" layer.\n",
    "\n",
    "\n",
    "It is important to remember that Python code is meant to be portable, and you have to be careful when using the `os` module to ensure that your code can still be run on your target systems. This course is designed for \\*nix  systems (including macOS and BSD), and certain code written in examples will not run on windows systems, especially code from this chapter. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2.1a - Basic Filesystem Interaction\n",
    "In this example, we will interact with the filesystem around us. The module `os` contains a lot of useful functionality for interacting with the filesystem, networking layers, and other OS-specific methods. For more information see the [`os` docs](https://docs.python.org/3/library/os.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ex. 2.1a - Filesystem interaction \n",
    "\n",
    "import os\n",
    "\n",
    "# We can list directories\n",
    "print(os.listdir('/home/users/glick/intro-to-hpc'))\n",
    "\n",
    "# We can make directories\n",
    "print(os.mkdir('/home/users/glick/intro-to-hpc/dira'))\n",
    "\n",
    "# List again to see the new directory\n",
    "print(os.listdir('/home/users/glick/intro-to-hpc'))\n",
    "\n",
    "# We can removie directories too\n",
    "print(os.rmdir('/home/users/glick/intro-to-hpc/dira'))\n",
    "\n",
    "# Finally, list again to see the new directory\n",
    "print(os.listdir('/home/users/glick/intro-to-hpc'))\n",
    "\n",
    "\n",
    "# Using open(), we can read and write files\n",
    "with open(\"/home/users/glick/intro-to-hpc/data/message.txt\", \"w\") as file:\n",
    "    file.write(\"Super Secret Message\")\n",
    "    \n",
    "with open(\"/home/users/glick/intro-to-hpc/data/message.txt\", \"r\") as file:\n",
    "    print(file.read())\n",
    "    \n",
    "# Delete our secret message\n",
    "os.remove(\"/home/users/glick/intro-to-hpc/data/message.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2.1b - Opening Processes\n",
    "For things as universal as filesystem interaction, `os` has built-in functions to help you out. However, there are some things that the Python developers simply can't prepare for. Let's say you want to compile some code, run it, and print the output, all from within Python. This is simple with the command line, and Python's `os` module provides a way to run system commands through the command line, through a function called `os.popen()`. Please remember that this can be unsafe if you allow malicious actors to access your computer, so only use the `os.popen()` function when necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Opening Processes\n",
    "# There is some code in /home/users/glick/intro-to-hpc/data/hello.c\n",
    "\n",
    "import os\n",
    "\n",
    "# Compile the code - There should be no output\n",
    "process = os.popen(\"cc /home/users/glick/intro-to-hpc/data/hello.c\\\n",
    "-o /home/users/glick/intro-to-hpc/data/hello.out -e /dev/null\")\n",
    "print(process.read())\n",
    "\n",
    "# Run the code\n",
    "process = os.popen(\"/home/users/glick/intro-to-hpc/data/hello.out\")\n",
    "print(process.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## II. What is a Resource Manager?\n",
    "You will recall that we defined an HPC system as one made up of smaller constituent systems all working together. However, until now, all of the code we have written has run on just this one computer, which is the login node of a cluster. This is because we have not yet learned to use a _resource manager_. A _resource manager_ is a program that contains both a server, running on a head node, and any number of clients, running on worker nodes. The client allows worker nodes to ask the head node for work, and the server provides jobs to carry out. Almost all clusters have some form of resource manager on them which allows users to submit and monitor jobs to be run on the worker nodes. Most resource managers also have scheduling systems which allow them to run jobs in different orders based on a number of parameters. The following image describes the job flow of Sun GridEngine, a commonly used resource manager: ![SGE architecture](https://2eof2j3oc7is20vt9q3g7tlo5xe-wpengine.netdna-ssl.com/wp-content/uploads/2015/10/Docker_Grid_Engine_R2.jpg)\n",
    "Users submit tasks to a queue, which are then ordered by priority rules set by administrators, and those jobs get run on any available backend resources."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2.2 - Submitting and Monitoring a Simple Resource Manager Job\n",
    "The resource manager we have installed on this system is called GridEngine, commonly abbreviated to SGE. In this example, we will use `os.popen()` and SGE to submit a simple job to the resource manager and monitor it.\n",
    "\n",
    "The SGE interface is based on the `qsub` and `qstat` commands. `qsub` allows users to submit arbitrary shell scripts and `qstat` is used for monitoring jobs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Interacting with the resource manager\n",
    "\n",
    "import os\n",
    "import subprocess\n",
    "import time\n",
    "\n",
    "path = \"/home/users/glick/intro-to-hpc/data/submit_script.sh\"\n",
    "outpath = \"/home/users/glick/intro-to-hpc/data/job_out.out\"\n",
    "\n",
    "os.environ['SGE_ROOT'] = '/local/cluster/sge'\n",
    "os.remove(path)\n",
    "os.remove(outpath)\n",
    "\n",
    "\n",
    "# Write submit scripts\n",
    "with open(path,\"w\") as file:\n",
    "    file.write(\"#!/bin/bash\\n\")\n",
    "    file.write(\"echo h3ll0! >> {}\\n\".format(outpath))\n",
    "    file.write(\"sleep 5\\n\")\n",
    "    file.write(\"echo h3ll0! >> {}\\n\".format(outpath))\n",
    "\n",
    "# Submit job\n",
    "!qsub /home/users/glick/intro-to-hpc/data/submit_script.sh -e /dev/null\n",
    "\n",
    "# List running jobs\n",
    "!qstat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read output file (after waiting for job to complete) - this could take 15 seconds or more\n",
    "with open(outputpath) as file:\n",
    "    print(file.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## III. Programmatic Interaction\n",
    "Sometimes, you don't want to manually submit a job for each task you need to carry out. In this example, we will use a python loop to submit lots of jobs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2.3 - Lots of Jobs!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Submit lots of SGE jobs all at once\n",
    "\n",
    "os.environ['SGE_ROOT'] = '/local/cluster/sge'\n",
    "\n",
    "manyout = \"/home/users/glick/intro-to-hpc/data/manyjob_out.out\"\n",
    "os.remove(manyout)\n",
    "\n",
    "for i in range(1,11):\n",
    "    !echo \"echo $i >> $manyout\" | qsub -e /dev/null"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read output file (after waiting for job to complete) - this could take 15 seconds or more\n",
    "with open(manyout) as file:\n",
    "    print(file.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because this job is running in parallel, each number is not guaranteed to finish in the order in which it started. This is something that always needs to be considered when writing parallel algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IV. Basic Dataflows\n",
    "Most of the time, a simple set of jobs running all at once is not enough for your purposes. Many times, you want to generate lots of different kinds of data all at once, and then perform some kind of summary statistics on it. Situations like this are often referred to as _dataflows_. Dataflows resemble real-world HPC workloads more accurately than the other examples we have looked at in the course so far. The following graphic represents a very simple dataflow to generate random numbers in parallel and take the average.\n",
    "```\n",
    "App Calls    rand() rand() rand()\n",
    "              \\     |     /\n",
    "               a    b    c\n",
    "                \\   |   /\n",
    "App Call        avg_points()\n",
    "                    |\n",
    "                   avg\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2.4 - Array Job Plus Summary Statistics\n",
    "In this example, we will implement the dataflow described in the descriptive text above.\n",
    "We will use Python's `random` library to generate random numbers and print them to a file using SGE, then we will average all of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import os\n",
    "os.environ['SGE_ROOT'] = '/local/cluster/sge'\n",
    "\n",
    "# Submit lots of SGE jobs all at once\n",
    "\n",
    "rand_out = \"/home/users/glick/intro-to-hpc/data/rand_out.out\"\n",
    "os.remove(rand_out)\n",
    "\n",
    "# Generate random numbers\n",
    "for _ in range(1,11):\n",
    "    !echo \"echo $((1 + RANDOM % 100)) >> $rand_out\" | qsub -e /dev/null"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read output file (after waiting for job to complete) - this could take 15 seconds or \n",
    "# Now, we compute the average\n",
    "\n",
    "rand_out = \"/home/users/glick/intro-to-hpc/data/rand_out.out\"\n",
    "\n",
    "with open(rand_out) as file:\n",
    "    lines = [int(i) for i in file.readlines()]\n",
    "    print(sum(lines)//len(lines))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2. Array Dataflow with `qsub`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code goes here"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}