05-Automation.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jovyan/agave\n",
      "Requirement already up-to-date: setvar in /opt/conda/lib/python3.6/site-packages\n",
      "\u001b[1;0mToken for agave.prod:training003 successfully refreshed and cached for 14400 seconds\n",
      "3ab74b5d74f8c336f108416e1b3c9e9\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!mkdir -p ~/agave/funwave-tvd-docker-automation\n",
    "\n",
    "%cd ~/agave\n",
    "\n",
    "!pip3 install --upgrade setvar\n",
    "\n",
    "import re\n",
    "import os\n",
    "import sys\n",
    "from setvar import *\n",
    "from time import sleep\n",
    "\n",
    "# This cell enables inline plotting in the notebook\n",
    "%matplotlib inline\n",
    "\n",
    "import matplotlib\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "!auth-tokens-refresh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Automating image building\n",
    "\n",
    "We saw in the last notebook how we can build images of our funwave-tvd code and use Agave to make the process a bit easier. We can take some lessons learned from the devops community to automate the building of our images and implement basic benchmarking and testing. \n",
    "\n",
    "While the Agave fork app we created is handy, it doesn't provide particularly good visibility, let along security. We certainly do not want to share an app like that for others to use. So, let's start start by creating a new Agave app that will build our Docker container. First up, our updated app assets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating more meaningful Dockerfiles\n",
    "\n",
    "While functional, our previous Dockerfile didn't give us much info we could use for things like attribution, discovery, etc. Let's add in some additional fields to give our Dockerfile meaning. We will use a couple new Dockerfile directives to do this\n",
    "\n",
    "ARG a runtime argument supplied to the `docker build` command\n",
    "\n",
    "LABEL one or more terms applied to the image as metadata\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing file `funwave-tvd-docker-automation/Dockerfile'\n"
     ]
    }
   ],
   "source": [
    "writefile(\"funwave-tvd-docker-automation/Dockerfile\",\"\"\"\n",
    "FROM stevenrbrandt/science-base\n",
    "MAINTAINER Steven R. Brandt <sbrandt@cct.lsu.edu>\n",
    "\n",
    "ARG BUILD_DATE\n",
    "ARG VERSION\n",
    "\n",
    "LABEL org.agaveplatform.ax.architecture=\"x86_64\"                                \\\n",
    "      org.agaveplatform.ax.build-date=\"\\$BUILD_DATE\"                             \\\n",
    "      org.agaveplatform.ax.version=\"\\$VERSION\"                             \\\n",
    "      org.agaveplatform.ax.name=\"${AGAVE_USERNAME}/funwave-tvd\"    \\\n",
    "      org.agaveplatform.ax.summary=\"Funwave-TVD is a code to simulate the shallow water and Boussinesq equations written by Dr. Fengyan Shi.\" \\\n",
    "      org.agaveplatform.ax.vcs-type=\"git\"                                       \\\n",
    "      org.agaveplatform.ax.vcs-url=\"https://github.com/fengyanshi/FUNWAVE-TVD\" \\\n",
    "      org.agaveplatform.ax.license=\"BSD 3-clause\"\n",
    "      \n",
    "USER root\n",
    "RUN mkdir -p /home/install\n",
    "RUN chown jovyan /home/install\n",
    "USER jovyan\n",
    "\n",
    "RUN cd /home/install && \\\n",
    "    git clone https://github.com/fengyanshi/FUNWAVE-TVD && \\\n",
    "    cd FUNWAVE-TVD/src && \\\n",
    "    perl -p -i -e 's/FLAG_8 = -DCOUPLING/#$&/' Makefile && \\\n",
    "    make\n",
    "\n",
    "WORKDIR /home/install/FUNWAVE-TVD/src\n",
    "RUN mkdir -p /home/jovyan/rundir\n",
    "WORKDIR /home/jovyan/rundir\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Single purpose wrapper scripts\n",
    "\n",
    "In our previous wrapper script, we simply took whatever was given to us an ran it. Here we will restrict the wrapper to run a specific build comman.  \n",
    " \n",
    "> Note that we mix an match a couple variable types. The `Agave_*` variable are template variables resolved by Agave at runtime with the vales from the job etails. The `version` variable is a parameter we will define in our app description.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing file `funwave-tvd-docker-automation/funwave-build-wrapper.txt'\n"
     ]
    }
   ],
   "source": [
    "writefile(\"funwave-tvd-docker-automation/funwave-build-wrapper.txt\",\"\"\"\n",
    "\n",
    "sudo docker build \\\n",
    "    --build-arg \"BUILD_DATE=\\${AGAVE_JOB_SUBMIT_TIME}\" \\\n",
    "    --build-arg \"VERSION=\\${code_version}\" \\\n",
    "    --rm -t funwave-tvd:\\${code_version} .\n",
    "\n",
    "docker inspect funwave-tvd:\\${code_version}\n",
    "\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## More descriptive apps  \n",
    "\n",
    "Now we need to create some JSON to tell Agave how to run and advertise our app. This app definition will look a lot like the fork app definition with a few changes. First, we are updating the app id so a new app will be created. Second we change the parameter.  \n",
    "\n",
    "* `code_version` is a string parameter describing the version of the code.   \n",
    "\n",
    "We have also removed the data file input from the previous app description. This is because our deployment folder contains the Dockerfile to build our image. No other info is needed to run our build app."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing file `funwave-tvd-docker-automation/funwave-build-app.txt'\n"
     ]
    }
   ],
   "source": [
    "writefile(\"funwave-tvd-docker-automation/funwave-build-app.txt\",\"\"\"\n",
    "{  \n",
    "   \"name\":\"${AGAVE_USERNAME}-${MACHINE_NAME}-funwave-dbuild\",\n",
    "   \"version\":\"1.0\",\n",
    "   \"label\":\"Builds the funwave docker image\",\n",
    "   \"shortDescription\":\"Funwave docker build\",\n",
    "   \"longDescription\":\"\",\n",
    "   \"deploymentSystem\":\"${AGAVE_STORAGE_SYSTEM_ID}\",\n",
    "   \"deploymentPath\":\"automation/funwave-tvd-docker-automation\",\n",
    "   \"templatePath\":\"funwave-build-wrapper.txt\",\n",
    "   \"testPath\":\"test.txt\",\n",
    "   \"executionSystem\":\"${AGAVE_EXECUTION_SYSTEM_ID}\",\n",
    "   \"executionType\":\"CLI\",\n",
    "   \"parallelism\":\"SERIAL\",\n",
    "   \"modules\":[],\n",
    "   \"inputs\":[],\n",
    "   \"parameters\":[{\n",
    "     \"id\" : \"code_version\",\n",
    "     \"value\" : {\n",
    "       \"visible\":true,\n",
    "       \"required\":true,\n",
    "       \"type\":\"string\",\n",
    "       \"order\":0,\n",
    "       \"enquote\":false,\n",
    "       \"default\":\"latest\"\n",
    "     },\n",
    "     \"details\":{\n",
    "         \"label\": \"Version of the code\",\n",
    "         \"description\": \"If true, output will be packed and compressed\",\n",
    "         \"argument\": null,\n",
    "         \"showArgument\": false,\n",
    "         \"repeatArgument\": false\n",
    "     },\n",
    "     \"semantics\":{\n",
    "         \"argument\": null,\n",
    "         \"showArgument\": false,\n",
    "         \"repeatArgument\": false\n",
    "     }\n",
    "   }],\n",
    "   \"outputs\":[]\n",
    "}\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is our default test file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing file `funwave-tvd-docker-automation/test.txt'\n"
     ]
    }
   ],
   "source": [
    "writefile(\"funwave-tvd-docker-automation/test.txt\",\"\"\"\n",
    "code_version=latest\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;0mSuccessfully created folder automation\u001b[0m\n",
      "Creating directory automation/funwave-tvd-docker-automation ...\n",
      "Uploading funwave-tvd-docker-automation/Dockerfile...\n",
      "######################################################################## 100.0%\n",
      "Uploading funwave-tvd-docker-automation/funwave-build-app.txt...\n",
      "######################################################################## 100.0%\n",
      "Uploading funwave-tvd-docker-automation/funwave-build-wrapper.txt...\n",
      "######################################################################## 100.0%\n",
      "Uploading funwave-tvd-docker-automation/job.json...\n",
      "######################################################################## 100.0%\n",
      "Uploading funwave-tvd-docker-automation/test.txt...\n",
      "######################################################################## 100.0%\n"
     ]
    }
   ],
   "source": [
    "!files-mkdir -S ${AGAVE_STORAGE_SYSTEM_ID} -N automation\n",
    "!files-upload -S ${AGAVE_STORAGE_SYSTEM_ID} -F funwave-tvd-docker-automation automation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;0m\u001b[1;0mSuccessfully added app training003-sandbox-funwave-dbuild-1.0\u001b[0m\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!apps-addupdate -F funwave-tvd-docker-automation/funwave-build-app.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Running the build"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WEBHOOK_URL=https://requestbin.agaveapi.co/126tsd91\n"
     ]
    }
   ],
   "source": [
    "requestbin_url = !requestbin-create\n",
    "os.environ['REQUESTBIN_URL'] = requestbin_url[0]\n",
    "setvar(\"\"\"\n",
    "WEBHOOK_URL=${REQUESTBIN_URL}\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll run our build using the following job request. This is very similar to before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing file `funwave-tvd-docker-automation/job.json'\n"
     ]
    }
   ],
   "source": [
    "writefile(\"funwave-tvd-docker-automation/job.json\",\"\"\"\n",
    " {\n",
    "   \"name\":\"funwave-build\",\n",
    "   \"appId\": \"${AGAVE_USERNAME}-${MACHINE_NAME}-funwave-dbuild-1.0\",\n",
    "   \"maxRunTime\":\"00:10:00\",\n",
    "   \"archive\": false,\n",
    "   \"notifications\": [\n",
    "    {\n",
    "      \"url\":\"${WEBHOOK_URL}\",\n",
    "      \"event\":\"*\",\n",
    "      \"persistent\":\"true\"\n",
    "    }\n",
    "   ],\n",
    "   \"parameters\": {\n",
    "     \"code_version\":\"latest\"\n",
    "   }\n",
    " }\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the setvar() command can evalute `$()` style bash shell substitutions, we will use it to submit our job. This will capture the output of the submit command, and allow us to parse it for the JOB_ID. We'll use the JOB_ID in several subsequent steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OUTPUT=Successfully submitted job 7446628594819600871-242ac11b-0001-007\n",
      "JOB_ID=7446628594819600871-242ac11b-0001-007\n"
     ]
    }
   ],
   "source": [
    "setvar(\"\"\"\n",
    "# Capture the output of the job submit command\n",
    "OUTPUT=$(jobs-submit -F funwave-tvd-docker-automation/job.json)\n",
    "# Parse out the job id from the output\n",
    "JOB_ID=$(echo $OUTPUT | cut -d' ' -f4)\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "STAT=PENDING\n",
      "STAT=PENDING\n",
      "STAT=PENDING\n",
      "STAT=STAGED\n",
      "STAT=SUBMITTING\n",
      "STAT=SUBMITTING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=FINISHED\n"
     ]
    }
   ],
   "source": [
    "for iter in range(20):\n",
    "    setvar(\"STAT=$(jobs-status $JOB_ID)\")\n",
    "    stat = os.environ[\"STAT\"]\n",
    "    sleep(5.0)\n",
    "    if stat == \"FINISHED\" or stat == \"FAILED\":\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sending build context to Docker daemon  15.87kB\r",
      "\r",
      "\r\n",
      "Step 1/13 : FROM stevenrbrandt/science-base\r\n",
      " ---> e93db47f971c\r\n",
      "Step 2/13 : MAINTAINER Steven R. Brandt <sbrandt@cct.lsu.edu>\r\n",
      " ---> Using cache\r\n",
      " ---> 13f7ea48e9cb\r\n",
      "Step 3/13 : ARG BUILD_DATE\r\n",
      " ---> Using cache\r\n",
      " ---> 548f7a1528d7\r\n",
      "Step 4/13 : ARG VERSION\r\n",
      " ---> Using cache\r\n",
      " ---> fdf915eadde3\r\n",
      "Step 5/13 : LABEL org.agaveplatform.ax.architecture \"x86_64\" org.agaveplatform.ax.build-date \"$BUILD_DATE\" org.agaveplatform.ax.version \"$VERSION\" org.agaveplatform.ax.name \"training003/funwave-tvd\" org.agaveplatform.ax.summary \"Funwave-TVD is a code to simulate the shallow water and Boussinesq equations written by Dr. Fengyan Shi.\" org.agaveplatform.ax.vcs-type \"git\" org.agaveplatform.ax.vcs-url \"https://github.com/fengyanshi/FUNWAVE-TVD\" org.agaveplatform.ax.license \"BSD 3-clause\"\r\n",
      " ---> Running in 96e93ee95458\r\n",
      " ---> d68fc3ac3cb6\r\n",
      "Removing intermediate container 96e93ee95458\r\n",
      "Step 6/13 : USER root\r\n",
      " ---> Running in bbd4a5e10730\r\n",
      " ---> 70866cf2dc8c\r\n",
      "Removing intermediate container bbd4a5e10730\r\n",
      "Step 7/13 : RUN mkdir -p /home/install\r\n",
      " ---> Running in 28094381b957\r\n",
      " ---> 19e36b026cc3\r\n",
      "Removing intermediate container 28094381b957\r\n",
      "Step 8/13 : RUN chown jovyan /home/install\r\n",
      " ---> Running in 38c928b7b45e\r\n",
      " ---> f8a893036dc5\r\n",
      "Removing intermediate container 38c928b7b45e\r\n",
      "Step 9/13 : USER jovyan\r\n",
      " ---> Running in 25180bf3086c\r\n",
      " ---> 13c32aac945e\r\n",
      "Removing intermediate container 25180bf3086c\r\n",
      "Step 10/13 : RUN cd /home/install &&     git clone https://github.com/fengyanshi/FUNWAVE-TVD &&     cd FUNWAVE-TVD/src &&     perl -p -i -e 's/FLAG_8 = -DCOUPLING/#$&/' Makefile &&     make\r\n",
      " ---> Running in 57c1021d18a0\r\n",
      "\u001b[91mCloning into 'FUNWAVE-TVD'...\r\n",
      "\u001b[0m/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     mod_param.F > mod_param.f90\r\n",
      "mpif90  -c     mod_param.f90\r\n",
      "/bin/rm mod_param.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     mod_global.F > mod_global.f90\r\n",
      "mpif90  -c     mod_global.f90\r\n",
      "/bin/rm mod_global.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     mod_input.F > mod_input.f90\r\n",
      "mpif90  -c     mod_input.f90\r\n",
      "/bin/rm mod_input.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     mod_vessel.F > mod_vessel.f90\r\n",
      "mpif90  -c     mod_vessel.f90\r\n",
      "/bin/rm mod_vessel.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     mod_bathy_correction.F > mod_bathy_correction.f90\r\n",
      "mpif90  -c     mod_bathy_correction.f90\r\n",
      "/bin/rm mod_bathy_correction.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     mod_meteo.F > mod_meteo.f90\r\n",
      "mpif90  -c     mod_meteo.f90\r\n",
      "/bin/rm mod_meteo.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     mod_parallel_field_io.F > mod_parallel_field_io.f90\r\n",
      "mpif90  -c     mod_parallel_field_io.f90\r\n",
      "/bin/rm mod_parallel_field_io.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     main.F > main.f90\r\n",
      "mpif90  -c     main.f90\r\n",
      "/bin/rm main.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     bc.F > bc.f90\r\n",
      "mpif90  -c     bc.f90\r\n",
      "/bin/rm bc.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     fluxes.F > fluxes.f90\r\n",
      "mpif90  -c     fluxes.f90\r\n",
      "/bin/rm fluxes.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     init.F > init.f90\r\n",
      "mpif90  -c     init.f90\r\n",
      "/bin/rm init.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     io.F > io.f90\r\n",
      "mpif90  -c     io.f90\r\n",
      "/bin/rm io.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     tridiagnal.F > tridiagnal.f90\r\n",
      "mpif90  -c     tridiagnal.f90\r\n",
      "/bin/rm tridiagnal.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     breaker.F > breaker.f90\r\n",
      "mpif90  -c     breaker.f90\r\n",
      "/bin/rm breaker.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     derivatives.F > derivatives.f90\r\n",
      "mpif90  -c     derivatives.f90\r\n",
      "/bin/rm derivatives.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     dispersion.F > dispersion.f90\r\n",
      "mpif90  -c     dispersion.f90\r\n",
      "/bin/rm dispersion.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     etauv_solver.F > etauv_solver.f90\r\n",
      "mpif90  -c     etauv_solver.f90\r\n",
      "/bin/rm etauv_solver.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     sponge.F > sponge.f90\r\n",
      "mpif90  -c     sponge.f90\r\n",
      "/bin/rm sponge.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     sources.F > sources.f90\r\n",
      "mpif90  -c     sources.f90\r\n",
      "/bin/rm sources.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     masks.F > masks.f90\r\n",
      "mpif90  -c     masks.f90\r\n",
      "/bin/rm masks.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     parallel.F > parallel.f90\r\n",
      "mpif90  -c     parallel.f90\r\n",
      "/bin/rm parallel.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     statistics.F > statistics.f90\r\n",
      "mpif90  -c     statistics.f90\r\n",
      "/bin/rm statistics.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     wavemaker.F > wavemaker.f90\r\n",
      "mpif90  -c     wavemaker.f90\r\n",
      "/bin/rm wavemaker.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     mixing.F > mixing.f90\r\n",
      "mpif90  -c     mixing.f90\r\n",
      "/bin/rm mixing.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     nesting.F > nesting.f90\r\n",
      "mpif90  -c     nesting.f90\r\n",
      "/bin/rm nesting.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     misc.F > misc.f90\r\n",
      "mpif90  -c     misc.f90\r\n",
      "/bin/rm misc.f90\r\n",
      "/usr/bin/cpp  -P -traditional  -P -traditional  -DDOUBLE_PRECISION  -DPARALLEL  -DCARTESIAN                     samples.F > samples.f90\r\n",
      "mpif90  -c     samples.f90\r\n",
      "/bin/rm samples.f90\r\n",
      "mpif90     -o funwave_vessel mod_param.o mod_global.o mod_input.o mod_vessel.o mod_bathy_correction.o mod_meteo.o mod_parallel_field_io.o main.o bc.o fluxes.o init.o io.o tridiagnal.o breaker.o derivatives.o dispersion.o etauv_solver.o sponge.o sources.o masks.o parallel.o statistics.o wavemaker.o mixing.o nesting.o misc.o samples.o \r\n",
      " ---> e9aeea2d9be8\r\n",
      "Removing intermediate container 57c1021d18a0\r\n",
      "Step 11/13 : WORKDIR /home/install/FUNWAVE-TVD/src\r\n",
      " ---> c080c05c2288\r\n",
      "Removing intermediate container 89912920df55\r\n",
      "Step 12/13 : RUN mkdir -p /home/jovyan/rundir\r\n",
      " ---> Running in a4ba7803d059\r\n",
      " ---> 48c6797449d5\r\n",
      "Removing intermediate container a4ba7803d059\r\n",
      "Step 13/13 : WORKDIR /home/jovyan/rundir\r\n",
      " ---> 3aec8b3a59ef\r\n",
      "Removing intermediate container 068c2f8d1901\r\n",
      "Successfully built 3aec8b3a59ef\r\n",
      "Successfully tagged funwave-tvd:latest\r\n",
      "[]\r\n"
     ]
    }
   ],
   "source": [
    "!jobs-output-get -P ${JOB_ID} funwave-build.out\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "To rebuild our Docker image, we can rerun our job submission command, or simply resumbit the previous job. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OUTPUT=Successfully resubmitted job 7446628594819600871-242ac11b-0001-007 as 495140868949470745-242ac11b-0001-007\n",
      "JOB_ID=7446628594819600871-242ac11b-0001-007\n"
     ]
    }
   ],
   "source": [
    "setvar(\"\"\"\n",
    "# Capture the output of the job submit command\n",
    "OUTPUT=$(jobs-resubmit ${JOB_ID})\n",
    "# Parse out the job id from the output\n",
    "JOB_ID=$(echo $OUTPUT | cut -d' ' -f4)\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Adding in automation  \n",
    "\n",
    "The reason we go through the trouble of defining a build app, is so we can use it as a tool in our automation process. We can create a simple integration to receive webhooks from github to run our build job on every commit. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}