Distributed-Scoring-Tensorflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distributed Scoring Demo\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instructions\n",
    "\n",
    "This example demonstrate how to run distributed Scoring demo on Azure Batch AI cluster.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Install Dependencies and Create Configuration file.\n",
    "Follow [instructions](/recipes) to install all dependencies and create configuration file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read Configuration and Create Batch AI client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "from datetime import datetime\n",
    "import os\n",
    "import sys\n",
    "\n",
    "from azure.storage.file import FileService\n",
    "from azure.storage.blob import BlockBlobService\n",
    "import azure.mgmt.batchai.models as models\n",
    "\n",
    "# utilities.py contains helper functions used by different notebooks\n",
    "import utilities\n",
    "\n",
    "cfg = utilities.Configuration('configuration.json')\n",
    "client = utilities.create_batchai_client(cfg)\n",
    "utilities.create_resource_group(cfg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Prepare Training Dataset and Script in Azure Storage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Azure Blob Container\n",
    "\n",
    "We will create a new Blob Container with name `batchaisample` under your storage account. This will be used to store the *input training dataset*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "azure_blob_container_name = 'batchaisample'\n",
    "BlockBlobService(cfg.storage_account_name, cfg.storage_account_key).create_container(azure_blob_container_name, fail_on_exist=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Azure File Share\n",
    "\n",
    "For this example we will create a new File Share with name `batchaisample` under your storage account. This will be used to share the *training script file* and *output file*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "azure_file_share_name = 'abc'\n",
    "file_service = FileService(cfg.storage_account_name, cfg.storage_account_key)\n",
    "file_service.create_share(azure_file_share_name, fail_on_exist=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Create Azure Batch AI Compute Cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Compute Cluster\n",
    "- For this example we will use a gpu cluster of 2 `STANDARD_NC6` nodes. You can increase the number of nodes by changing `nodes_count` variable;\n",
    "- We will mount file share at folder with name `external`. Full path of this folder on a computer node will be `$AZ_BATCHAI_MOUNT_ROOT/external`;\n",
    "- We will call the cluster `nc6`;\n",
    "\n",
    "So, the cluster will have the following parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "azure_file_share = 'external_AFS'\n",
    "azure_blob = 'external_ABFS'\n",
    "\n",
    "nodes_count = 8\n",
    "cluster_name = 'nc24sv2cx'\n",
    "\n",
    "volumes = models.MountVolumes(\n",
    "    azure_file_shares=[\n",
    "        models.AzureFileShareReference(\n",
    "            account_name=cfg.storage_account_name,\n",
    "            credentials=models.AzureStorageCredentialsInfo(\n",
    "                account_key=cfg.storage_account_key),\n",
    "            azure_file_url='https://{0}.file.core.windows.net/{1}'.format(\n",
    "                cfg.storage_account_name, azure_file_share_name),\n",
    "            relative_mount_path=azure_file_share)\n",
    "    ],\n",
    "    azure_blob_file_systems=[\n",
    "        models.AzureBlobFileSystemReference(\n",
    "            account_name=cfg.storage_account_name,\n",
    "            credentials=models.AzureStorageCredentialsInfo(\n",
    "                account_key=cfg.storage_account_key),\n",
    "            container_name=azure_blob_container_name,\n",
    "            relative_mount_path=azure_blob)\n",
    "    ]\n",
    ")\n",
    "\n",
    "parameters = models.ClusterCreateParameters(\n",
    "    location=cfg.location,\n",
    "    vm_size=\"STANDARD_NC24S_V2\",\n",
    "    scale_settings=models.ScaleSettings(\n",
    "        manual=models.ManualScaleSettings(target_node_count=nodes_count)\n",
    "    ),\n",
    "    node_setup=models.NodeSetup(\n",
    "        mount_volumes=volumes,\n",
    "    ),\n",
    "    user_account_settings=models.UserAccountSettings(\n",
    "        admin_user_name=cfg.admin,\n",
    "        admin_user_password=cfg.admin_password,\n",
    "        admin_user_ssh_public_key=cfg.admin_ssh_key\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Compute Cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cluster = client.clusters.create(cfg.resource_group, cluster_name, parameters).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitor Cluster Creation\n",
    "\n",
    "Monitor the just created cluster. utilities.py contains a helper function to print out detail status of the cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cluster = client.clusters.get(cfg.resource_group, cluster_name)\n",
    "utilities.print_cluster_status(cluster)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Run Batch AI Job for Cloth Recognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Update inference script in Azure File Share if need\n",
    "\n",
    "Just in case you modified the inference script, you need to update the local modified script to cloud."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "script_to_deploy = 'dist_inference.py'\n",
    "file_service.create_file_from_path(\n",
    "    azure_file_share_name, '', script_to_deploy, script_to_deploy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Job\n",
    "\n",
    "- The job will use `tensorflow/tensorflow:1.4.0-gpu` container.\n",
    "- Will use job preparation to install dependencies in container\n",
    "- Will store console log and output label of the job in File Share:\n",
    "- Will use configured previously input and output directories.\n",
    "- The job will be able to reference those directories using environment variables:\n",
    "    - AZ_BATCHAI_INPUT_ROOT : refers to the mounted path of Azure File Share, where pretained model and scripts locate\n",
    "    - AZ_BATCHAI_INPUT_DATASET : refers to the mounted path of Azure Container Blob, where image dataset locate\n",
    "- Will use custom toolkit to start OpenMPI cross workers\n",
    "- Specify which inference task in command line 1) `cloth_recognition` or 2) `classification`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "inference_task = 'cloth_recognition'\n",
    "parameters = models.job_create_parameters.JobCreateParameters(\n",
    "     location=cfg.location,\n",
    "     cluster=models.ResourceId(cluster.id),\n",
    "     node_count=8,\n",
    "     input_directories=[\n",
    "         models.InputDirectory(\n",
    "             id='DATASET',\n",
    "             path='$AZ_BATCHAI_MOUNT_ROOT/{0}/{1}'.format(azure_blob, 'images')),\n",
    "         models.InputDirectory(\n",
    "             id='ROOT',\n",
    "             path='$AZ_BATCHAI_MOUNT_ROOT/{0}'.format(azure_file_share))],\n",
    "     output_directories = [\n",
    "         models.OutputDirectory(\n",
    "             id='LABEL',\n",
    "             path_prefix='$AZ_BATCHAI_MOUNT_ROOT/{0}'.format(azure_file_share),\n",
    "             path_suffix=\"Label\")],\n",
    "     std_out_err_path_prefix=\"$AZ_BATCHAI_MOUNT_ROOT/{0}\".format(azure_file_share),\n",
    "     job_preparation=models.JobPreparation(\n",
    "        command_line=\"pip install opencv-python EasyDict pyyaml;apt update && apt install -y libsm6 libxext6 libxrender-dev mpi-default-dev mpi-default-bin\"),\n",
    "     container_settings=models.ContainerSettings(\n",
    "         models.ImageSourceRegistry(image='tensorflow/tensorflow:1.4.0-gpu')),\n",
    "     custom_toolkit_settings = models.CustomToolkitSettings(\n",
    "         command_line='mpirun -mca btl_tcp_if_exclude docker0,lo --allow-run-as-root --hostfile $AZ_BATCHAI_MPI_HOST_FILE python -u $AZ_BATCHAI_INPUT_ROOT/dist_inference.py --datadir $AZ_BATCHAI_INPUT_DATASET --outputdir $AZ_BATCHAI_OUTPUT_LABEL --rootdir $AZ_BATCHAI_INPUT_ROOT --max 100000 --inference {0}'.format(inference_task))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a training Job and wait for Job completion\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created Job: cloth_recognition_03_12_2018_073609\n"
     ]
    }
   ],
   "source": [
    "job_name = inference_task + datetime.utcnow().strftime(\"_%m_%d_%Y_%H%M%S\")\n",
    "job = client.jobs.create(cfg.resource_group, job_name, parameters).result()\n",
    "print('Created Job: {}'.format(job_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wait for Job to Finish\n",
    "The job will start running when the cluster will have enought idle nodes. The following code waits for job to start running printing the cluster state. During job run, the code prints current content of stdeout-0.txt (the output of the worker running on the first node)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cluster state: AllocationState.steady Target: 8; Allocated: 8; Idle: 0; Unusable: 0; Running: 8; Preparing: 0; Leaving: 0\n",
      "Job state: running ExitCode: None\n",
      "Waiting for job output to become available...\n",
      "Split dataset based on given task 28\n",
      "Split dataset based on given task 30\n",
      "Split dataset based on given task 29\n",
      "Split dataset based on given task 31\n",
      "Worker 31 needs to process 3124 images\n",
      "Worker 29 needs to process 3124 images\n",
      "Worker 28 needs to process 3124 images\n",
      "Worker 30 needs to process 3124 images\n",
      "Split dataset based on given task 12\n",
      "Split dataset based on given task 14\n",
      "Split dataset based on given task 13\n",
      "Split dataset based on given task 15\n",
      "Worker 15 needs to process 3124 images\n",
      "Worker 13 needs to process 3124 images\n",
      "Worker 12 needs to process 3124 images\n",
      "Worker 14 needs to process 3124 images\n",
      "Split dataset based on given task 24\n",
      "Split dataset based on given task 25\n",
      "Split dataset based on given task 26\n",
      "Split dataset based on given task 4\n",
      "Split dataset based on given task 5\n",
      "Split dataset based on given task 6\n",
      "Split dataset based on given task 7\n",
      "Worker 25 needs to process 3124 images\n",
      "Worker 26 needs to process 3124 images\n",
      "Worker 24 needs to process 3124 images\n",
      "Split dataset based on given task 9\n",
      "Split dataset based on given task 11\n",
      "Worker 5 needs to process 3124 images\n",
      "Split dataset based on given task 10\n",
      "Worker 7 needs to process 3124 images\n",
      "Split dataset based on given task 8\n",
      "Worker 6 needs to process 3124 images\n",
      "Worker 4 needs to process 3124 images\n",
      "Split dataset based on given task 27\n",
      "Worker 27 needs to process 3124 images\n",
      "Worker 8 needs to process 3124 images\n",
      "Worker 10 needs to process 3124 images\n",
      "Worker 11 needs to process 3124 images\n",
      "Worker 9 needs to process 3124 images\n",
      "Split dataset based on given task 0\n",
      "Split dataset based on given task 1\n",
      "Split dataset based on given task 2\n",
      "Split dataset based on given task 3\n",
      "Split dataset based on given task 22\n",
      "Split dataset based on given task 20\n",
      "Split dataset based on given task 23\n",
      "Split dataset based on given task 21\n",
      "Split dataset based on given task 16\n",
      "Split dataset based on given task 17\n",
      "Split dataset based on given task 19\n",
      "Split dataset based on given task 18\n",
      "Worker 22 needs to process 3124 images\n",
      "Worker 23 needs to process 3124 images\n",
      "Worker 21 needs to process 3124 images\n",
      "Worker 20 needs to process 3124 images\n",
      "Worker 3 needs to process 3124 images\n",
      "Worker 1 needs to process 3124 images\n",
      "Worker 19 needs to process 3124 images\n",
      "Worker 16 needs to process 3124 images\n",
      "Worker 17 needs to process 3124 images\n",
      "Worker 2 needs to process 3124 images\n",
      "Worker 0 needs to process 3124 images\n",
      "Worker 18 needs to process 3124 images\n",
      "Worker 30 Processed 100/3124 images \n",
      "Worker 28 Processed 100/3124 images \n",
      "Worker 31 Processed 100/3124 images \n",
      "Worker 29 Processed 100/3124 images \n",
      "Worker 26 Processed 100/3124 images \n",
      "Worker 25 Processed 100/3124 images \n",
      "Worker 13 Processed 100/3124 images \n",
      "Worker 27 Processed 100/3124 images \n",
      "Worker 24 Processed 100/3124 images \n",
      "Worker 9 Processed 100/3124 images \n",
      "Worker 15 Processed 100/3124 images \n",
      "Worker 16 Processed 100/3124 images \n",
      "Worker 10 Processed 100/3124 images \n",
      "Worker 5 Processed 100/3124 images \n",
      "Worker 21 Processed 100/3124 images \n",
      "Worker 8 Processed 100/3124 images \n",
      "Worker 20 Processed 100/3124 images \n",
      "Worker 22 Processed 100/3124 images \n",
      "Worker 19 Processed 100/3124 images \n",
      "Worker 23 Processed 100/3124 images \n",
      "Worker 28 Processed 200/3124 images \n",
      "Worker 31 Processed 200/3124 images \n",
      "Worker 29 Processed 200/3124 images \n",
      "Worker 30 Processed 200/3124 images \n",
      "Worker 7 Processed 200/3124 images \n",
      "Worker 2 Processed 100/3124 images \n",
      "Worker 26 Processed 200/3124 images \n",
      "Worker 25 Processed 200/3124 images \n",
      "Worker 16 Processed 200/3124 images \n",
      "Worker 15 Processed 200/3124 images \n",
      "Worker 9 Processed 200/3124 images \n",
      "Worker 14 Processed 200/3124 images \n",
      "Worker 0 Processed 100/3124 images \n",
      "Worker 18 Processed 200/3124 images \n",
      "Worker 10 Processed 200/3124 images \n",
      "Worker 20 Processed 200/3124 images \n",
      "Worker 8 Processed 200/3124 images \n",
      "Worker 17 Processed 200/3124 images \n",
      "Worker 19 Processed 200/3124 images \n",
      "Worker 22 Processed 200/3124 images \n",
      "Worker 23 Processed 200/3124 images \n",
      "Worker 26 Processed 300/3124 images \n",
      "Worker 25 Processed 300/3124 images \n",
      "Worker 28 Processed 300/3124 images \n",
      "Worker 29 Processed 300/3124 images \n",
      "Worker 27 Processed 300/3124 images \n",
      "Worker 30 Processed 300/3124 images \n",
      "Worker 24 Processed 300/3124 images \n",
      "Worker 31 Processed 300/3124 images \n",
      "Worker 5 Processed 300/3124 images \n",
      "Worker 1 Processed 200/3124 images \n",
      "Worker 4 Processed 300/3124 images \n",
      "Worker 9 Processed 300/3124 images \n",
      "Worker 18 Processed 300/3124 images \n",
      "Worker 10 Processed 300/3124 images \n",
      "Worker 2 Processed 200/3124 images \n",
      "Worker 8 Processed 300/3124 images \n",
      "Worker 3 Processed 200/3124 images \n",
      "Worker 11 Processed 300/3124 images \n",
      "Worker 0 Processed 200/3124 images \n",
      "Worker 23 Processed 300/3124 images \n",
      "Worker 26 Processed 400/3124 images \n",
      "Worker 25 Processed 400/3124 images \n",
      "Worker 7 Processed 400/3124 images \n",
      "Worker 29 Processed 400/3124 images \n",
      "Worker 24 Processed 400/3124 images \n",
      "Worker 28 Processed 400/3124 images \n",
      "Worker 27 Processed 400/3124 images \n",
      "Worker 30 Processed 400/3124 images \n",
      "Worker 4 Processed 400/3124 images \n",
      "Worker 13 Processed 400/3124 images \n",
      "Worker 18 Processed 400/3124 images \n",
      "Worker 19 Processed 400/3124 images \n",
      "Worker 10 Processed 400/3124 images \n",
      "Worker 22 Processed 400/3124 images \n",
      "Worker 21 Processed 400/3124 images \n",
      "Worker 8 Processed 400/3124 images \n",
      "Worker 11 Processed 400/3124 images \n",
      "Worker 23 Processed 400/3124 images \n",
      "Worker 26 Processed 500/3124 images \n",
      "Worker 25 Processed 500/3124 images \n",
      "Worker 29 Processed 500/3124 images \n",
      "Worker 0 Processed 300/3124 images \n",
      "Worker 5 Processed 500/3124 images \n",
      "Worker 4 Processed 500/3124 images \n",
      "Worker 27 Processed 500/3124 images \n",
      "Worker 31 Processed 500/3124 images \n",
      "Worker 13 Processed 500/3124 images \n",
      "Worker 30 Processed 500/3124 images \n",
      "Worker 9 Processed 500/3124 images \n",
      "Worker 14 Processed 500/3124 images \n",
      "Worker 15 Processed 500/3124 images \n",
      "Worker 17 Processed 500/3124 images \n",
      "Worker 19 Processed 500/3124 images \n",
      "Worker 22 Processed 500/3124 images \n",
      "Worker 10 Processed 500/3124 images \n",
      "Worker 21 Processed 500/3124 images \n",
      "Worker 1 Processed 400/3124 images \n",
      "Worker 8 Processed 500/3124 images \n",
      "Worker 26 Processed 600/3124 images \n",
      "Worker 25 Processed 600/3124 images \n",
      "Worker 7 Processed 600/3124 images \n",
      "Worker 23 Processed 500/3124 images \n",
      "Worker 29 Processed 600/3124 images \n",
      "Worker 4 Processed 600/3124 images \n",
      "Worker 28 Processed 600/3124 images \n",
      "Worker 3 Processed 400/3124 images \n",
      "Worker 27 Processed 600/3124 images \n",
      "Worker 31 Processed 600/3124 images \n",
      "Worker 30 Processed 600/3124 images \n",
      "Worker 9 Processed 600/3124 images \n",
      "Worker 17 Processed 600/3124 images \n",
      "Worker 14 Processed 600/3124 images \n",
      "Worker 15 Processed 600/3124 images \n",
      "Worker 19 Processed 600/3124 images \n",
      "Worker 0 Processed 400/3124 images \n",
      "Worker 22 Processed 600/3124 images \n",
      "Worker 8 Processed 600/3124 images \n",
      "Worker 11 Processed 600/3124 images \n",
      "Worker 16 Processed 700/3124 images \n",
      "Worker 1 Processed 500/3124 images \n",
      "Worker 26 Processed 700/3124 images \n",
      "Worker 25 Processed 700/3124 images \n",
      "Worker 29 Processed 700/3124 images \n",
      "Worker 24 Processed 700/3124 images \n",
      "Worker 20 Processed 700/3124 images \n",
      "Worker 28 Processed 700/3124 images \n",
      "Worker 27 Processed 700/3124 images \n",
      "Worker 31 Processed 700/3124 images \n",
      "Worker 9 Processed 700/3124 images \n",
      "Worker 30 Processed 700/3124 images \n",
      "Worker 2 Processed 500/3124 images \n",
      "Worker 14 Processed 700/3124 images \n",
      "Worker 19 Processed 700/3124 images \n",
      "Worker 15 Processed 700/3124 images \n",
      "Worker 21 Processed 700/3124 images \n",
      "Worker 3 Processed 500/3124 images \n",
      "Worker 10 Processed 700/3124 images \n",
      "Worker 8 Processed 700/3124 images \n",
      "Worker 11 Processed 700/3124 images \n",
      "Worker 16 Processed 800/3124 images \n",
      "Worker 12 Processed 800/3124 images \n",
      "Worker 26 Processed 800/3124 images \n",
      "Worker 7 Processed 800/3124 images \n",
      "Worker 29 Processed 800/3124 images \n",
      "Worker 4 Processed 800/3124 images \n",
      "Worker 24 Processed 800/3124 images \n",
      "Worker 20 Processed 800/3124 images \n",
      "Worker 18 Processed 800/3124 images \n",
      "Worker 23 Processed 700/3124 images \n",
      "Worker 17 Processed 800/3124 images \n",
      "Worker 28 Processed 800/3124 images \n",
      "Worker 27 Processed 800/3124 images \n",
      "Worker 31 Processed 800/3124 images \n",
      "Worker 9 Processed 800/3124 images \n",
      "Worker 30 Processed 800/3124 images \n",
      "Worker 13 Processed 800/3124 images \n",
      "Worker 14 Processed 800/3124 images \n",
      "Worker 19 Processed 800/3124 images \n",
      "Worker 15 Processed 800/3124 images \n",
      "Worker 21 Processed 800/3124 images \n",
      "Worker 6 Processed 800/3124 images \n",
      "Worker 10 Processed 800/3124 images \n",
      "Worker 8 Processed 800/3124 images \n",
      "Worker 11 Processed 800/3124 images \n",
      "Worker 2 Processed 600/3124 images \n",
      "Worker 26 Processed 900/3124 images \n",
      "Worker 4 Processed 900/3124 images \n",
      "Worker 29 Processed 900/3124 images \n",
      "Worker 3 Processed 600/3124 images \n",
      "Worker 16 Processed 900/3124 images \n",
      "Worker 24 Processed 900/3124 images \n",
      "Worker 20 Processed 900/3124 images \n",
      "Worker 18 Processed 900/3124 images \n",
      "Worker 31 Processed 900/3124 images \n",
      "Worker 1 Processed 700/3124 images \n",
      "Worker 22 Processed 900/3124 images \n",
      "Worker 9 Processed 900/3124 images \n",
      "Worker 30 Processed 900/3124 images \n",
      "Worker 14 Processed 900/3124 images \n",
      "Worker 13 Processed 900/3124 images \n",
      "Worker 5 Processed 1000/3124 images \n",
      "Worker 15 Processed 900/3124 images \n",
      "Worker 19 Processed 900/3124 images \n",
      "Worker 21 Processed 900/3124 images \n",
      "Worker 6 Processed 900/3124 images \n",
      "Worker 26 Processed 1000/3124 images \n",
      "Worker 8 Processed 900/3124 images \n",
      "Worker 25 Processed 1000/3124 images \n",
      "Worker 11 Processed 900/3124 images \n",
      "Worker 4 Processed 1000/3124 images \n",
      "Worker 29 Processed 1000/3124 images \n",
      "Worker 20 Processed 1000/3124 images \n",
      "Worker 27 Processed 1000/3124 images \n",
      "Worker 3 Processed 700/3124 images \n",
      "Worker 18 Processed 1000/3124 images \n",
      "Worker 22 Processed 1000/3124 images \n",
      "Worker 31 Processed 1000/3124 images \n",
      "Worker 23 Processed 900/3124 images \n",
      "Worker 14 Processed 1000/3124 images \n",
      "Worker 9 Processed 1000/3124 images \n",
      "Worker 30 Processed 1000/3124 images \n",
      "Worker 5 Processed 1100/3124 images \n",
      "Worker 21 Processed 1000/3124 images \n",
      "Worker 15 Processed 1000/3124 images \n",
      "Worker 26 Processed 1100/3124 images \n",
      "Worker 19 Processed 1000/3124 images \n",
      "Worker 25 Processed 1100/3124 images \n",
      "Worker 10 Processed 1000/3124 images \n",
      "Worker 11 Processed 1000/3124 images \n",
      "Worker 8 Processed 1000/3124 images \n",
      "Worker 4 Processed 1100/3124 images \n",
      "Worker 29 Processed 1100/3124 images \n",
      "Worker 24 Processed 1100/3124 images \n",
      "Worker 20 Processed 1100/3124 images \n",
      "Worker 27 Processed 1100/3124 images \n",
      "Worker 28 Processed 1100/3124 images \n",
      "Worker 31 Processed 1100/3124 images \n",
      "Worker 18 Processed 1100/3124 images \n",
      "Worker 23 Processed 1000/3124 images \n",
      "Worker 2 Processed 800/3124 images \n",
      "Worker 5 Processed 1200/3124 images \n",
      "Worker 14 Processed 1100/3124 images \n",
      "Worker 9 Processed 1100/3124 images \n",
      "Worker 13 Processed 1100/3124 images \n",
      "Worker 3 Processed 800/3124 images \n",
      "Worker 21 Processed 1100/3124 images \n",
      "Worker 26 Processed 1200/3124 images \n",
      "Worker 15 Processed 1100/3124 images \n",
      "Worker 19 Processed 1100/3124 images \n",
      "Worker 4 Processed 1200/3124 images \n",
      "Worker 29 Processed 1200/3124 images \n",
      "Worker 10 Processed 1100/3124 images \n",
      "Worker 16 Processed 1200/3124 images \n",
      "Worker 8 Processed 1100/3124 images \n",
      "Worker 17 Processed 1200/3124 images \n",
      "Worker 11 Processed 1100/3124 images \n",
      "Worker 24 Processed 1200/3124 images \n",
      "Worker 20 Processed 1200/3124 images \n",
      "Worker 22 Processed 1200/3124 images \n",
      "Worker 27 Processed 1200/3124 images \n",
      "Worker 28 Processed 1200/3124 images \n",
      "Worker 5 Processed 1300/3124 images \n",
      "Worker 31 Processed 1200/3124 images \n",
      "Worker 14 Processed 1200/3124 images \n",
      "Worker 25 Processed 1300/3124 images \n",
      "Worker 21 Processed 1200/3124 images \n",
      "Worker 9 Processed 1200/3124 images \n",
      "Worker 30 Processed 1200/3124 images \n",
      "Worker 13 Processed 1200/3124 images \n",
      "Worker 26 Processed 1300/3124 images \n",
      "Worker 19 Processed 1200/3124 images \n",
      "Worker 7 Processed 1300/3124 images \n",
      "Worker 29 Processed 1300/3124 images \n",
      "Worker 12 Processed 1300/3124 images \n",
      "Worker 2 Processed 900/3124 images \n",
      "Worker 4 Processed 1300/3124 images \n",
      "Worker 10 Processed 1200/3124 images \n",
      "Worker 8 Processed 1200/3124 images \n",
      "Worker 0 Processed 900/3124 images \n",
      "Worker 1 Processed 1000/3124 images \n",
      "Worker 20 Processed 1300/3124 images \n",
      "Worker 11 Processed 1200/3124 images \n",
      "Worker 22 Processed 1300/3124 images \n",
      "Worker 27 Processed 1300/3124 images \n",
      "Worker 28 Processed 1300/3124 images \n",
      "Worker 18 Processed 1300/3124 images \n",
      "Worker 25 Processed 1400/3124 images \n",
      "Worker 14 Processed 1300/3124 images \n",
      "Worker 21 Processed 1300/3124 images \n",
      "Worker 26 Processed 1400/3124 images \n",
      "Worker 30 Processed 1300/3124 images \n",
      "Worker 23 Processed 1200/3124 images \n",
      "Worker 9 Processed 1300/3124 images \n",
      "Worker 19 Processed 1300/3124 images \n",
      "Worker 29 Processed 1400/3124 images \n",
      "Worker 15 Processed 1300/3124 images \n",
      "Worker 7 Processed 1400/3124 images \n",
      "Worker 17 Processed 1400/3124 images \n",
      "Worker 4 Processed 1400/3124 images \n",
      "Worker 8 Processed 1300/3124 images \n",
      "Worker 10 Processed 1300/3124 images \n",
      "Worker 22 Processed 1400/3124 images \n",
      "Worker 24 Processed 1400/3124 images \n",
      "Worker 20 Processed 1400/3124 images \n",
      "Worker 11 Processed 1300/3124 images \n",
      "Worker 28 Processed 1400/3124 images \n",
      "Worker 2 Processed 1000/3124 images \n",
      "Worker 25 Processed 1500/3124 images \n",
      "Worker 18 Processed 1400/3124 images \n",
      "Worker 14 Processed 1400/3124 images \n",
      "Worker 26 Processed 1500/3124 images \n",
      "Worker 21 Processed 1400/3124 images \n",
      "Worker 31 Processed 1400/3124 images \n",
      "Worker 23 Processed 1300/3124 images \n",
      "Worker 13 Processed 1400/3124 images \n",
      "Worker 30 Processed 1400/3124 images \n",
      "Worker 9 Processed 1400/3124 images \n",
      "Worker 19 Processed 1400/3124 images \n",
      "Worker 15 Processed 1400/3124 images \n",
      "Worker 17 Processed 1500/3124 images \n",
      "Worker 7 Processed 1500/3124 images \n",
      "Worker 4 Processed 1500/3124 images \n",
      "Worker 10 Processed 1400/3124 images \n",
      "Worker 5 Processed 1600/3124 images \n",
      "Worker 20 Processed 1500/3124 images \n",
      "Worker 28 Processed 1500/3124 images \n",
      "Worker 25 Processed 1600/3124 images \n",
      "Worker 11 Processed 1400/3124 images \n",
      "Worker 14 Processed 1500/3124 images \n",
      "Worker 18 Processed 1500/3124 images \n",
      "Worker 1 Processed 1200/3124 images \n",
      "Worker 23 Processed 1400/3124 images \n",
      "Worker 26 Processed 1600/3124 images \n",
      "Worker 21 Processed 1500/3124 images \n",
      "Worker 31 Processed 1500/3124 images \n",
      "Worker 2 Processed 1100/3124 images \n",
      "Worker 17 Processed 1600/3124 images \n",
      "Worker 19 Processed 1500/3124 images \n",
      "Worker 29 Processed 1600/3124 images \n",
      "Worker 8 Processed 1500/3124 images \n",
      "Worker 13 Processed 1500/3124 images \n",
      "Worker 30 Processed 1500/3124 images \n",
      "Worker 7 Processed 1600/3124 images \n",
      "Worker 16 Processed 1600/3124 images \n",
      "Worker 15 Processed 1500/3124 images \n",
      "Worker 4 Processed 1600/3124 images \n",
      "Worker 24 Processed 1600/3124 images \n",
      "Worker 10 Processed 1500/3124 images \n",
      "Worker 20 Processed 1600/3124 images \n",
      "Worker 28 Processed 1600/3124 images \n",
      "Worker 23 Processed 1500/3124 images \n",
      "Worker 11 Processed 1500/3124 images \n",
      "Worker 26 Processed 1700/3124 images \n",
      "Worker 21 Processed 1600/3124 images \n",
      "Worker 18 Processed 1600/3124 images \n",
      "Worker 1 Processed 1300/3124 images \n",
      "Worker 31 Processed 1600/3124 images \n",
      "Worker 0 Processed 1200/3124 images \n",
      "Worker 8 Processed 1600/3124 images \n",
      "Worker 29 Processed 1700/3124 images \n",
      "Worker 19 Processed 1600/3124 images \n",
      "Worker 13 Processed 1600/3124 images \n",
      "Worker 3 Processed 1200/3124 images \n",
      "Worker 30 Processed 1600/3124 images \n",
      "Worker 4 Processed 1700/3124 images \n",
      "Worker 16 Processed 1700/3124 images \n",
      "Worker 27 Processed 1700/3124 images \n",
      "Worker 20 Processed 1700/3124 images \n",
      "Worker 24 Processed 1700/3124 images \n",
      "Worker 10 Processed 1600/3124 images \n",
      "Worker 2 Processed 1200/3124 images \n",
      "Worker 25 Processed 1800/3124 images \n",
      "Worker 28 Processed 1700/3124 images \n",
      "Worker 23 Processed 1600/3124 images \n",
      "Worker 26 Processed 1800/3124 images \n",
      "Worker 14 Processed 1700/3124 images \n",
      "Worker 17 Processed 1800/3124 images \n",
      "Worker 11 Processed 1600/3124 images \n",
      "Worker 31 Processed 1700/3124 images \n",
      "Worker 6 Processed 1700/3124 images \n",
      "Worker 12 Processed 1800/3124 images \n",
      "Worker 19 Processed 1700/3124 images \n",
      "Worker 1 Processed 1400/3124 images \n",
      "Worker 13 Processed 1700/3124 images \n",
      "Worker 27 Processed 1800/3124 images \n",
      "Worker 20 Processed 1800/3124 images \n",
      "Worker 30 Processed 1700/3124 images \n",
      "Worker 15 Processed 1700/3124 images \n",
      "Worker 16 Processed 1800/3124 images \n",
      "Worker 24 Processed 1800/3124 images \n",
      "Worker 9 Processed 1700/3124 images \n",
      "Worker 25 Processed 1900/3124 images \n",
      "Worker 10 Processed 1700/3124 images \n",
      "Worker 28 Processed 1800/3124 images \n",
      "Worker 26 Processed 1900/3124 images \n",
      "Worker 14 Processed 1800/3124 images \n",
      "Worker 22 Processed 1900/3124 images \n",
      "Worker 21 Processed 1800/3124 images \n",
      "Worker 17 Processed 1900/3124 images \n",
      "Worker 2 Processed 1300/3124 images \n",
      "Worker 5 Processed 2000/3124 images \n",
      "Worker 11 Processed 1700/3124 images \n",
      "Worker 6 Processed 1800/3124 images \n",
      "Worker 4 Processed 1900/3124 images \n",
      "Worker 31 Processed 1800/3124 images \n",
      "Worker 18 Processed 1800/3124 images \n",
      "Worker 13 Processed 1800/3124 images \n",
      "Worker 12 Processed 1900/3124 images \n",
      "Worker 24 Processed 1900/3124 images \n",
      "Worker 1 Processed 1500/3124 images \n",
      "Worker 19 Processed 1800/3124 images \n",
      "Worker 15 Processed 1800/3124 images \n",
      "Worker 30 Processed 1800/3124 images \n",
      "Worker 16 Processed 1900/3124 images \n",
      "Worker 23 Processed 1800/3124 images \n",
      "Worker 10 Processed 1800/3124 images \n",
      "Worker 26 Processed 2000/3124 images \n",
      "Worker 28 Processed 1900/3124 images \n",
      "Worker 21 Processed 1900/3124 images \n",
      "Worker 11 Processed 1800/3124 images \n",
      "Worker 20 Processed 2000/3124 images \n",
      "Worker 8 Processed 1900/3124 images \n",
      "Worker 18 Processed 1900/3124 images \n",
      "Worker 31 Processed 1900/3124 images \n",
      "Worker 27 Processed 2000/3124 images \n",
      "Worker 25 Processed 2100/3124 images \n",
      "Worker 13 Processed 1900/3124 images \n",
      "Worker 24 Processed 2000/3124 images \n",
      "Worker 12 Processed 2000/3124 images \n",
      "Worker 19 Processed 1900/3124 images \n",
      "Worker 30 Processed 1900/3124 images \n",
      "Worker 16 Processed 2000/3124 images \n",
      "Worker 9 Processed 1900/3124 images \n",
      "Worker 26 Processed 2100/3124 images \n",
      "Worker 23 Processed 1900/3124 images \n",
      "Worker 1 Processed 1600/3124 images \n",
      "Worker 28 Processed 2000/3124 images \n",
      "Worker 10 Processed 1900/3124 images \n",
      "Worker 21 Processed 2000/3124 images \n",
      "Worker 5 Processed 2200/3124 images \n",
      "Worker 8 Processed 2000/3124 images \n",
      "Worker 11 Processed 1900/3124 images \n",
      "Worker 18 Processed 2000/3124 images \n",
      "Worker 25 Processed 2200/3124 images \n",
      "Worker 31 Processed 2000/3124 images \n",
      "Worker 29 Processed 2100/3124 images \n",
      "Worker 13 Processed 2000/3124 images \n",
      "Worker 7 Processed 2100/3124 images \n",
      "Worker 24 Processed 2100/3124 images \n",
      "Worker 19 Processed 2000/3124 images \n",
      "Worker 26 Processed 2200/3124 images \n",
      "Worker 23 Processed 2000/3124 images \n",
      "Worker 12 Processed 2100/3124 images \n",
      "Worker 16 Processed 2100/3124 images \n",
      "Worker 9 Processed 2000/3124 images \n",
      "Worker 30 Processed 2000/3124 images \n",
      "Worker 15 Processed 2000/3124 images \n",
      "Worker 21 Processed 2100/3124 images \n",
      "Worker 2 Processed 1500/3124 images \n",
      "Worker 4 Processed 2200/3124 images \n",
      "Worker 11 Processed 2000/3124 images \n",
      "Worker 8 Processed 2100/3124 images \n",
      "Worker 27 Processed 2200/3124 images \n",
      "Worker 18 Processed 2100/3124 images \n",
      "Worker 24 Processed 2200/3124 images \n",
      "Worker 7 Processed 2200/3124 images \n",
      "Worker 29 Processed 2200/3124 images \n",
      "Worker 31 Processed 2100/3124 images \n",
      "Worker 26 Processed 2300/3124 images \n",
      "Worker 19 Processed 2100/3124 images \n",
      "Worker 12 Processed 2200/3124 images \n",
      "Worker 16 Processed 2200/3124 images \n",
      "Worker 15 Processed 2100/3124 images \n",
      "Worker 9 Processed 2100/3124 images \n",
      "Worker 28 Processed 2200/3124 images \n",
      "Worker 30 Processed 2100/3124 images \n",
      "Worker 20 Processed 2300/3124 images \n",
      "Worker 1 Processed 1800/3124 images \n",
      "Worker 25 Processed 2400/3124 images \n",
      "Worker 27 Processed 2300/3124 images \n",
      "Worker 8 Processed 2200/3124 images \n",
      "Worker 24 Processed 2300/3124 images \n",
      "Worker 29 Processed 2300/3124 images \n",
      "Worker 23 Processed 2200/3124 images \n",
      "Worker 13 Processed 2200/3124 images \n",
      "Worker 0 Processed 1700/3124 images \n",
      "Worker 3 Processed 1700/3124 images \n",
      "Worker 16 Processed 2300/3124 images \n",
      "Worker 14 Processed 2300/3124 images \n",
      "Worker 5 Processed 2500/3124 images \n",
      "Worker 31 Processed 2200/3124 images \n",
      "Worker 22 Processed 2400/3124 images \n",
      "Worker 9 Processed 2200/3124 images \n",
      "Worker 30 Processed 2200/3124 images \n",
      "Worker 20 Processed 2400/3124 images \n",
      "Worker 4 Processed 2400/3124 images \n",
      "Worker 27 Processed 2400/3124 images \n",
      "Worker 1 Processed 1900/3124 images \n",
      "Worker 8 Processed 2300/3124 images \n",
      "Worker 24 Processed 2400/3124 images \n",
      "Worker 26 Processed 2500/3124 images \n",
      "Worker 23 Processed 2300/3124 images \n",
      "Worker 29 Processed 2400/3124 images \n",
      "Worker 7 Processed 2400/3124 images \n",
      "Worker 13 Processed 2300/3124 images \n",
      "Worker 12 Processed 2400/3124 images \n",
      "Worker 19 Processed 2300/3124 images \n",
      "Worker 28 Processed 2400/3124 images \n",
      "Worker 16 Processed 2400/3124 images \n",
      "Worker 5 Processed 2600/3124 images \n",
      "Worker 14 Processed 2400/3124 images \n",
      "Worker 15 Processed 2300/3124 images \n",
      "Worker 22 Processed 2500/3124 images \n",
      "Worker 4 Processed 2500/3124 images \n",
      "Worker 30 Processed 2300/3124 images \n",
      "Worker 25 Processed 2600/3124 images \n",
      "Worker 31 Processed 2300/3124 images \n",
      "Worker 11 Processed 2300/3124 images \n",
      "Worker 3 Processed 1800/3124 images \n",
      "Worker 27 Processed 2500/3124 images \n",
      "Worker 24 Processed 2500/3124 images \n",
      "Worker 17 Processed 2600/3124 images \n",
      "Worker 1 Processed 2000/3124 images \n",
      "Worker 26 Processed 2600/3124 images \n",
      "Worker 8 Processed 2400/3124 images \n",
      "Worker 18 Processed 2400/3124 images \n",
      "Worker 29 Processed 2500/3124 images \n",
      "Worker 7 Processed 2500/3124 images \n",
      "Worker 13 Processed 2400/3124 images \n",
      "Worker 19 Processed 2400/3124 images \n",
      "Worker 12 Processed 2500/3124 images \n",
      "Worker 28 Processed 2500/3124 images \n",
      "Worker 16 Processed 2500/3124 images \n",
      "Worker 4 Processed 2600/3124 images \n",
      "Worker 25 Processed 2700/3124 images \n",
      "Worker 14 Processed 2500/3124 images \n",
      "Worker 10 Processed 2400/3124 images \n",
      "Worker 15 Processed 2400/3124 images \n",
      "Worker 21 Processed 2500/3124 images \n",
      "Worker 22 Processed 2600/3124 images \n",
      "Worker 11 Processed 2400/3124 images \n",
      "Worker 30 Processed 2400/3124 images \n",
      "Worker 31 Processed 2400/3124 images \n",
      "Worker 24 Processed 2600/3124 images \n",
      "Worker 6 Processed 2600/3124 images \n",
      "Worker 26 Processed 2700/3124 images \n",
      "Worker 8 Processed 2500/3124 images \n",
      "Worker 7 Processed 2600/3124 images \n",
      "Worker 18 Processed 2500/3124 images \n",
      "Worker 28 Processed 2600/3124 images \n",
      "Worker 12 Processed 2600/3124 images \n",
      "Worker 19 Processed 2500/3124 images \n",
      "Worker 25 Processed 2800/3124 images \n",
      "Worker 5 Processed 2800/3124 images \n",
      "Worker 16 Processed 2600/3124 images \n",
      "Worker 14 Processed 2600/3124 images \n",
      "Worker 11 Processed 2500/3124 images \n",
      "Worker 3 Processed 1900/3124 images \n",
      "Worker 31 Processed 2500/3124 images \n",
      "Worker 27 Processed 2700/3124 images \n",
      "Worker 21 Processed 2600/3124 images \n",
      "Worker 24 Processed 2700/3124 images \n",
      "Worker 2 Processed 1900/3124 images \n",
      "Worker 6 Processed 2700/3124 images \n",
      "Worker 26 Processed 2800/3124 images \n",
      "Worker 23 Processed 2600/3124 images \n",
      "Worker 29 Processed 2700/3124 images \n",
      "Worker 8 Processed 2600/3124 images \n",
      "Worker 13 Processed 2600/3124 images \n",
      "Worker 7 Processed 2700/3124 images \n",
      "Worker 9 Processed 2600/3124 images \n",
      "Worker 28 Processed 2700/3124 images \n",
      "Worker 0 Processed 2000/3124 images \n",
      "Worker 18 Processed 2600/3124 images \n",
      "Worker 20 Processed 2800/3124 images \n",
      "Worker 25 Processed 2900/3124 images \n",
      "Worker 19 Processed 2600/3124 images \n",
      "Worker 1 Processed 2200/3124 images \n",
      "Worker 5 Processed 2900/3124 images \n",
      "Worker 14 Processed 2700/3124 images \n",
      "Worker 11 Processed 2600/3124 images \n",
      "Worker 16 Processed 2700/3124 images \n",
      "Worker 31 Processed 2600/3124 images \n",
      "Worker 15 Processed 2600/3124 images \n",
      "Worker 27 Processed 2800/3124 images \n",
      "Worker 24 Processed 2800/3124 images \n",
      "Worker 21 Processed 2700/3124 images \n",
      "Worker 10 Processed 2600/3124 images \n",
      "Worker 22 Processed 2800/3124 images \n",
      "Worker 30 Processed 2600/3124 images \n",
      "Worker 26 Processed 2900/3124 images \n",
      "Worker 3 Processed 2000/3124 images \n",
      "Worker 19 Processed 2700/3124 images \n",
      "Worker 18 Processed 2700/3124 images \n",
      "Worker 12 Processed 2800/3124 images \n",
      "Worker 9 Processed 2700/3124 images \n",
      "Worker 5 Processed 3000/3124 images \n",
      "Worker 11 Processed 2700/3124 images \n",
      "Worker 1 Processed 2300/3124 images \n",
      "Worker 16 Processed 2800/3124 images \n",
      "Worker 31 Processed 2700/3124 images \n",
      "Worker 14 Processed 2800/3124 images \n",
      "Worker 27 Processed 2900/3124 images \n",
      "Worker 15 Processed 2700/3124 images \n",
      "Worker 24 Processed 2900/3124 images \n",
      "Worker 23 Processed 2800/3124 images \n",
      "Worker 30 Processed 2700/3124 images \n",
      "Worker 22 Processed 2900/3124 images \n",
      "Worker 26 Processed 3000/3124 images \n",
      "Worker 7 Processed 2900/3124 images \n",
      "Worker 8 Processed 2800/3124 images \n",
      "Worker 19 Processed 2800/3124 images \n",
      "Worker 25 Processed 3100/3124 images \n",
      "Worker 28 Processed 2900/3124 images \n",
      "Worker 12 Processed 2900/3124 images \n",
      "Worker 25 Processed 3124 images, took 0:07:29.907227\n",
      "Worker 9 Processed 2800/3124 images \n",
      "Worker 31 Processed 2800/3124 images \n",
      "Worker 11 Processed 2800/3124 images \n",
      "Worker 14 Processed 2900/3124 images \n",
      "Worker 16 Processed 2900/3124 images \n",
      "Worker 24 Processed 3000/3124 images \n",
      "Worker 27 Processed 3000/3124 images \n",
      "Worker 5 Processed 3124 images, took 0:07:33.888286\n",
      "Worker 1 Processed 2400/3124 images \n",
      "Worker 23 Processed 2900/3124 images \n",
      "Worker 17 Processed 3124 images, took 0:07:32.088720\n",
      "Worker 29 Processed 3000/3124 images \n",
      "Worker 26 Processed 3100/3124 images \n",
      "Worker 30 Processed 2800/3124 images \n",
      "Worker 21 Processed 2900/3124 images \n",
      "Worker 0 Processed 2200/3124 images \n",
      "Worker 13 Processed 2900/3124 images \n",
      "Worker 22 Processed 3000/3124 images \n",
      "Worker 10 Processed 2800/3124 images \n",
      "Worker 4 Processed 3100/3124 images \n",
      "Worker 26 Processed 3124 images, took 0:07:40.088747\n",
      "Worker 8 Processed 2900/3124 images \n",
      "Worker 4 Processed 3124 images, took 0:07:42.249542\n",
      "Worker 28 Processed 3000/3124 images \n",
      "Worker 31 Processed 2900/3124 images \n",
      "Worker 20 Processed 3124 images, took 0:07:42.490891\n",
      "Worker 14 Processed 3000/3124 images \n",
      "Worker 2 Processed 2200/3124 images \n",
      "Worker 6 Processed 3100/3124 images \n",
      "Worker 24 Processed 3100/3124 images \n",
      "Worker 29 Processed 3100/3124 images \n",
      "Worker 23 Processed 3000/3124 images \n",
      "Worker 16 Processed 3000/3124 images \n",
      "Worker 27 Processed 3100/3124 images \n",
      "Worker 15 Processed 2900/3124 images \n",
      "Worker 30 Processed 2900/3124 images \n",
      "Worker 21 Processed 3000/3124 images \n",
      "Worker 6 Processed 3124 images, took 0:07:52.014537\n",
      "Worker 24 Processed 3124 images, took 0:07:52.615546\n",
      "Worker 29 Processed 3124 images, took 0:07:55.842106\n",
      "Worker 27 Processed 3124 images, took 0:07:52.380407\n",
      "Worker 7 Processed 3100/3124 images \n",
      "Worker 13 Processed 3000/3124 images \n",
      "Worker 8 Processed 3000/3124 images \n",
      "Worker 7 Processed 3124 images, took 0:07:56.888533\n",
      "Worker 0 Processed 2300/3124 images \n",
      "Worker 22 Processed 3124 images, took 0:07:56.019456\n",
      "Worker 12 Processed 3100/3124 images \n",
      "Worker 3 Processed 2200/3124 images \n",
      "Worker 11 Processed 3000/3124 images \n",
      "Worker 9 Processed 3000/3124 images \n",
      "Worker 28 Processed 3124 images, took 0:08:06.295131\n",
      "Worker 12 Processed 3124 images, took 0:08:04.032741\n",
      "Worker 23 Processed 3100/3124 images \n",
      "Worker 16 Processed 3100/3124 images \n",
      "Worker 15 Processed 3000/3124 images \n",
      "Worker 23 Processed 3124 images, took 0:08:04.844923\n",
      "Worker 30 Processed 3000/3124 images \n",
      "Worker 19 Processed 3100/3124 images \n",
      "Worker 14 Processed 3124 images, took 0:08:09.362103\n",
      "Worker 16 Processed 3124 images, took 0:08:06.255468\n",
      "Worker 13 Processed 3100/3124 images \n",
      "Worker 2 Processed 2300/3124 images \n",
      "Worker 1 Processed 2600/3124 images \n",
      "Worker 21 Processed 3124 images, took 0:08:08.913304\n",
      "Worker 8 Processed 3100/3124 images \n",
      "Worker 19 Processed 3124 images, took 0:08:09.357111\n",
      "Worker 13 Processed 3124 images, took 0:08:14.316996\n",
      "Worker 11 Processed 3100/3124 images \n",
      "Worker 8 Processed 3124 images, took 0:08:14.693738\n",
      "Worker 18 Processed 3100/3124 images \n",
      "Worker 31 Processed 3124 images, took 0:08:20.319410\n",
      "Worker 9 Processed 3100/3124 images \n",
      "Worker 11 Processed 3124 images, took 0:08:18.256543\n",
      "Worker 18 Processed 3124 images, took 0:08:16.661916\n",
      "Worker 15 Processed 3100/3124 images \n",
      "Worker 9 Processed 3124 images, took 0:08:22.110053\n",
      "Worker 0 Processed 2400/3124 images \n",
      "Worker 30 Processed 3100/3124 images \n",
      "Worker 15 Processed 3124 images, took 0:08:26.398753\n",
      "Worker 30 Processed 3124 images, took 0:08:32.005622\n",
      "Worker 2 Processed 2400/3124 images \n",
      "Worker 10 Processed 3100/3124 images \n",
      "Worker 10 Processed 3124 images, took 0:08:33.822500\n",
      "Worker 3 Processed 2400/3124 images \n",
      "Worker 0 Processed 2500/3124 images \n",
      "Worker 1 Processed 2800/3124 images \n",
      "Worker 2 Processed 2500/3124 images \n",
      "Worker 3 Processed 2500/3124 images \n",
      "Worker 1 Processed 2900/3124 images \n",
      "Worker 3 Processed 2600/3124 images \n",
      "Worker 2 Processed 2700/3124 images \n",
      "Worker 1 Processed 3100/3124 images \n",
      "Worker 0 Processed 2800/3124 images \n",
      "Worker 1 Processed 3124 images, took 0:09:43.772801\n",
      "Worker 2 Processed 2800/3124 images \n",
      "Worker 3 Processed 2900/3124 images \n",
      "Worker 0 Processed 3124 images, took 0:10:43.983157\n",
      "Worker 3 Processed 3000/3124 images \n",
      "Worker 2 Processed 3124 images, took 0:10:55.710531\n",
      "Worker 3 Processed 3100/3124 images \n",
      "Worker 3 Processed 3124 images, took 0:11:10.364066\n",
      "Job state: succeeded ExitCode: 0\n"
     ]
    }
   ],
   "source": [
    "utilities.wait_for_job_completion(client, cfg.resource_group, job_name, cluster_name, 'stdouterr', 'stdout.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Download label files for the Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "files = client.jobs.list_output_files(cfg.resource_group, job_name, models.JobsListOutputFilesOptions(\"LABEL\")) \n",
    "for f in list(files):\n",
    "    print(f.name + \": \" + str(f.content_length) + ' Bytes')\n",
    "    utilities.download_file(f.download_url, f.name)\n",
    "print(\"All files downloaded\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Clean Up (Optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "_ = client.jobs.delete(cfg.resource_group, job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the Cluster\n",
    "When you are finished with the sample and don't want to submit any more jobs you can delete the cluster using the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "_ = client.clusters.delete(cfg.resource_group, cluster_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Delete File Share\n",
    "When you are finished with the sample and don't want to submit any more jobs you can delete the file share completely with all files using the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "service = FileService(cfg.storage_account_name, cfg.storage_account_key)\n",
    "service.delete_share(azure_file_share_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}