📈 More training (Tuakau), upgrade dnn libraries

Improvements to keras model again by training on more data. This time using 0.1m resolution imagery from Waikato. Tuakau was chosen as it was close to Auckland, and had a good amount of greenhouses (tile bb32_4735) to decrease bias and increase detection by a bit on those types of shiny buildings. Model trained on 100 epochs, resuming from the previous checkpoint (Wigram and Hastings), bringing total training to 300 epochs. Mean_iou accuracy metric is now >90%. TQDM progress bars added in the data processing step. predict.py script now accepts a threshold value, so one can run something like `python predict.py 512 50` to get a building detector of size 512*512 and probabilty >50%. DNN libraries updated to CUDA 9, CUDNN 7.0.5, Keras 2.1.4, Tensorflow 1.5 (from https://github.com/mind/wheels). There is a known issue keras-team/keras#9394 in having to compile a loaded model before predict can work...
weiji14 · Feb 28, 2018 · 832a615 · 832a615
1 parent 1dd00b8
commit 832a615
Show file tree

Hide file tree

Showing 5 changed files with 101 additions and 76 deletions.
diff --git a/README.md b/README.md
@@ -32,9 +32,14 @@ Launch Binder
 | data/test             | wellington-03m-rural-aerial-photos-2012-2013.tif | \*.tif    | unzipped files similar to those in data/raster |
 
 ## Near realtime prediction
-    
+
     source activate nz_convnet
-    python predict.py 512  #set to output pixel size e.g. 256, 512, 1024
+    python predict.py
+
+    #You can also set 2 integer parameters:
+    #  1st argument - output pixel size e.g. 256, 512, 1024 (default: 256)
+    #  2nd argument - prediction threshold e.g from least accurate 0% accept anything to 100% won't output much (default: 50)
+    python predict.py 512 50 
 
 Live testing on imagery of Karori, Wellington.
 
@@ -52,7 +57,7 @@ Using freely available data from [LINZ Data Service](https://data.linz.govt.nz/)
 | ------------------------------------- |:-----------------------------------:| --------------:|
 | General Electorate Boundaries 2014    | Wigram                              | [Canterbury 0.3m Rural Aerial Photos (2015-16)](https://data.linz.govt.nz/layer/53519-canterbury-03m-rural-aerial-photos-2015-16/) |
 | Manual Tile Selection\*               | Hastings 2015_BK39_5000_{XXXX}_RGB  | [0401](https://data.linz.govt.nz/x/vnGVkg) [0402](https://data.linz.govt.nz/x/aA5XSv) [0403](https://data.linz.govt.nz/x/DYsY9B) [0404](https://data.linz.govt.nz/x/qvgapR) [0405](https://data.linz.govt.nz/x/VKVcWf) [0501](https://data.linz.govt.nz/x/8hJeCu) [0502](https://data.linz.govt.nz/x/k57ftA) [0503](https://data.linz.govt.nz/x/QTuhaQ) [0504](https://data.linz.govt.nz/x/3qijGe) [0505](https://data.linz.govt.nz/x/gEXkwt) [0601](https://data.linz.govt.nz/x/KcLnd9) [0602](https://data.linz.govt.nz/x/wy9pLP) [0603](https://data.linz.govt.nz/x/bNwq2d) [0604](https://data.linz.govt.nz/x/Ekkshs) [0605](https://data.linz.govt.nz/x/r9ZuP8) |
-
+| Manual Tile Selection\*               | Tuakau bb32_{XXXX}                  | [4630](https://data.linz.govt.nz/x/9s9M9A) [4631](https://data.linz.govt.nz/x/nGwPpQ) [4632](https://data.linz.govt.nz/x/RekRWe) [4633](https://data.linz.govt.nz/x/43ZTCt) [4634](https://data.linz.govt.nz/x/hRNUs9) [4635](https://data.linz.govt.nz/x/LoBWaP) [4636](https://data.linz.govt.nz/x/yByYGd) [4637](https://data.linz.govt.nz/x/Fwbbd8) [4638](https://data.linz.govt.nz/x/tLQdLN) [4639](https://data.linz.govt.nz/x/XiDe2c) [4730](https://data.linz.govt.nz/x/oUpiP7) [4731](https://data.linz.govt.nz/x/Srdj6M) [4732](https://data.linz.govt.nz/x/6FSmmb) [4733](https://data.linz.govt.nz/x/yAvvdB) [4734](https://data.linz.govt.nz/x/idFoTq) [4735](https://data.linz.govt.nz/x/Mz4p96) [4736](https://data.linz.govt.nz/x/zPrrqL) [4737](https://data.linz.govt.nz/x/dmftXa) [4738](https://data.linz.govt.nz/x/HAUvDp) [4739](https://data.linz.govt.nz/x/uYHwt5) [4830](https://data.linz.govt.nz/x/pgh3xo) [4831](https://data.linz.govt.nz/x/T5W5e4) [4832](https://data.linz.govt.nz/x/7TK7MJ) [4833](https://data.linz.govt.nz/x/jp883Y) [4834](https://data.linz.govt.nz/x/PDwAin) [4835](https://data.linz.govt.nz/x/2bkCQ3) [4836](https://data.linz.govt.nz/x/eyZD7H) [4837](https://data.linz.govt.nz/x/JNNFnX) [4838](https://data.linz.govt.nz/x/vkBHUm) [4839](https://data.linz.govt.nz/x/Z8yKA2) [4930](https://data.linz.govt.nz/x/DWnLrG) [4931](https://data.linz.govt.nz/x/qtbNYW) [4932](https://data.linz.govt.nz/x/VHQQEk) [4933](https://data.linz.govt.nz/x/8fDRuz) [4934](https://data.linz.govt.nz/x/k32TcF) [4935](https://data.linz.govt.nz/x/QRpVJV) [4936](https://data.linz.govt.nz/x/3odWyj) [4937](https://data.linz.govt.nz/x/gCSYfy) [4938](https://data.linz.govt.nz/x/KaFaNE) [4939](https://data.linz.govt.nz/x/ww4b4U) [5030](https://data.linz.govt.nz/x/bLrdji) [5031](https://data.linz.govt.nz/x/EiffRx) [5032](https://data.linz.govt.nz/x/r7Ug8D) [5033](https://data.linz.govt.nz/x/WVHioT) [5034](https://data.linz.govt.nz/x/9r6kVh) [5035](https://data.linz.govt.nz/x/nFtnBw) [5036](https://data.linz.govt.nz/x/RdhosC) [5037](https://data.linz.govt.nz/x/42WqZS) [5038](https://data.linz.govt.nz/x/hQKsFg) [5039](https://data.linz.govt.nz/x/Lm8tvv) 
 
 * Manual tile selection selects tiles manually from the tiles table, e.g. [here](https://data.linz.govt.nz/layer/53401-hawkes-bay-03m-rural-aerial-photos-2014-15/data/)
 

diff --git a/data/model/model-weights.hdf5 b/data/model/model-weights.hdf5
diff --git a/environment.yml b/environment.yml
@@ -1,20 +1,25 @@
-name: nz_convnet
-channels:
-  - conda-forge
-  - defaults
-dependencies:
-  - dask
-  - gdal
-  - h5py
-  - imagemagick
-  - jupyterlab
-  - keras>=2.1.3
-  - matplotlib
-  - numpy
-  - opencv
-  - scikit-learn
-  - scikit-image
-  - tensorflow-gpu>=1.4.1
-  - pip:
-    - pyscreenshot
-prefix: /home/user/miniconda3
+name: nz_convnet
+channels:
+  - conda-forge
+  - defaults
+dependencies:
+  #- numba::cudatoolkit==9.0
+  #- cudnn==7.0.5
+  - dask
+  - gdal
+  - h5py
+  - imagemagick
+  - jupyterlab
+  - matplotlib
+  - numpy
+  - opencv
+  - scikit-learn
+  - scikit-image
+  - tqdm
+  - pip:
+    - pyscreenshot
+    - keras==2.1.4
+    #- tensorflow==1.5
+    #- tensorflow-gpu==1.5
+    - https://github.com/mind/wheels/releases/download/tf1.5-gpu-nomkl/tensorflow-1.5.0-cp36-cp36m-linux_x86_64.whl
+prefix: /home/user/miniconda3
diff --git a/nz_convnet.ipynb b/nz_convnet.ipynb
@@ -127,8 +127,11 @@
     "import random\n",
     "import re\n",
     "import sys\n",
+    "import time\n",
     "import zipfile\n",
     "\n",
+    "import tqdm\n",
+    "\n",
     "import matplotlib.pyplot as plt\n",
     "import skimage.io                                     #Used for imshow function\n",
     "import skimage.transform                              #Used for resize function\n",
@@ -183,7 +186,7 @@
     "# Set seed values\n",
     "seed = 42\n",
     "random.seed = seed\n",
-    "np.random.seed(seed=seed)"
+    "np.random.seed(seed=seed)\n",
    ]
   },
   {
@@ -271,7 +274,7 @@
     "    urx = ulx + (width * px)\n",
     "    ury = uly\n",
     "    bbox = (llx, lly, urx, ury)  #minx, miny, maxx, maxy\n",
-    "    print(\"min_xy:({0},{1}), max_xy:({2},{3})\".format(*bbox))\n",
+    "    #print(\"min_xy:({0},{1}), max_xy:({2},{3})\".format(*bbox))\n",
     "    \n",
     "    \n",
     "    ## Open vector data source\n",
@@ -345,7 +348,9 @@
     "                continue\n",
     "            ary_list.append(crop_ary)\n",
     "    \n",
-    "    return np.stack(ary_list)  \n"
+    "    return np.stack(ary_list)  \n",
+    "\n",
+    "splitName = lambda filename: filename.split(os.sep)[-1].split('.')[0]   #function to strip filename of its directories and extension"
    ]
   },
   {
@@ -355,13 +360,16 @@
    "outputs": [],
    "source": [
     "uzFullPath = topDir+\"/raster/\"  #full path to location of the will-be unzipped files\n",
-    "for zFile in glob.glob(topDir+\"/raster/downloads/*.zip\"):\n",
-    "    with zipfile.ZipFile(file=zFile) as zf:\n",
-    "        for eachFile in zf.infolist():\n",
-    "            assert(len(eachFile.filename.split('/')) == 1)\n",
-    "            if not os.path.exists(os.path.join(uzFullPath, eachFile.filename)):\n",
-    "                print(\"Unzipping\", eachFile.filename)\n",
-    "                zf.extract(eachFile.filename, uzFullPath)  "
+    "with tqdm.tqdm(total=len(glob.glob(topDir+\"/raster/downloads/*.zip\"))) as pbar:\n",
+    "    for i, zFile in enumerate(sorted(glob.glob(topDir+\"/raster/downloads/*.zip\"))):\n",
+    "        pbar.set_description('processing: {0}'.format(splitName(zFile)))\n",
+    "        pbar.update(1)\n",
+    "        with zipfile.ZipFile(file=zFile) as zf:\n",
+    "            for eachFile in zf.infolist():\n",
+    "                assert(len(eachFile.filename.split('/')) == 1)\n",
+    "                if not os.path.exists(os.path.join(uzFullPath, eachFile.filename)):\n",
+    "                    #print(\"Unzipping\", eachFile.filename)\n",
+    "                    zf.extract(eachFile.filename, uzFullPath)"
    ]
   },
   {
@@ -370,22 +378,23 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "splitName = lambda filename: filename.split(os.sep)[-1].split('.')[0]   #function to strip filename of its directories and extension\n",
     "if not os.path.exists('train/X_data.npz') or not os.path.exists('train/Y_data.npz'):\n",
-    "    for i, raster in enumerate(glob.glob(topDir+\"/raster/*[!_mask].tif\")):\n",
-    "        tifName = splitName(raster)\n",
-    "        print(\"Processing {0}/{1}: {2}\".format(i, len(glob.glob(topDir+\"/raster/*[!_mask].tif\")), tifName))\n",
-    "        if not os.path.exists(\"train/X_{0}.hdf5\".format(tifName)):\n",
-    "            img_ary_tmp, msk_ary_tmp = vectorPoly_to_rasterMask(vector='vector/nz-building-outlines-pilot.shp', raster=raster)\n",
-    "            # Generate stacked numpy tile blocks of size (img_height, img_width) from one geotiff\n",
-    "            X_data_tmp = ary_to_tiles(img_ary_tmp, shape=(img_height, img_width))\n",
-    "            Y_data_tmp = ary_to_tiles(msk_ary_tmp, shape=(img_height, img_width))\n",
-    "            # Convert stacked numpy array to dask array\n",
-    "            X_da_tmp = da.from_array(X_data_tmp, chunks=(32,img_height,img_width,1))\n",
-    "            Y_da_tmp = da.from_array(Y_data_tmp, chunks=(32,img_height,img_width,1))\n",
-    "            # Save dask array to HDF5 file format with maximum compression\n",
-    "            X_da_tmp.to_hdf5(filename=\"train/X_{0}.hdf5\".format(tifName), datapath=tifName, compression=\"gzip\", compression_opts=9)\n",
-    "            Y_da_tmp.to_hdf5(filename=\"train/Y_{0}.hdf5\".format(tifName), datapath=tifName, compression=\"gzip\", compression_opts=9)\n",
+    "    with tqdm.tqdm(total=len(glob.glob(topDir+\"/raster/downloads/*.zip\"))) as pbar:\n",
+    "        for i, raster in enumerate(sorted(glob.glob(topDir+\"/raster/*[!_mask].tif\"))):\n",
+    "            tifName = splitName(raster)\n",
+    "            pbar.set_description('processing: {0}'.format(tifName))\n",
+    "            pbar.update(1)\n",
+    "            if not os.path.exists(\"train/X_{0}.hdf5\".format(tifName)):\n",
+    "                img_ary_tmp, msk_ary_tmp = vectorPoly_to_rasterMask(vector='vector/nz-building-outlines-pilot.shp', raster=raster)\n",
+    "                # Generate stacked numpy tile blocks of size (img_height, img_width) from one geotiff\n",
+    "                X_data_tmp = ary_to_tiles(img_ary_tmp, shape=(img_height, img_width))\n",
+    "                Y_data_tmp = ary_to_tiles(msk_ary_tmp, shape=(img_height, img_width))\n",
+    "                # Convert stacked numpy array to dask array\n",
+    "                X_da_tmp = da.from_array(X_data_tmp, chunks=(32,img_height,img_width,1))\n",
+    "                Y_da_tmp = da.from_array(Y_data_tmp, chunks=(32,img_height,img_width,1))\n",
+    "                # Save dask array to HDF5 file format with maximum compression\n",
+    "                X_da_tmp.to_hdf5(filename=\"train/X_{0}.hdf5\".format(tifName), datapath=tifName, compression=\"gzip\", compression_opts=9)\n",
+    "                Y_da_tmp.to_hdf5(filename=\"train/Y_{0}.hdf5\".format(tifName), datapath=tifName, compression=\"gzip\", compression_opts=9)\n",
     "            \n",
     "    xdsets = [h5py.File(fn)['/'+splitName(fn)[2:]] for fn in sorted(glob.glob(\"train/X_*.hdf5\"))]\n",
     "    ydsets = [h5py.File(fn)['/'+splitName(fn)[2:]] for fn in sorted(glob.glob(\"train/Y_*.hdf5\"))]\n",
@@ -403,16 +412,6 @@
     "    Y_data = np.load('train/Y_data.npz')['Y_data']"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "np.savez('train/Y_data.npz', Y_data=ydata[~zero_mask].compute())  #apply zero_mask to Y_train (masks) and save to compressed numpy format\n",
-    "np.savez('train/X_data.npz', X_data=xdata[~zero_mask].compute())  #apply zero_mask to X_train (images) and save to compressed numpy format"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -721,7 +720,8 @@
     "                                                    Y_data,\n",
     "                                                    train_size=1-validation_split,\n",
     "                                                    test_size=validation_split,\n",
-    "                                                    random_state=seed)"
+    "                                                    random_state=seed)\n",
+    "print(\"X_train: {0} tiles, X_test: {1} tiles\".format(len(X_train), len(X_test)))"
    ]
   },
   {
@@ -742,7 +742,7 @@
    "source": [
     "# Finally train the model!!\n",
     "batch_size = 32\n",
-    "epochs = 200\n",
+    "epochs = 300\n",
     "\n",
     "try:  #to load a checkpoint file if it exists\n",
     "    checkfile = sorted(glob.glob(topDir+\"/model/weights-*-*.hdf5\"))[-1]\n",
@@ -751,7 +751,11 @@
     "    print(\"Model weights loaded, resuming from epoch {0}\".format(initial_epoch))\n",
     "except IndexError:\n",
     "    initial_epoch = 0\n",
-    "    pass\n",
+    "    try:\n",
+    "        model.load_weights(topDir+\"/model/model-weights.hdf5\")\n",
+    "        print(\"Model weights loaded, starting from epoch {0}\".format(initial_epoch))\n",
+    "    except OSError:\n",
+    "        pass\n",
     "\n",
     "model.fit(x=X_train, y=Y_train, verbose=1, validation_split=0.25, batch_size=batch_size, epochs=epochs, callbacks=callbackList, initial_epoch=initial_epoch)\n",
     "#model.fit(x=X_train, y=Y_train, verbose=1, validation_data=(X_test, Y_test), batch_size=batch_size, epochs=epochs, callbacks=callbackList)"
@@ -809,7 +813,15 @@
    "source": [
     "# Reload the model\n",
     "model_loaded = keras_model(img_width=img_width, img_height=img_height)\n",
-    "model_loaded.load_weights(topDir+\"/model/model-weights.hdf5\")"
+    "model_loaded.load_weights(filepath=topDir+\"/model/model-weights.hdf5\")\n",
+    "model_loaded.compile(optimizer=optimizer, loss=loss, metrics=metrics)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Visualize predictions on the cross-validation test data"
    ]
   },
   {
@@ -829,19 +841,7 @@
    "source": [
     "# Use model to predict test labels\n",
     "Y_hat_test = model_loaded.predict(X_test, verbose=1)\n",
-    "print(Y_hat_test.shape, Y_hat_test.dtype)\n",
-    "Y_hat_train = model_loaded.predict(X_train, verbose=1)\n",
-    "print(Y_hat_train.shape, Y_hat_train.dtype)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "colab_type": "text",
-    "id": "Z8RFkbUXePUj"
-   },
-   "source": [
-    "## Visualize predictions on the cross-validation test data"
+    "print(Y_hat_test.shape, Y_hat_test.dtype)"
    ]
   },
   {
@@ -883,6 +883,16 @@
     "## Visualize predictions on the training data"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "Y_hat_train = model_loaded.predict(X_train, verbose=1)\n",
+    "print(Y_hat_train.shape, Y_hat_train.dtype)"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,

diff --git a/predict.py b/predict.py
@@ -12,9 +12,14 @@
 try:
     size = int(sys.argv[1])
     px, py = (size,size)
+
 except IndexError:
     px, py = (512,512)
 
+try:
+    threshold = (int(sys.argv[2])/100)
+except IndexError:
+    threshold = 0.5
 
 x, y = ((1680-px)/2, (1050-py)/2)    #set to your screen resolution
 model_loaded = keras_model(img_width=px, img_height=py)
@@ -33,7 +38,7 @@
 
             ary_hat = model_loaded.predict(np.stack([ary]))[0][:,:,0]
             ary_hat = np.array(ary_hat*255, dtype=np.uint8)  #rescale from float32 to uint8
-            ret,thresh = cv2.threshold(ary_hat, 128, 255, 0) #set threshold as over 0.5 (255/2=128)
+            ret,thresh = cv2.threshold(ary_hat, int(255*threshold), 255, 0) #set threshold as over 0.5 (255/2=128)
             img, contours, hierarchy = cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
             cv2.imshow('Live building detector', cv2.drawContours(ary, contours, -1, (0,255,0), 2)[:,:,::-1])