NIRC2_Preprocessing_ streamlined.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#NIRC2 preprocessing streamlined"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Streamlined version of the notebook NIRC2_Preprocessing.\n",
    "\n",
    "Stable with vip v.0.4.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Import <a id='import'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%reset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from NIRC2_Preprocessing import *\n",
    "import numpy as np\n",
    "\n",
    "from vip.fits import vipDS9\n",
    "from os.path import join\n",
    "\n",
    "%matplotlib inline\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open a ds9 window"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ds9 = vipDS9()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Paths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define paths to RAW images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path_flat = '/Users/Olivier/Documents/ULg/VORTEX/Data/RAW/Keck/HR8799/20151024/flat/'\n",
    "path_sky = '/Users/Olivier/Documents/ULg/VORTEX/Data/RAW/Keck/HR8799/20151024/sky/'\n",
    "path_sci = '/Users/Olivier/Documents/ULg/VORTEX/Data/RAW/Keck/HR8799/20151024/sci/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define paths to PREPROCESSED images if already created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path_sky_preprocessed = '/Users/Olivier/Documents/ULg/VORTEX/Data/RAW/Keck/HR8799/20151024/processed/sky_flatted/msky.fits'\n",
    "path_sci_preprocessed = '/Users/Olivier/Documents/ULg/VORTEX/Data/RAW/Keck/HR8799/20151024/processed/sci_flatted/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the folder path where results are saved (if not required, use None). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path_output = '/Users/Olivier/Documents/ULg/VORTEX/Data/RAW/Keck/HR8799/20151024/processed/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z = open_fits(join(path_flat,'n0393.fits'))\n",
    "display_array_ds9(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If already created, you can load the preprocessed sky and sci images..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sky_preprocessed = load_images(path_sky_preprocessed)\n",
    "sci_preprocessed = load_images(path_sci_preprocessed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... if not, do you want to save the mflat and preprocessed images ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "save = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build the master flat and preprocess the sci and sky images: Let's run !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# File list\n",
    "filelist_flat = listing(path_flat, selection=False, ext='fits')\n",
    "filelist_sci = listing(path_sci, selection=False, ext='fits')\n",
    "filelist_sky = listing(path_sky, selection=False, ext='fits')\n",
    "\n",
    "# Let's go\n",
    "mflat, bad_pixel_map = masterFlat(filelist_flat, save=save, bpm=True, path_output=path_output, verbose=True, filtering=5)\n",
    "sci_preprocessed = applyFlat(filelist_sci, mflat, save=save, path_output=path_output, verbose=True)\n",
    "sky_preprocessed = applyFlat(filelist_sky, mflat, save=save, path_output=path_output, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optional : build the master sky. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "filename = 'msky'\n",
    "\n",
    "sky_preprocessed = master(listing(join(path_output,'sky_flatted')), norm=False, save=save, filename=filename, \n",
    "                          path_output=join(path_output,'sky_flatted'), verbose=True, filtering=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vortex position"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the sci and sky preprocessed images to estimate the position of a dust and the vortex."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ds9.display(sci_preprocessed,sky_preprocessed, mflat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the estimates for the dust and vortex positions. In addition, we define both the dust and vortex model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "dust_center = (599,605)\n",
    "dust_box_size = 41\n",
    "dust_fun = gauss2d_sym\n",
    "dust_fun_parameters = [5]\n",
    "\n",
    "vortex_center = (527,730)\n",
    "vortex_box_size = 28\n",
    "vortex_fun = gauss2d_sym\n",
    "vortex_fun_parameters = [5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run ! If **full_output** is True, the vortex_center_from_dust_signature() function will also return:\n",
    "- The positions of the vortex in the sci images\n",
    "- The positions of the dust in the sci images\n",
    "- The positions of the vortex in the sky images\n",
    "- The positions of the dust in the sky images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dust_options = {'center':dust_center, 'size':vortex_box_size, 'fun':dust_fun, 'parameters':dust_fun_parameters}\n",
    "vortex_options = {'center':vortex_center, 'size':vortex_box_size, 'fun':vortex_fun, 'parameters':vortex_fun_parameters}\n",
    "\n",
    "center_all = vortex_center_from_dust_signature(sci_preprocessed, sky_preprocessed, dust_options, vortex_options, \n",
    "                                               verbose=True, full_output=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Registration and cropping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the size of the final cube. If **None**, the optimal size is automatically calculated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "size = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cube_reg = cube_registration(sci_preprocessed, center_all, cube_output_size=size, save=True, \n",
    "                             verbose=True, path_output=path_output, filename='toto')\n",
    "\n",
    "ds9.display(sci_preprocessed, cube_reg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Bad pixel removal\n",
    "\n",
    "Under construction ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##ADI or RDI PCA "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from vip.pca.pca_local import pca_adi_annular_quad as annular_pca\n",
    "from vip.pca import pca\n",
    "from vip.fits import vipDS9, write_fits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the parallactic angles.\n",
    "\n",
    "**NOTE**: If *path_sci_preprocessed* has been used instead of *path_sci*, modify *path_sci* in the argument of **get_parallactic_angles** into *path_sci_preprocessed*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "angs = get_parallactic_angles(listing(path_sci), save=True, path_output=path_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the reference cube for RDI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path_ref_for_RDI = '/Users/Olivier/Documents/ULg/VORTEX/Data/Cube_PSF_PA/Keck/HR8799/20150609/cube_HD219196_20150609.fits'\n",
    "cube_ref = open_fits(path_ref_for_RDI)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "image = pca(cube_reg, angs, cube_ref=None, ncomp=3, svd_mode='randsvd', full_output=False)\n",
    "ds9.display(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to perform **annular_pca( )** instead of normal **pca()**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "image = annular_pca(cube_reg, angs, ncomp=5, fwhm=8, svd_mode='randsvd', full_output=False)\n",
    "ds9.display(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "filename = 'final_image.fits'\n",
    "write_fits(join(path_output,filename), image)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}