Polar Alignment - Solve Failure

[wtgee]

When doing the polar alignment, if the pole image fails to solve the whole procedure gets stuck.

/home/panoptes/anaconda3/lib/python3.6/subprocess.py:761: ResourceWarning: subprocess 31739 is still running
  ResourceWarning, source=self)
/home/panoptes/anaconda3/lib/python3.6/subprocess.py:761: ResourceWarning: subprocess 31753 is still running
  ResourceWarning, source=self)
Moving back to home
2017-11-11 18:06:01 Panmass root[31381] INFO PANCHAT Moving back to home
Solving celestial pole image
2017-11-11 18:06:01 Panmass root[31381] INFO PANCHAT Solving celestial pole image
2017-11-11 18:06:13 Panmass panoptes[31381] ERROR SolveError: File not solved
Traceback (most recent call last):
  File "bin/pocs_shell", line 795, in <module>
    PocsShell().cmdloop()
  File "/home/panoptes/anaconda3/lib/python3.6/cmd.py", line 138, in cmdloop
    stop = self.onecmd(line)
  File "/home/panoptes/anaconda3/lib/python3.6/cmd.py", line 217, in onecmd
    return func(arg)
  File "bin/pocs_shell", line 287, in do_polar_alignment_test
    pole_center = polar_alignment.analyze_polar_rotation(pole_fn)
  File "/var/panoptes/PIAA/piaa/utils/polar_alignment.py", line 21, in analyze_polar_rotation
    img_utils.get_solve_field(pole_fn)
  File "/var/panoptes/POCS/pocs/utils/images.py", line 148, in get_solve_field
    raise error.SolveError('File not solved')
pocs.utils.error.SolveError: File not solved

[wtgee]

Reopening per @jamessynge question in #112

We need to figure out why solves are failing or perhaps have a mechanism to retry a few times. Sometimes the solver does just fail with a success on repeat.

[wtgee]

@jamessynge Are you satisfied with this? For the polar alignment test it might make sense to retry the solve a few times. During normal operations I don't think we want to do that however as it takes time and, as of now, solving is not required for operation.

If we want to incorporate the retries attempt we should probably bundle it with another issue that relates to splitting the polar alignment from the peas_shell

[jamessynge]

Yes, this is now resolved after:

1. Upgrading installation of astrometry.
2. Fixing the location of the index files.
3. Removing spurious, empty directory $PANDIR/POCS/astrometry, which was throwing off the Python module search.
4. Adding a $PANDIR/POCS/matplotlibrc file with backend:Agg, thus fixing the rendering of the overlay.

[wtgee]

Adding a $PANDIR/POCS/matplotlibrc file with backend:Agg, thus fixing the rendering of the overlay.

Just want to link the above with #119