Install script for panoptes dependencies #129

jamessynge · 2017-11-24T20:12:51Z

This installs:

Linux packages
Mongodb (including starting the service, but not setting up the panoptes db)
Anaconda (miniconda)
Astrometry (builds software and fetches indices)
Python packages (pip)

Has code for building and installing latest version of cfitsio, but that is currently disabled in favor of using the latest linux package for cfitsio. Avoiding the build because I was finding some problems with using astrometry.

Partially addresses #115. Would still like to include clone of git repos, and properly configuring linux for ssh, mongodb for security, etc.

Wilfred who is doing a similar exercise with the Raspberry Pi.

…ls so far because fitsio headers can't be found. Also added download of index files.

and command-line flags to disable each of the various types of dependencies from beig installed. Still need to remove excess crap at end of file, and to resolve a problem running pytest.

Ready for review AND testing (perhaps in a docker container).

codecov · 2017-11-24T20:28:19Z

Codecov Report

Merging #129 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop     #129   +/-   ##
========================================
  Coverage    84.39%   84.39%           
========================================
  Files           35       35           
  Lines         2345     2345           
  Branches       281      281           
========================================
  Hits          1979     1979           
  Misses         285      285           
  Partials        81       81

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2d4e906...5d34a22. Read the comment docs.

Add libcfitsio-bin to apt packages.

jamessynge · 2017-11-24T21:12:25Z

Another TODO: it would be nice to not install astrometry if it hasn't changed, but they don't seem to expose a convenient way to determine the installed version. Sigh. May work around by writing a version number file from the install script.

wtgee

I have not actually run this yet but am heading over to lab so will try it soon.

wtgee · 2017-11-24T21:31:24Z

.travis.yml

@@ -59,7 +59,7 @@ install:
  - source activate test-environment

  - cd $TRAVIS_BUILD_DIR
-  - pip install -r requirements.txt
+  - pip install -r scripts/install/requirements.txt


Just a note to move requirements.txt back to root

wtgee · 2017-11-24T21:33:50Z

scripts/install/apt-packages-list.txt

+# dcraw decodes Canon raw images, and gphoto2 takes pictures with
+# Canon DSLRs, among others.
+dcraw
+gphoto2


gphoto2-updater is a nice install script that we might want to use if we need a newer version.

https://github.com/gonzalo/gphoto2-updater

The apt version might be simpler but I haven't had issue with the script.

I can add a todo, but if we're happy enough with the package version, I think we should stick with it for simplicity.

wtgee · 2017-11-24T21:35:02Z

scripts/install/apt-packages-list.txt

+# of these.
+# TODO(jamessynge): Discuss these with Wilfred as an additional way to get
+# temperature data.
+# lm-sensors


Not entirely sure we need this right now because as you say we need a reliable way to do it. Currently not being used although it was intended to be.

I guess would prefer to leave out of comments for now and maybe make a separate Issue

Added an issue.

wtgee · 2017-11-24T21:35:31Z

scripts/install/apt-packages-list.txt

+
+# These are used if you want to setup SSH access into the computer.
+# openssh-client
+# openssh-server


I would think we will always want this.

wtgee · 2017-11-24T21:36:56Z

scripts/install/apt-packages-list.txt

+# Graphviz is used for rendering the state machine of POCS.
+# TODO(wtgee): Are there any other uses?
+graphviz
+libgraphviz-dev


No other uses and sometimes the install can go wrong. We detect if installed and don't plot otherwise.

Really the state machine images only need to be generated once ever by any machine and then can be stored. We might just want to include the images themselves in the repo. Would become an issue if state configuration changed.

Removed TODO.

wtgee · 2017-11-24T21:57:28Z

scripts/install/install-dependencies.sh

+function install_mongodb() {
+  # This is based on https://www.howtoforge.com/tutorial/install-mongodb-on-ubuntu-16.04/
+  # Note this function does not configure mongodb itself, i.e. no users or
+  # security settings.


IIRC the default install only exposes on localhost. No authentication or authorization has been added yet.

wtgee · 2017-11-24T22:01:01Z

scripts/install/install-dependencies.sh

+[Service]
+User=mongodb
+Group=mongodb
+ExecStart=/usr/bin/mongod --quiet --config /etc/mongod.conf


We should also specify a --dbpath (or alter the mongod.conf). I've put this in $PANDIR/data previously. Our current setup writes a lot of entries to the database and it grows very large (usually needs > 4GB to run). On PAN001 all of $PANDIR is on a separate mount from the rest of /var (and everything else)

Note also that mongo does not require you to actually create a collection (read: database) before using it. You can just start writing to something and it will store properly

Not surprisingly, I didn't know that info about --dbpath.
Does mongodb allow multiple dbpath directories (i.e. can we arrange for our collections to be in /var/panoptes)?

No change made yet.

There is some info (for instance, here) but I think the better answer would be to run what we want in a container.

wtgee · 2017-11-24T22:02:37Z

scripts/install/install-dependencies.sh

+  local -r the_script="${PANDIR}/tmp/miniconda.sh"
+  echo_bar
+  echo
+  echo "Installing miniconda"


Just to be good sports we should mention here that there is a default acceptance of the license agreement.

Added link to license.

wtgee · 2017-11-24T22:05:48Z

scripts/install/install-dependencies.sh

+  # See http://astrometry.net/doc/readme.html#getting-index-files
+  # for info on the index files you might need.
+  if [[ "${DO_ASTROMETRY_FULL_INDICES}" -eq 1 ]] ; then
+    if [[ ! -f index-4107.fits ]] ; then


Just a note that if download/install gets interrupted this will not work properly second time around.

We also have $POCS/pocs/utils/data.py that does this same checking and is run before tests.

I notice that it installs more index files.

Do you have a suggestion regarding the interruption? Explicitly do the download of each?
Or just run $POCS/pocs/utils/data.py near the end of the install.

FWIW, I'd like to eliminate all downloading from tests.

The tests shouldn't download but they rely on having the files previously downloaded. As of right now they download here.

The astroplan IRES stuff still needs to be figured out better as per #121

wtgee · 2017-11-24T22:11:16Z

scripts/install/install-dependencies.sh

+fi
+
+
+if [[ "${DO_PIP_REQUIREMENTS}" -eq 1 ]] ; then


Let's actually create a separate panoptes environment in conda and use that. I've just started calling mine panoptes-env. So we will want to create it, install pip items there, and optionally set it as the default environment for the system (via ~/.profile). Something like:

$ conda create -y -n panoptes-env python=3 # Wait $ source activate panoptes-env (panoptes-env) $ pip install -U pip (panoptes-env) $ pip install -r requirements.txt (panoptes-env) $ echo "source activate panoptes-env" >> ~/.profile # Assume we always want

Note also we can create a conda environment file that can do all this and have the right collection of files. That is what I was doing in the consolidated branch but was having hassle. We should get this install script working first and then later worry about the environment file. But I do think we want to create/use a specific panoptes environment as outlined above.

Why do you say "# Wait" after conda create?

I just meant that it takes a minute and there is output between those two steps.

Done (well, no wait included).

wtgee · 2017-11-24T22:12:58Z

Another TODO: it would be nice to not install astrometry if it hasn't changed, but they don't seem to expose a convenient way to determine the installed version. Sigh. May work around by writing a version number file from the install script.

We should submit a patch to astrometry.net to do this. It annoys me when a --version flag outputs the --help.

wtgee · 2017-11-24T22:14:48Z

scripts/install/apt-packages-list.txt

+python-pyfits
+swig
+zlib1g-dev
+


Looks like byobu comes with Ubuntu server but not desktop. Let's put it in here to make sure we have it. It will pull in tmux also (I think we should default to byobu but will make comments in #97 )

Added byobu & tmux (explicitly).

Emit message reminding the user that PATH needs to be updated in current shell.

Create a conda environment for all the python packages, etc. Install python packages before building astrometry.net, as it uses numpy.

Create panoptes-env in conda.

jamessynge · 2017-11-25T17:15:37Z

Closing and re-opening PR to trigger another travis build. I reviewed the failure, and it smells like a problem interacting with temp files and subprocesses, rather than with the commits, so I want to see if it is reproducible as is.

jamessynge · 2017-11-25T18:22:15Z

Well, that was a false hope. The failure was essentially identical:

=================================== FAILURES ===================================
__________________________ test_solve_field_unsolved ___________________________
unsolved_fits_file = '/home/travis/build/panoptes/POCS/pocs/tests/data/unsolved.fits'
def test_solve_field_unsolved(unsolved_fits_file):
im0 = Image(unsolved_fits_file)
    assert isinstance(im0, Image)
    assert im0.wcs is None
    assert im0.pointing is None
  im0.solve_field(verbose=True, replace=False, radius=4)
pocs/tests/test_images.py:79:

pocs/images.py:174: in solve_field
**kwargs)

fname = '/home/travis/build/panoptes/POCS/pocs/tests/data/unsolved.fits'
replace = False, remove_extras = True
kwargs = {'dec': 44.456875, 'ra': 301.56439, 'radius': 4, 'verbose': True}
verbose = True, out_dict = {}
output = 'Reading input file 1 of 1: "/home/travis/build/panoptes/POCS/pocs/tests/data/unsolved.fits"...\nExtracting sources......er from file "/tmp/tmp.wcs.Wwe3cD" extension 0\n qfits:errfunc error: Failed to read FITS file "/tmp/tmp.wcs.Wwe3cD"\n'
errs = None, proc = <subprocess.Popen object at 0x7ffb3c771320>

wtgee · 2017-11-25T20:39:25Z

pocs/utils/data.py

            df = data.download_file(url)
            try:
                shutil.move(df, dest)
            except OSError as e:
                print("Problem saving. (Maybe permissions?): {}".format(e))

+    for i in range(4112, 4119):


Just reproduced the test failure on my local and it's because we are missing 4110 and 4111 index files. The fits files we use for testing have a small field-of-view, which is what those index files will search. 10 and 11 are fairly large (24MB and 10MB) so this is a bit annoying as we only really need them for the tests.

I also removed the 42xx from my local and all tests still pass. The 41xx are wide-field so should be all that we need although I suppose it doesn't hurt to have all of them.

OK. Can we replace the file used for testing to a wide field also?

jamessynge · 2017-11-25T22:26:11Z

I've discovered that writing to .profile didn't leave me with a working profile (activate was apparently in the wrong place). Will investigate further.

images that have narrow fields, unlike regular Panoptes images.

jamessynge added 6 commits November 19, 2017 17:23

wip - Starting on creating some setup scripts. Need to coordinate with

11d26a1

Wilfred who is doing a similar exercise with the Raspberry Pi.

wip - Install cfitsio.

8dc2835

wip - Added download, build and install of astrometry.net, but it fai…

2f539f8

…ls so far because fitsio headers can't be found. Also added download of index files.

wip - Fix creation of astrometry/data directory.

17e1a80

wip - Added functions for each type of dependenciy to be installed

9d06b89

and command-line flags to disable each of the various types of dependencies from beig installed. Still need to remove excess crap at end of file, and to resolve a problem running pytest.

Installation script for PANOPTES.

0dafb45

Ready for review AND testing (perhaps in a docker container).

jamessynge requested a review from wtgee November 24, 2017 20:12

Move install script and input files into POCS/scripts/install.

0cb3691

Fix path to requirements.txt.

1e688c2

Add libcfitsio-bin to apt packages.

And fix the requirements.txt path for travis too.

8d6062a

wtgee reviewed Nov 24, 2017

View reviewed changes

jamessynge added 5 commits November 24, 2017 17:47

Move requirements.txt back to root of repo.

a2cbb07

Emit message reminding the user that PATH needs to be updated in current shell.

Write paths to .profile instead of .bashrc.

8873b1b

Create a conda environment for all the python packages, etc. Install python packages before building astrometry.net, as it uses numpy.

Modify data.py to install both sets of index files.

d223d29

Create panoptes-env in conda.

Fix path in travis.yml.

b472eea

Remove trailing blank line.

384b27d

jamessynge closed this Nov 25, 2017

jamessynge reopened this Nov 25, 2017

wtgee reviewed Nov 25, 2017

View reviewed changes

Fetch more of the 4100 series indices, needed for some tests that use

5d34a22

images that have narrow fields, unlike regular Panoptes images.

wtgee mentioned this pull request Nov 26, 2017

Don't download IRES files every time tests are run #121

Closed

wtgee merged commit b400143 into panoptes:develop Nov 26, 2017

jamessynge deleted the install branch November 26, 2017 12:57

Install script for panoptes dependencies #129

Install script for panoptes dependencies #129

Conversation

jamessynge commented Nov 24, 2017

codecov bot commented Nov 24, 2017 • edited Loading

Codecov Report

jamessynge commented Nov 24, 2017

wtgee left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wtgee commented Nov 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jamessynge commented Nov 25, 2017

jamessynge commented Nov 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jamessynge commented Nov 25, 2017

codecov bot commented Nov 24, 2017 •

edited

Loading