Fix install-dependencies.sh for Ubuntu 18.04 #585

jamessynge · 2018-09-09T02:05:48Z

Update for conda 4.4
Avoid astroscrappy build problem by requiring Python 3.6
Make the install process quieter by logging the output of
the astrometry make commands, instead flooding the terminal
with content most folks won't care about.
Add some more context messages to the output.
Create an logs/install directory specific to this install.- Move all of the command line arg handling to the start of the
file.
Add comments to most functions.
Apply most fixes suggested by shellcheck, e.g. quoting
variable expansions and replacing fgrep with grep -F.
Insert changes into the appropriate file used by the shell
at login.

- Move all of the command line arg handling to the start of the file. - Add comments to most functions. - Apply most fixes suggested by shellcheck, e.g. quoting variable expansions and replacing fgrep with grep -F. - Insert changes into the appropriate file used by the shell at login. - Update for conda 4.4 - Avoid astroscrappy build problem by requiring Python 3.6 - Make the install process quieter by logging the output of the astrometry make commands, instead flooding the terminal with content most folks won't care about. - Add some more context messages to the output. - Create an logs/install directory specific to this install.

codecov · 2018-09-09T02:28:27Z

Codecov Report

Merging #585 into develop will decrease coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop     #585      +/-   ##
===========================================
- Coverage    70.45%   70.43%   -0.02%     
===========================================
  Files           64       64              
  Lines         5520     5520              
  Branches       769      769              
===========================================
- Hits          3889     3888       -1     
- Misses        1423     1425       +2     
+ Partials       208      207       -1

Impacted Files	Coverage Δ
pocs/camera/camera.py	`89.72% <0%> (-1.09%)`	⬇️
pocs/serial_handlers/protocol_arduinosimulator.py	`75.76% <0%> (+0.38%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 29d420f...18126b5. Read the comment docs.

It may often be possible to refresh the installation without logging out and back in, but sometimes things are different enough that that would be best (or at least the easiest way to get a good environment).

jamessynge · 2018-09-09T19:59:16Z

This script doesn't quite work right w.r.t. detecting the presence of conda due to conda/conda#7753. I'll workaround that by checking for the environment variables that they do export.

See: conda/conda#7753

Use a checksum of the file listing to determine whether the conda packages (including pip) have changed. Approximately correct.

jamessynge · 2018-09-09T23:50:47Z

After a ton of experiments and commits, this is ready for review. @wtgee Could you handle that?

jamessynge · 2018-09-09T23:52:06Z

Oh, note that it is currently configured to install the "latest" version of astrometry (i.e. not stable). Let me know if you want me to return it to 0.72 or to the current latest, 0.76.

wtgee

Approving with minor comments although I'd still like to test this out.

One thing we are not using is an Anaconda environment, which we could save as a single yaml file and then host on the panoptes Anaconda channel. I haven't looked into that much but might be worth exploring (for future PR).

wtgee · 2018-09-10T01:31:24Z

scripts/install/apt-packages-list.txt

+
+# Want git for upgrading to the latest version. Probably already installed
+# but not necessarily (i.e. it is possible to download a zip of POCS, which
+# includes this file).


Actually, I've wondered if we should skip git and just install the zip. An idea for later if we are finding we want to force atomic updates or something like that.

Hmm, an interesting idea. The downside is that it makes it harder to keep track of what is installed.

wtgee · 2018-09-10T11:28:24Z

scripts/install/install-dependencies.sh

+    echo
+    echo "Creating conda environment 'panoptes-env' with Python 3.6"
+    # 3.6 works around a problem building astroscrappy in 3.7.
+    conda create -n panoptes-env --yes --quiet python=3.6


Seems like the python version might be better stored in a variable somewhere else? It's 3.6 for now but will certainly change at some point and this is buried deep in the script.

Good point, will do.

wtgee · 2018-09-10T21:27:32Z

scripts/install/install-dependencies.sh

+    # We need to know the version that is in the tar file in order to
+    # enter that directory.
+    ASTROMETRY_VERSION=""
+    for version in $(find astrometry.net-* -maxdepth 0 -type d | sed "s/astrometry.net-//")


Can't we just rename the directory when we untar or wget it?

You're thinking of the name of the tar file, while this is the name of the top-level directory inside the tar file, which isn't known when we download "latest".

wtgee · 2018-09-10T21:27:56Z

scripts/install/install-dependencies.sh

  echo "Fetching astrometry into directory $(pwd)"
-
  wget "${URL}"


Rename dir here with -O flag. Comment below.

wtgee · 2018-09-10T21:30:05Z

scripts/install/install-dependencies.sh

 function install_astrometry_indices() {
+  echo_bar
+  echo
+  echo "Fetching astrometry and other indices."


Don't we still somewhere need to tell the astrometry.cfg file about the data dir? Or does it default to data inside it's base dir?

I don't recall needing to do that.

wtgee · 2018-09-10T21:31:12Z

scripts/install/install-dependencies.sh

 if [[ "${DO_APT_GET}" -eq 1 ]] ; then
  install_apt_packages
 fi

-
+# Install Mongodb, a document database. Used by POCS for storing environment
+# readings, from which weather plots are generated.


Also stores observations metadata. Not really important but comment undersells how much we use it to some extent.

wtgee · 2018-09-10T21:39:52Z

scripts/install/install-dependencies.sh

 fi

-
+# Install cfitsio, native tools for reading and writing FITS files.


Did cfitsio move to a conda package and if so does it need a separate option for installing? Would we ever not install cfitsio?

We now get it via apt-get, hence the DO_CFITSIO=0 at the top of the file. Ideally we'd use conda.

jamessynge

New commit TBD.

jamessynge · 2018-09-10T23:10:26Z

scripts/install/apt-packages-list.txt

+
+# Want git for upgrading to the latest version. Probably already installed
+# but not necessarily (i.e. it is possible to download a zip of POCS, which
+# includes this file).


Hmm, an interesting idea. The downside is that it makes it harder to keep track of what is installed.

jamessynge · 2018-09-10T23:10:54Z

scripts/install/install-dependencies.sh

+    echo
+    echo "Creating conda environment 'panoptes-env' with Python 3.6"
+    # 3.6 works around a problem building astroscrappy in 3.7.
+    conda create -n panoptes-env --yes --quiet python=3.6


Good point, will do.

jamessynge · 2018-09-10T23:55:12Z

scripts/install/install-dependencies.sh

+    # We need to know the version that is in the tar file in order to
+    # enter that directory.
+    ASTROMETRY_VERSION=""
+    for version in $(find astrometry.net-* -maxdepth 0 -type d | sed "s/astrometry.net-//")


You're thinking of the name of the tar file, while this is the name of the top-level directory inside the tar file, which isn't known when we download "latest".

jamessynge · 2018-09-10T23:57:32Z

scripts/install/install-dependencies.sh

 function install_astrometry_indices() {
+  echo_bar
+  echo
+  echo "Fetching astrometry and other indices."


I don't recall needing to do that.

jamessynge · 2018-09-10T23:58:42Z

scripts/install/install-dependencies.sh

 if [[ "${DO_APT_GET}" -eq 1 ]] ; then
  install_apt_packages
 fi

-
+# Install Mongodb, a document database. Used by POCS for storing environment
+# readings, from which weather plots are generated.


jamessynge · 2018-09-10T23:59:58Z

scripts/install/install-dependencies.sh

 fi

-
+# Install cfitsio, native tools for reading and writing FITS files.


We now get it via apt-get, hence the DO_CFITSIO=0 at the top of the file. Ideally we'd use conda.

wtgee · 2018-09-11T22:42:11Z

What files does this depend on? Ideally I want to wget this file and run it and do nothing else but I guess it's not quite to that point.

Pulled down the file. Env variables weren't set. Pulled down the default-env-vars.sh. Running it doesn't actually set my env vars, so I have to manually do that before I can run script. I know we want to clean up env vars otherwise but seems like install script should set them up if they don't exist.

wtgee · 2018-09-11T22:48:45Z

Because some of this runs as sudo it requires that the env variables be set up for the sudo user. Not sure if that is intended or needed.

Edit: Requires correct permissions set on $PANDIR for $PANUSER. Should be explicit.

panoptes@pan008-nuc:~/Downloads$ head install-dependencies.sh 
#!/bin/bash -e

# Run with --help to see your options. With no options, does a complete install
# of dependencies, though attempts to reuse existing installs.

THIS_DIR="$(dirname "$(readlink -f "${0}")")"
THIS_PROGRAM="$(basename "${0}")"

if [[ -z "${PANDIR}" || -z "${POCS}" || -z "${PAWS}" || -z "${PANLOG}" ||
      -z "${PANUSER}" ]] ; then

panoptes@pan008-nuc:~/Downloads$ echo $PANDIR
/var/panoptes

panoptes@pan008-nuc:~/Downloads$ echo $POCS
/var/panoptes/POCS

panoptes@pan008-nuc:~/Downloads$ echo $PAWS
/var/panoptes/PAWS

panoptes@pan008-nuc:~/Downloads$ echo $PANLOG
/var/panoptes/logs

panoptes@pan008-nuc:~/Downloads$ echo $PANUSER
panoptes

panoptes@pan008-nuc:~/Downloads$ ./install-dependencies.sh 
mkdir: cannot create directory ‘tmp’: Permission denied

panoptes@pan008-nuc:~/Downloads$ sudo ./install-dependencies.sh 
Please set the Panoptes environment variables, then re-run this script.

panoptes@pan008-nuc:~/Downloads$

Generate /var/panoptes/set-panoptes-env.sh, and source that from the rc file of the user's shell (e.g. $HOME/.bashrc). Explicitly choose astrometry version 0.76 because testing with "latest" as the version causes lots of downloads. Stop suggesting that the user logout and log back in, and instead recommend sourcing the generated shell setup file. Not yet done with conda related updates. For one thing, if --no-conda is specified, the remaining conda updates may fail. Sigh. I may want to change what --no-conda means.

We're now using the version we can get from apt-get, so removing this support for the sake of reducing complexity.

Ensure that conda.sh is sourced where needed, repeating if necessary.

Should have escaped the $ in $PATH so that the literal string "$PATH" was added, not the current value of PATH.

the setup script is executed, not the value when the install script is executed.

If we have the user "run python setup.py install", without appending "; pytest", then once they run the first command they'll have lost sight of the second command to run.

jamessynge · 2018-09-12T22:17:45Z

Our Intel NUC Setup doc describes how to run this, including how to setup the environment variables.

My intent though is to eventually have a "setup computer" script, that is used right after creating the first user (e.g. "james") on the computer, and takes care of creating the "panoptes" user, sets up the cron jobs, etc., leaving you with a system that is ready to go... though we'd need some second action that actually enables POCS to run automatically. This new script would certainly be derived from this one.

wtgee · 2018-09-13T05:22:59Z

Our Intel NUC Setup doc describes how to run this, including how to setup the environment variables.

And I apologize, I should have followed the docs while doing this. Let's consider my testing to be that of someone who thinks they know better so doesn't have to read the instructions so tries anyway. :)

I'm going to test out newest version on NUC later today and will follow instructions.

Agree that the fully automated one will be in future.

wtgee · 2018-09-14T00:32:22Z

Tried latest version on clean install while following docs and worked well.

One note, the install script gives instructions to follow at the end, which I did. This is slightly different than what the docs say. I made a comment in the docs.

jamessynge requested a review from a team September 9, 2018 02:05

jamessynge added 3 commits September 9, 2018 11:09

Only download, build and install astrometry if necessary

7ae70d5

Warn user when the profile or conda has changed

6207cf4

It may often be possible to refresh the installation without logging out and back in, but sometimes things are different enough that that would be best (or at least the easiest way to get a good environment).

Fix up final message

aa2d84d

jamessynge mentioned this pull request Sep 9, 2018

Conda command not exported to subshells conda/conda#7753

Open

jamessynge added 3 commits September 9, 2018 17:29

Workaround conda being a non-exported shell function

3ad92ba

See: conda/conda#7753

Take a different approach to detecting conda changes

1a52666

Use a checksum of the file listing to determine whether the conda packages (including pip) have changed. Approximately correct.

Fix typo

aee048a

wtgee approved these changes Sep 10, 2018

View reviewed changes

jamessynge commented Sep 11, 2018

View reviewed changes

WIP

ee21f43

jamessynge added 6 commits September 11, 2018 20:01

Remove support for building cfitsio from scratch

93dc8d6

We're now using the version we can get from apt-get, so removing this support for the sake of reducing complexity.

Fix how conda is found

5665796

Ensure that conda.sh is sourced where needed, repeating if necessary.

Fix typo in generate script

68da834

Should have escaped the $ in $PATH so that the literal string "$PATH" was added, not the current value of PATH.

Escape $PANDIR so that we use the variable's value when

5747cd3

the setup script is executed, not the value when the install script is executed.

Combine recommended commands on one line

18126b5

If we have the user "run python setup.py install", without appending "; pytest", then once they run the first command they'll have lost sight of the second command to run.

jamessynge merged commit d39379a into panoptes:develop Sep 14, 2018

jamessynge deleted the tidy-install-dependencies branch September 14, 2018 15:09

wtgee mentioned this pull request Oct 16, 2018

install-dependencies.sh script fails on Ubuntu 18.04 LTS #536

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix install-dependencies.sh for Ubuntu 18.04 #585

Fix install-dependencies.sh for Ubuntu 18.04 #585

jamessynge commented Sep 9, 2018

codecov bot commented Sep 9, 2018 •

edited

Loading

jamessynge commented Sep 9, 2018

jamessynge commented Sep 9, 2018

jamessynge commented Sep 9, 2018

wtgee left a comment

wtgee Sep 10, 2018

jamessynge Sep 10, 2018

wtgee Sep 10, 2018

jamessynge Sep 10, 2018

wtgee Sep 10, 2018

jamessynge Sep 10, 2018

wtgee Sep 10, 2018

wtgee Sep 10, 2018

jamessynge Sep 10, 2018

wtgee Sep 10, 2018

jamessynge Sep 10, 2018

wtgee Sep 10, 2018

jamessynge Sep 10, 2018

jamessynge left a comment

jamessynge Sep 10, 2018

jamessynge Sep 10, 2018

jamessynge Sep 10, 2018

jamessynge Sep 10, 2018

jamessynge Sep 10, 2018

jamessynge Sep 10, 2018

wtgee commented Sep 11, 2018

wtgee commented Sep 11, 2018 •

edited

Loading

jamessynge commented Sep 12, 2018

wtgee commented Sep 13, 2018

wtgee commented Sep 14, 2018 •

edited

Loading

		echo "Fetching astrometry into directory $(pwd)"

		wget "${URL}"

		fi


		# Install cfitsio, native tools for reading and writing FITS files.

Fix install-dependencies.sh for Ubuntu 18.04 #585

Fix install-dependencies.sh for Ubuntu 18.04 #585

Conversation

jamessynge commented Sep 9, 2018

codecov bot commented Sep 9, 2018 • edited Loading

Codecov Report

jamessynge commented Sep 9, 2018

jamessynge commented Sep 9, 2018

jamessynge commented Sep 9, 2018

wtgee left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jamessynge left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wtgee commented Sep 11, 2018

wtgee commented Sep 11, 2018 • edited Loading

jamessynge commented Sep 12, 2018

wtgee commented Sep 13, 2018

wtgee commented Sep 14, 2018 • edited Loading

codecov bot commented Sep 9, 2018 •

edited

Loading

wtgee commented Sep 11, 2018 •

edited

Loading

wtgee commented Sep 14, 2018 •

edited

Loading