Add scripts/install/auto-install.sh #824

jamessynge · 2019-04-15T15:17:56Z

Add script which can be used to install POCS and dependencies via a single command:

wget -O - https://raw.githubusercontent.com/panoptes/POCS/develop/scripts/install/auto-install.sh | bash

Description

The script does the following:

Installs git if it isn't already installed
Creates /var/panoptes with the appropriate ownership.
Clones POCS if it isn't already cloned, else does a git pull to refresh it
Runs scripts/install/install-dependencies.sh

Related Issue

#531

How Has This Been Tested?

Tested by repeatedly setting up PAN005 for Doug and Christina.

Screenshots (if appropriate):

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x ] My code follows the code style of this project.
[ x] My change requires a change to the documentation.
I have updated the documentation accordingly.
[ x] I have read the CONTRIBUTING document.
I have added tests to cover my changes.

The intent is for the user to execute this command AFTER creating the panoptes user and logging in as the panoptes user: bash <(curl -s https://raw.githubusercontent.com/panoptes/POCS/develop/scripts/install/auto-install.sh) We can later augment this to create the panoptes user and then do the rest of the setup as that user.

Provide two different suggestions for how to download and execute. Allow for overriding of the github repo url and branch, as this will make it easier to test this script.

PAWS is expected by install-dependencies.sh, so installing it will ease the life of the builder.

codecov · 2019-04-15T15:41:09Z

Codecov Report

Merging #824 into develop will increase coverage by 0.02%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop     #824      +/-   ##
===========================================
+ Coverage    81.24%   81.27%   +0.02%     
===========================================
  Files           66       68       +2     
  Lines         5513     5516       +3     
  Branches       758      759       +1     
===========================================
+ Hits          4479     4483       +4     
- Misses         836      837       +1     
+ Partials       198      196       -2

Impacted Files	Coverage Δ
pocs/utils/images/fits.py	`80.3% <0%> (-0.2%)`	⬇️
pocs/utils/logger.py	`98.24% <0%> (ø)`	⬆️
pocs/camera/simulator.py
pocs/camera/simulator/dslr.py	`98.07% <0%> (ø)`
pocs/camera/simulator/__init__.py	`100% <0%> (ø)`
pocs/camera/simulator/ccd.py	`100% <0%> (ø)`
pocs/camera/camera.py	`92.77% <0%> (+0.05%)`	⬆️
pocs/utils/config.py	`92.42% <0%> (+0.11%)`	⬆️
pocs/camera/__init__.py	`72.81% <0%> (+0.26%)`	⬆️
pocs/serial_handlers/protocol_arduinosimulator.py	`99.6% <0%> (+0.39%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 52b43e2...f742e52. Read the comment docs.

wtgee · 2019-04-15T20:45:17Z

scripts/install/auto-install.sh

+
+[[ -z "${POCS_GITHUB_USER}" ]] && export POCS_GITHUB_USER="panoptes"
+[[ -z "${POCS_GIT_URL}" ]] && export POCS_GIT_URL="https://github.com/${POCS_GITHUB_USER}/POCS.git"
+[[ -z "${POCS_BRANCH}" ]] && export POCS_BRANCH="develop"


I think this is fine, but I have been thinking lately that we should go back to pushing release versions to master and having non-developer units always pull from master. Mostly the onus here would be on me to actually be good about making releases.

I think that might be a good idea to use master that way. I.e. on rare occasions, and after testing on multiple units, we declare a new version that non-developers should use.

scripts/install/auto-install.sh

wtgee · 2019-04-15T20:57:40Z

scripts/install/auto-install.sh

+echo "
+Ensuring that ${PANDIR} is owned by user ${PANUSER}
+"
+do_sudo chown "${PANUSER}" "${PANDIR}"


Suggested change

do_sudo chown "${PANUSER}" "${PANDIR}"

do_sudo chown -R "${PANUSER}:${PANUSER}" "${PANDIR}"

I added the recursive option because I was thinking the subdirectories existed at this point but it looks like not but the option shouldn't hurt. Also not sure if wanted the group changed or not, I usually default to changing both.

I tried that a while ago, even had it in this code earlier today, but then I remembered that it was really very slow if the full tree was already there. I can reconsider/retest if you like.

Interesting. It seems like there shouldn't be that many files in the tree. But okay not doing it. That means are assuming if it already exists then the user has already set their permissions accordingly, which I think is fine.

I added it, and it worked fine. I now recall that the slow case was inside of docker.

wtgee · 2019-04-15T20:58:48Z

scripts/install/auto-install.sh

+do_sudo chown "${PANUSER}" "${PANDIR}"
+
+clone_or_update "${POCS}" "${POCS_GIT_URL}" "${POCS_BRANCH}"
+clone_or_update "${PAWS}" "https://github.com/panoptes/PAWS.git" "develop"


Could obviously add the ability to clone a forked version of PAWS but I suppose not entirely necessary at this point. Can leave until we overhaul PAWS, which needs to happen soonish.

wtgee · 2019-04-15T21:01:17Z

scripts/install/install-dependencies.sh

@@ -574,7 +574,10 @@ function maybe_install_conda() {
 function get_installed_astrometry_version() {
  local -r solve_field="${ASTROMETRY_DIR}/bin/solve-field"
  if [[ -x "${solve_field}" ]] ; then
-    "${solve_field}" --help|(grep -E '^Revision [0-9.]+,' || /bin/true)|cut -c10-|cut -d, -f1
+    ("${solve_field}" --help \


I use $SOLVE_FIELD elsewhere to point to the solve-field executable. Here it is just checking a revision number, correct? (Also, I keep meaning to submit a PR to astrometry.net to add a simple --version flag. Very annoying it doesn't have one.)

Yup, just checking version number.

wtgee

This looks nice and simple, which is good. It looks like this assumes everything is set up w.r.t. the user, in particular membership in the dialout and/or plugdev groups, etc.

wtgee · 2019-04-25T21:19:59Z

@jamessynge what's the plan with this? Should I merge?

Set prompt correctly (with panoptes-env) in interactive shells.

Move various helper functions from install-dependencies.sh into install-helper-functions.sh. Remove unused functions from install-dependencies.sh.

jamessynge · 2019-05-05T16:32:10Z

PTAL.

I've tested/adjusted this repeatedly, trying to streamline the process, and improve the experience for folks that aren't Linux experts. For example, it now invokes the set-panoptes-env.sh script twice, once at the start of .bashrc, and again at the end. This ensures that non-interactive shells are configured with the desired environment (e.g. for crontab), and also updates the prompt for interactive shells.

and for $HOME/.cache. The latter because I found that if it doesn't exist, some install step creates it as owned by root:root, breaking later install steps.

setup.py and pytest at the end of auto-install.sh. Could also consider doing so in install-dependencies.sh.

…y it.

during install-dependencies.sh (defaults to doing so).

Don't check if stdin is a terminal when attempting to get column width. I think this messes with some of the methods which can probe stdout.

Remove unintended indentation in message.

Don't require user to hit enter during gcloud install.

wtgee

👍

wtgee · 2019-05-05T22:48:25Z

requirements.txt

@@ -22,4 +22,8 @@ scikit_image >= 0.12.3
 scipy >= 0.17.1
 transitions >= 0.4.0
 tweepy
+
+# requests 2.21 doesn't like urllib3 >=1.25
+urllib3 >= 1.24, < 1.25  


I wish we could somehow put a timer on this. urllib3 1.25.x is only about two weeks old, so I expect this will resolve itself shortly.

Of course, there are not really any features in 1.25 that we require so should be okay.

wtgee · 2019-05-05T22:50:49Z

scripts/install/apt-packages-list.txt

@@ -78,3 +78,6 @@ coreutils
 # includes this file).
 git

+# Installing some POCS dependencies (esp. astroscrappy) require a C compiler,


I think astroscrappy is required by ccdproc, which is still in develop but which I think I have removed from, i.e. #279. We don't really use it for POCS (we might for PIAA).

Not terrible to have build-essential here, but if we use this script to build Docker images later we will want to minimize this as it causes bloat.

Acknowledged.

wtgee · 2019-05-05T22:56:39Z

scripts/install/auto-install.sh

+################################################################################
+# Clone the POCS repo from github.
+
+clone_or_update "${POCS}" https://github.com/panoptes/POCS.git upstream develop


Do you want the upstream here? Seems like it is not set anywhere above if I'm following correctly. You do set it right below this.

Yes. This is used as the origin name when cloning, so that https://github.com/panoptes/POCS.git will be the upstream repo, and https://github.com/${POCS_GITHUB_USER}/POCS.git will be the origin.

wtgee · 2019-05-05T22:58:01Z

scripts/install/auto-install.sh

+fi
+
+clone_or_update \
+    "${PAWS}" "https://github.com/panoptes/PAWS.git" upstream develop


Same comment about upstream.

Yup, that is intended.

wtgee · 2019-05-05T23:01:16Z

scripts/install/auto-install.sh

+# QUESTION: Should we add these lines here:
+# source $PANDIR/set-panoptes-env.sh
+# cd $POCS
+# python setup.py install


This should be set to develop instead of install.

wtgee · 2019-05-05T23:02:20Z

scripts/install/auto-install.sh

+${POCS}/scripts/install/install-dependencies.sh
+
+# QUESTION: Should we add these lines here:
+# source $PANDIR/set-panoptes-env.sh


What is the alternative? Set them manually?

Yes. The install-dependencies.sh script spits out some lines for the user to run by hand, and I'm suggesting that I could do so automatically as part of validating the install.

wtgee · 2019-05-05T23:07:18Z

scripts/install/auto-install.sh

+    cd "${REPO_DIR}"
+    git fetch --all
+    git checkout "${BRANCH}"
+    git pull


If they have already set up the repo they might have origin set to their repo and this pull might not necessarily be doing what we want. Or they might have files in staging, in which case this will fail.

I think if the dir is there and a repo we just do nothing other than maybe print a message reminding them to update. Thoughts?

Yeah, clearly if there are already files staged then we need to avoid messing with the repo.
Of course, if they have a repo, they don't need this auto-install script, and can run install-dependencies.sh themselves.

I'll adjust the script to stop if there is a POCS or PAWS repo.

wtgee · 2019-05-05T23:16:34Z

@jamessynge I wonder if we also shouldn't copy pocs.yaml to pocs_local.yaml somewhere in this process.

jamessynge · 2019-05-11T20:22:39Z

I've added some code for copying pocs.yaml to pocs_local.yaml, and reminding the user to customize it.

a message indicating how to proceed. Run setup.py install, and pytest after installing dependencies. Clone pocs.yaml, and remind user to modify pocs_local.yaml (the clone).

And fix it so that it actually sets the PATH! Fix install-dependencies.sh so that it is more likely to remove lines it added to .bashrc previously, even if slightly different.

jamessynge added 9 commits April 14, 2019 16:00

Fix name of default branch (develop, not master).

d4ebdf8

Provide two different suggestions for how to download and execute. Allow for overriding of the github repo url and branch, as this will make it easier to test this script.

Add cloning/pulling of PAWS

8f17431

PAWS is expected by install-dependencies.sh, so installing it will ease the life of the builder.

Re-enable calling install-dependencies.sh

b5a9f2d

Add some more whitespace, so it is easier to parse the output

7cdc970

Add a separator line (echo_bar) between major commands

52dd23a

Remove debugging option

41cdf85

Adapt to change in astrometry version string format.

c16558b

Correctly punctuate a multi-line pipe

d11f8fe

jamessynge requested a review from wtgee April 15, 2019 15:18

wtgee reviewed Apr 15, 2019

View reviewed changes

scripts/install/auto-install.sh Show resolved Hide resolved

wtgee reviewed Apr 15, 2019

View reviewed changes

scripts/install/auto-install.sh Outdated Show resolved Hide resolved

wtgee reviewed Apr 15, 2019

View reviewed changes

scripts/install/auto-install.sh Outdated Show resolved Hide resolved

wtgee reviewed Apr 15, 2019

View reviewed changes

scripts/install/auto-install.sh Outdated Show resolved Hide resolved

wtgee reviewed Apr 15, 2019

View reviewed changes

scripts/install/auto-install.sh Outdated Show resolved Hide resolved

wtgee reviewed Apr 15, 2019

View reviewed changes

wtgee approved these changes Apr 15, 2019

View reviewed changes

jamessynge added 2 commits April 15, 2019 19:56

Add install-gcloud.sh, which installs Google Cloud SDK

4031a81

Add additional documentation

9795b1e

jamessynge added 6 commits May 4, 2019 11:52

Add info about how create the PANUSER.

9efc869

Provide more guidance on how to run the script.

ef1cd69

Add missing fi

3f60c0d

Fix then

9627824

wip

13c2c91

fix redirects

452d8bc

jamessynge added 12 commits May 4, 2019 12:56

wip

1abd4e1

Set group owner of /var/panoptes

e1ed9cf

Try $COLUMNS for width of terminal.

32f3095

Try "tput cols" for width of terminal.

c4188fa

Add build-essentials dependency

8523041

Fix build-essential dependency

c5431c6

Add urllib3 to requirements.txt

35e8bdb

Set prompt correctly (with panoptes-env) in interactive shells.

Restrict version of urllib3

e8cdc37

Support both origin and upstream for POCS clone.

c84fdda

Fix bug in update_shell_rc_file.

f452e32

Move various helper functions from install-dependencies.sh into install-helper-functions.sh. Remove unused functions from install-dependencies.sh.

Set name of remote during clone.

42c290d

Add missing punctuation, fix comments.

1b14b96

jamessynge added 10 commits May 5, 2019 12:52

Add function ensure_directory_ownership, use for $PANDIR

db22f84

and for $HOME/.cache. The latter because I found that if it doesn't exist, some install step creates it as owned by root:root, breaking later install steps.

Add a question for Wilfred about whether to execute

a0e2ad2

setup.py and pytest at the end of auto-install.sh. Could also consider doing so in install-dependencies.sh.

Change format of an output string so that shellcheck isn't confused b…

e2753a1

…y it.

Add support for conditionally invoking install-gcloud.sh

02c3d1e

during install-dependencies.sh (defaults to doing so).

Remove verbose logging from install-apt-packages.sh.

8118344

Reformat message from echo_running_sudo.

96350bd

Don't check if stdin is a terminal when attempting to get column width. I think this messes with some of the methods which can probe stdout.

Replace use of curl with wget.

7d7b52d

Remove unintended indentation in message.

Make apt-get quiet.

3955798

Don't require user to hit enter during gcloud install.

Remove duplicate sources for the gcloud sdk.

433471f

Reformat sudo message.

732a2ca

wtgee self-requested a review May 5, 2019 22:43

wtgee approved these changes May 5, 2019

View reviewed changes

jamessynge added 2 commits May 11, 2019 16:27

Don't clone POCS or PAWS if it appears they exist; instead issue

0f77dc6

a message indicating how to proceed. Run setup.py install, and pytest after installing dependencies. Clone pocs.yaml, and remind user to modify pocs_local.yaml (the clone).

Change clean_PATH so it can be standalone.

f742e52

And fix it so that it actually sets the PATH! Fix install-dependencies.sh so that it is more likely to remove lines it added to .bashrc previously, even if slightly different.

jamessynge merged commit 5c83434 into panoptes:develop May 21, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add scripts/install/auto-install.sh #824

Add scripts/install/auto-install.sh #824

jamessynge commented Apr 15, 2019

codecov bot commented Apr 15, 2019 •

edited

Loading

wtgee Apr 15, 2019

jamessynge May 5, 2019

wtgee Apr 15, 2019

jamessynge May 5, 2019

wtgee May 5, 2019

jamessynge May 10, 2019 •

edited

Loading

wtgee Apr 15, 2019

jamessynge May 5, 2019

wtgee Apr 15, 2019

jamessynge May 5, 2019

wtgee left a comment

wtgee commented Apr 25, 2019

jamessynge commented May 5, 2019

wtgee left a comment

wtgee May 5, 2019

wtgee May 5, 2019

jamessynge May 10, 2019

wtgee May 5, 2019

jamessynge May 11, 2019

wtgee May 5, 2019

jamessynge May 11, 2019

wtgee May 5, 2019

wtgee May 5, 2019

jamessynge May 11, 2019

wtgee May 5, 2019

jamessynge May 11, 2019

wtgee commented May 5, 2019

jamessynge commented May 11, 2019

	do_sudo chown "${PANUSER}" "${PANDIR}"
	do_sudo chown -R "${PANUSER}:${PANUSER}" "${PANDIR}"

Add scripts/install/auto-install.sh #824

Add scripts/install/auto-install.sh #824

Conversation

jamessynge commented Apr 15, 2019

Description

Related Issue

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

codecov bot commented Apr 15, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jamessynge May 10, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wtgee left a comment

Choose a reason for hiding this comment

wtgee commented Apr 25, 2019

jamessynge commented May 5, 2019

wtgee left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wtgee commented May 5, 2019

jamessynge commented May 11, 2019

codecov bot commented Apr 15, 2019 •

edited

Loading

jamessynge May 10, 2019 •

edited

Loading