Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make startup scripts & downloading of files more robust #507

Merged

Conversation

jamessynge
Copy link
Contributor

Fix issue #506 by making the downloading code ignore failures to
download files.

Fix issue panoptes#506 by making the downloading code ignore failures to
download files.
@jamessynge jamessynge requested a review from wtgee March 24, 2018 20:51
@codecov
Copy link

codecov bot commented Mar 24, 2018

Codecov Report

Merging #507 into develop will decrease coverage by 0.16%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #507      +/-   ##
===========================================
- Coverage    70.48%   70.32%   -0.17%     
===========================================
  Files           62       62              
  Lines         5418     5418              
  Branches       751      752       +1     
===========================================
- Hits          3819     3810       -9     
- Misses        1392     1398       +6     
- Partials       207      210       +3
Impacted Files Coverage Δ
pocs/serial_handlers/protocol_arduinosimulator.py 75% <0%> (-2.31%) ⬇️
pocs/scheduler/scheduler.py 95.04% <0%> (-0.96%) ⬇️
pocs/dome/protocol_astrohaven_simulator.py 79.74% <0%> (-0.85%) ⬇️
pocs/dome/astrohaven.py 71.05% <0%> (-0.26%) ⬇️
pocs/observatory.py 81.86% <0%> (ø) ⬆️
pocs/scheduler/dispatch.py 98.03% <0%> (ø) ⬆️
pocs/state/states/default/scheduling.py 69.56% <0%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 064762f...4e22dca. Read the comment docs.

plot_weather.py writes into /var/panoptes/plot_weather/, but
there is nothing that creates that directory, so this commit
adds that to plot_weather.py.

Running python directly from bash command line in a crontab
rule is failing, so this commit adds two shell scripts
for doing so: plot_weather.sh and update_indices.sh.

Updates README.md accordingly.

Updates the 'Current user:' lines in some scripts to use
the whoami command rather than $USER. It turns out that Posix
specifies $LOGNAME (i.e. login name) as the envvar to be set
to the name of the logged in user by the login command, not
$USER.
@jamessynge jamessynge changed the title Make downloading of files more robust Make startup scripts & downloading of files more robust Mar 25, 2018
Copy link
Member

@wtgee wtgee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made some comments that should be looked at but will go ahead and approve now. Thanks!

_suppress_IERS_A_warning.original(message, *args, **kwargs)


try:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused as to this setup. Why not just use the regular method of suppressing warnings?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd never seen that method for suppressing warnings, thanks. I'll update accordingly.


from astroplan import download_IERS_A
from astropy.utils import data
def _default_data_folder():
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be a function or could it just be a global variable here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to a global var.

"""
Args:
data_folder: Path to directory into which to copy the astrometry.net indices.
wide_field: If true, downloads wide field astrometry.net indices.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

true -> True. Also should mark the defaults. Same with narrow_field below.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

keep_going: If False, exceptions are not suppressed. If True, returns False if there
are any download failures, else returns True.
"""
self.data_folder = data_folder or _default_data_folder()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should probably handle folder creation.

if narrow_field:
for i in range(4210, 4219):
download_one_file('4200/index-{}.fits'.format(i))
def download_all_files(data_folder=None, wide_field=True, narrow_field=False, keep_going=True):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem necessary and seems like it could just be handled in the __main__. Any specific reason for the wrapper?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And if you want to keep it I would suggest a different name.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've removed the function. It was intended as a helper function for other parts of POCS that would import this file, but isn't happening anymore. In fact, we could probably move this to POCS/scripts/

action='store_true',
help='Ignore download failures and keep going to the other downloads (default)')
group.add_argument(
'--nokeep_going', action='store_true', help='Fail immediately if any download fails')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Somewhat related to lack of convention mentioned above, I would separate the "no", i.e. --no-keep-going.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.


args = parser.parse_args()

if args.folder and not os.path.exists(args.folder):
print("{} does not exist.".format(args.folder))
print("Warning, data folder {} does not exist.".format(args.folder))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warn or just auto-create? We create directories in a number of different places so might just make sense here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switched to auto-create.

@@ -0,0 +1,51 @@
#!/bin/bash
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lots of boilerplate in these bash scripts. Any way we can consolidate?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, there is a lot of boilerplate... and some of it is even valuable! ;-)

I don't have a great solution yet, so will defer.

@@ -0,0 +1,52 @@
#!/bin/bash
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment about boilerplate above.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack.

@@ -0,0 +1,52 @@
#!/bin/bash
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the filename here isn't clear. Maybe something explicit like "download_support_files.sh"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed the name.

Copy link
Member

@wtgee wtgee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Thanks for making the changes!

return dl.download_all_files()


if __name__ == '__main__':
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to say that the argparse statements should be in this __name__ and that those arguments should then be passed to a main but I can't find anything to support why that would be a preferred way.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a need for that... except that I didn't pass in sys.argv[1:] so that it could be passed to argparse.parse(). However, the examples I found didn't bother with that. And it wouldn't make sense to call the method main if the parsing had already occurred Anyway, we can fix that later if it makes sense.


set -x

python "${POCS}/pocs/utils/data.py"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This whole wrapper script seems to be to guarantee that the environment is correct for downloading but it doesn't check the python version. Should probably source activate panoptes-env if we are still recommending that route otherwise would need to make sure python3

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The source activate panoptes-env should be in $HOME/.profile, and that is what the instructions call for. After all, this is a script designed to be run from cron on a PANOPTES scope, not necessarily for a dev box.

Nonetheless, I suspect you're right that I should make such a change, along with other changes to reduce the boilerplate. Since this PR has been going for a while, I'd like to handle those changes in another PR. I filed #519 to track this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, thanks. This PR is still approved so I'll leave to you to merge when ready.


set -x

"${POCS}/scripts/plot_weather.py"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above although we should make them consistent. This one appears to be executable where the data.py above is not?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, the POCS/scripts are executable, but I don't think we have many in POCS/pocs that are executable.

@jamessynge jamessynge merged commit aef07d5 into panoptes:develop May 28, 2018
@jamessynge jamessynge deleted the issue-506-do-not-require-internet branch May 28, 2018 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants