Merge branch 'develop' of https://github.com/Keck-DataReductionPipeli…

…nes/OsirisDRP into develop

CONTRIBUTING.md

@@ -0,0 +1,119 @@
+# Contributing to the Osiris DRP
+
+## Reporting Issues
+
+Please report [issues](https://github.com/Keck-DataReductionPipelines/OsirisDRP/issues) on GitHub to the [OsirisDRP repository](https://github.com/Keck-DataReductionPipelines/OsirisDRP). Please include the version of IDL you are using, and the shell you are using.
+
+## Contributing code
+So you're interested in contributing code to to the OsirisDRP? Excellent!
+
+Most contributions to OsirisDRP are done via pull requests from GitHub users'
+forks of the [OsirisDRP repository](https://github.com/astropy/astropy). If you're new to this style of development,
+Astropy has a good summary of this [development workflow](http://docs.astropy.org/en/latest/development/workflow/development_workflow.html), but we'll describe it for the OsirisDRP below.
+
+### Getting the development version
+
+First, you'll need an account on [GitHub](http://github.com). Then go to <https://github.com/Keck-DataReductionPipelines/OsirisDRP> and click on the "Fork" button in the upper right hand corner. This will make a "Forked" copy of the Osiris DRP in your GitHub Account.
+
+Then, clone the OsirisDRP to your computer:
+
+```
+    
+    $ git clone https://github.com/your-user-name/OsirisDRP.git
+    Cloning into 'OsirisDRP'...
+    remote: Counting objects: 2386, done.
+    remote: Compressing objects: 100% (97/97), done.
+    remote: Total 2386 (delta 60), reused 0 (delta 0), pack-reused 2288
+    Receiving objects: 100% (2386/2386), 5.41 MiB | 957.00 KiB/s, done.
+    Resolving deltas: 100% (1674/1674), done.
+    Checking connectivity... done.
+    $ cd OsirisDRP/
+    $ git checkout develop
+```
+
+Now, you need to set your repository up so that you can collect the latest changes from the Keck version of the OsirisDRP. To do this, add a git remote (called ``upstream`` by convention):
+
+```
+    $ git remote add upstream https://github.com/Keck-DataReductionPipelines/OsirisDRP.git
+    $ git fetch upstream
+    Fetching upstream
+    remote: Counting objects: 630, done.
+    remote: Compressing objects: 100% (78/78), done.
+    remote: Total 630 (delta 68), reused 30 (delta 30), pack-reused 522
+    Receiving objects: 100% (630/630), 1.31 MiB | 365.00 KiB/s, done.
+    Resolving deltas: 100% (86/86), completed with 24 local objects.
+    From https://github.com/Keck-DataReductionPipelines/OsirisDRP
+     * [new branch]      develop    -> upstream/develop
+     * [new branch]      master     -> upstream/master
+```
+
+To get the latest changes to the development version of the pipeline, pull from ``upstream/develop``
+
+```
+    $ git pull upstream develop
+    From https://github.com/Keck-DataReductionPipelines/OsirisDRP
+     * branch            develop    -> FETCH_HEAD
+    Already up-to-date.
+```
+
+### Making changes
+
+Now make your awesome changes to the pipeline! When you are done, commit them to your git repository.
+
+For example, lets pretend you've added ``my-awesome-file``
+
+```
+    $ git add my-awesome-file
+    $ git commit
+```
+
+### Testing your changes
+
+The OSIRIS Data reduction pipeline has a testing framework. It requires ``python``, ``py.test`` and ``astropy``. If you use anaconda for your python you can install ``py.test`` and ``astropy`` with ``$ conda install pytest astropy``. If you have a standard python installation, you can try installing ``py.test`` and ``astropy`` using pip, with ``$ pip install pytest astropy``.
+
+To test your changes, you can use the existing test framework. You can find information on writing new tests in ``tests/README.md``. You can then run your tests with ``make test``
+
+```
+    $ make test
+```
+
+### Giving your changes back to the community
+
+Now you need to publish your changes to GitHub:
+
+```
+    $ git push develop
+```
+
+Then, you can go to your repository on GitHub (e.g. <http://github.com/your-github-username/OsirisDRP.git>), and there should be a button there to create a pull request. Create the pull request, add a description!
+
+### Things to consider about your pull request
+
+Once you open a pull request (which should be opened against the ``develop``
+branch, not against any of the other branches), please make sure that you
+include the following:
+
+- **Code**: the code you are adding, which should follow as much as possible
+  our [coding guidelines](http://docs.astropy.org/en/latest/development/codeguide.html).
+
+- **Tests**: these are usually tests that ensures that code that previously
+  failed now works (regression tests) or tests that cover as much as possible
+  of the new functionality to make sure it doesn't break in future, and also
+  returns consistent results on all platforms (since we run these tests on many
+  platforms/configurations). For more information about how to write tests, see
+  ``tests/README.md``.
+
+- **Changelog entry**: whether you are fixing a bug or adding new
+  functionality, you should add an entry to the ``CHANGES.rst`` file that
+  includes if possible the issue number (if you are opening a pull request you
+  may not know this yet, but you can add it once the pull request is open). If
+  you're not sure where to put the changelog entry, wait at least until a
+  maintainer has reviewed your PR and assigned it to a milestone.
+
+  You do not need to include a changelog entry for fixes to bugs introduced in
+  the developer version and which are not present in the stable releases.  In
+  general you do not need to include a changelog entry for minor documentation
+  or test updates.  Only user-visible changes (new features/API changes, fixed
+  issues) need to be mentioned.  If in doubt ask the core maintainer reviewing
+  your changes.
+

Makefile.local

@@ -14,4 +14,4 @@ include modules/source/Makefile.local
 
 
 test: all
-	py.test
+	. scripts/osirisSetup.sh && osirisSetup && py.test

backbone/drpBackbone__define.pro

@@ -238,7 +238,7 @@ PRO drpBackbone::ConsumeQueue, QueueDir
 	queueDirName = QueueDir + '*.waiting'
 	FileNameArray = FILE_SEARCH(queueDirName)
     s = size(FileNameArray)
-    WHILE s[0] GT 0 AND DRPCONTINUE DO BEGIN
+    WHILE N_ELEMENTS(FileNameArray) GT 0 AND DRPCONTINUE DO BEGIN
     	CATCH, Error; Catch errors inside the pipeline
     	IF Error EQ 0 THEN BEGIN
     		CurrentDRF = drpGetNextWaitingFile(FileNameArray)
@@ -252,10 +252,10 @@ PRO drpBackbone::ConsumeQueue, QueueDir
     	ENDELSE
         IF CurrentDRF.Name NE '' THEN BEGIN
             done = where(FileNameArray EQ DRFFileName, nmatch)
-            IF s[0] EQ 1 THEN BEGIN
+            IF N_ELEMENTS(FileNameArray) EQ 1 THEN BEGIN
                 DRPCONTINUE=0
             ENDIF
-            IF NMATCH GT 0 AND s[0] GT 1 THEN BEGIN
+            IF NMATCH GT 0 AND N_ELEMENTS(FileNameArray) GT 1 THEN BEGIN
                 remove, done, FileNameArray
                 s = SIZE(FileNameArray)
             ENDIF

backbone/drpResolve.pro

@@ -0,0 +1,5 @@
+
+RESOLVE_ALL, RESOLVE_PROCEDURE='drpRun', /CONTINUE_ON_ERROR
+FILES = FILE_SEARCH(getenv("OSIRIS_IDL_BASE") + "/modules/*_[0-9][0-9][0-9].pro")
+FOR I=0, N_ELEMENTS(FILES)-1 DO $
+        RESOLVE_ALL, RESOLVE_EITHER=FILE_BASENAME(FILES[I], '.pro'), /CONTINUE_ON_ERROR

tests/drpStartup.pro

@@ -5,5 +5,3 @@ drs_idl_root=getenv('OSIRIS_IDL_BASE')
 !PATH=expand_path('+'+drs_idl_root+'/tests/')+":"+!PATH
 .compile drpTest.pro
 !quiet=0
-
-PRINT, !PATH

tests/drptestbones/backbone.py

@@ -30,7 +30,7 @@ def prepare_queue_directory(queue_directory):
         pipeline_filepath = os.path.join(os.path.dirname(xml_filename), pipeline_filename)
         shutil.copy(xml_filename, pipeline_filepath)
         pipeline_status_glob = "{0:03d}.{1:s}.*".format(i+1, pipeline_file)
-        for filename in glob.iglob(pipeline_status_glob):
+        for filename in glob.iglob(os.path.join(queue_directory, pipeline_status_glob)):
             base, status = os.path.splitext(filename)
             if status != '.waiting':
                 os.remove(filename)
@@ -75,10 +75,14 @@ def consume_queue_directory(queue_directory, test_directory=None):
     proc.wait()
 
     for drf in waiting_drfs:
+        drf = os.path.relpath(os.path.abspath(drf), os.path.realpath(os.environ["OSIRIS_ROOT"]))
         done_drf = os.path.splitext(drf)[0] + ".done"
         failed_drf = os.path.splitext(drf)[0] + ".failed"
-        if os.path.exists(failed_drf) or not os.path.exists(done_drf):
+        if os.path.exists(failed_drf):
             raise BackboneError("The backbone seems to have failed on DRF {0}".format(drf))
+        if not os.path.exists(done_drf):
+            raise BackboneError("The backbone doesn't appear to have finished DRF {0}".format(drf))
+
     return proc.returncode