askap-vast · ajstewart · Nov 26, 2020 · Oct 9, 2020 · Oct 9, 2020 · Oct 9, 2020
diff --git a/.gitignore b/.gitignore
@@ -109,3 +109,4 @@ Dockerfile
 docker/
 .dockerignore
 docker-compose.yml
+.idea/
diff --git a/.nvmrc b/.nvmrc
@@ -0,0 +1,2 @@
+12.17.0
+
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 #### Added
 
+- Added test cases [#412](https://github.com/askap-vast/vast-pipeline/pull/412)
 - Added [askap-vast/forced_phot](https://github.com/askap-vast/forced_phot) to pip requirements [#408](https://github.com/askap-vast/vast-pipeline/pull/408).
 - Added pipeline configuration parameter, `SOURCE_AGGREGATE_PAIR_METRICS_MIN_ABS_VS`, to filter measurement pairs before calculating aggregate metrics [#407](https://github.com/askap-vast/vast-pipeline/pull/407).
 - Added custom 404.html and 500.html templates for error pages [#415](https://github.com/askap-vast/vast-pipeline/pull/415)
@@ -96,6 +97,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 #### Fixed
 
+- Fixed nodejs version [#412](https://github.com/askap-vast/vast-pipeline/pull/412)
+- Fixed npm start failure [#412](https://github.com/askap-vast/vast-pipeline/pull/412)
 - All queries using the 2-epoch metric `Vs` now operate on `abs(Vs)`. The original `Vs` stored in MeasurementPair objects is still signed [#407](https://github.com/askap-vast/vast-pipeline/pull/407).
 - Changed aggregate 2-epoch metric calculation for Source objects to ensure they come from the same pair [#407](https://github.com/askap-vast/vast-pipeline/pull/407).
 - Fixed new sources rms measurement returns when no measurements are valid [#417](https://github.com/askap-vast/vast-pipeline/pull/417).
@@ -110,6 +113,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 #### Removed
 
+- Removed/Disabled obsolete test cases[#412](https://github.com/askap-vast/vast-pipeline/pull/412)
 - Removed `vast_pipeline/pipeline/forced_phot.py` [#408](https://github.com/askap-vast/vast-pipeline/pull/408).
 - Removed 'selavy' from homepage measurements count label [#391](https://github.com/askap-vast/vast-pipeline/pull/391).
 - Removed leftover `pipeline/plots.py` file [#391](https://github.com/askap-vast/vast-pipeline/pull/391).
@@ -120,6 +124,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 #### List of PRs
 
+- [#412](https://github.com/askap-vast/vast-pipeline/pull/412) feat: added some unit tests.
 - [#419](https://github.com/askap-vast/vast-pipeline/pull/419) feat: Update alerts to use toasts.
 - [#408](https://github.com/askap-vast/vast-pipeline/pull/408) feat: use forced_phot dependency instead of copied code.
 - [#407](https://github.com/askap-vast/vast-pipeline/pull/407) fix, model: modified 2-epoch metric calculation.

diff --git a/INSTALL.md b/INSTALL.md
@@ -150,6 +150,41 @@ $ npm ci && npm start
 
 __NOTE__: make sure you are in the root of the repo, as shown above. That command "clean install" all the dependencies, copies files into the `static/vendor` folder and minified CSS and JS files. For more details of compilation of frontend assets (e.g. single tasks), and developement set up read the [Front End `README.md`](./static/README.md).
 
+__NOTE__: When `npm start` or `npm run start` was run in a Ubuntu(20.04 LTS) virtual environment, for some unknown
+reasons, both commands failed with the following error.
+
+```
+[12:48:19] 'js9Make' errored after 7.67 ms
+[12:48:19] Error: spawn make ENOENT
+    at Process.ChildProcess._handle.onexit (internal/child_process.js:267:19)
+    at onErrorNT (internal/child_process.js:469:16)
+    at processTicksAndRejections (internal/process/task_queues.js:84:21)
+[12:48:19] 'default' errored after 2.63 s
+npm ERR! code ELIFECYCLE
+npm ERR! errno 1
+npm ERR! [email protected] start: `gulp default`
+npm ERR! Exit status 1
+npm ERR! 
+npm ERR! Failed at the [email protected] start script.
+npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
+
+npm ERR! A complete log of this run can be found in:
+npm ERR!     /home/vast/.npm/_logs/2020-10-06T01_48_19_215Z-debug.log
+
+```
+
+The way around for this issue is unorthodox. The following steps were followed to overcome the issue:
+
+```
+$ cd node_modules/js9/
+$ ./configure
+$ make
+$ make install
+$ cd ~/vast-pipeline/  ## (to comeback to the root folder of the project)
+$ npm install
+```
+That somehow solved the issue mentioned above.
+
 ---
 
 Done! Now open the [`README.md`](./README.md) file to see how to initialize and run the pipeline. Otherwise if you intend on developing the repo open the [`CONTRIBUTING.md`](./CONTRIBUTING.md) file for instructions on how to contribute to the repo.
diff --git a/PROFILE.md b/PROFILE.md
@@ -0,0 +1,9 @@
+Running on 12GB of data with 464MB peak memory usage takes 4 mins:
+_performance:_ ~80% `final_operations`, ~10% `get_src_skyregion_merged_df`. `final_operations` calls other functions, out of these the largest is 50% in `make_upload_sources` which spends about 20% of time on `utils` `<method 'execute' of 'psycopg2.extensions.cursor' objects>`. The `get_src_skyregion_merged_df` time sink is in `threading` 15% of time is on `wait` (`final_operations` spends some time on `threading` as well, hence 15% > 10%).
+_memory:_ 40% `pyarrow` `parquet` `write_table`, rest is mostly fragmented, some more `pyarrow` and some `pandas`
+
+Running on 3MB of data with peak memory usage 176MB, takes 1.5s:
+_performance:_ ~30% goes to pipeline (about 9% of this is `pickle`), ~11% goes to `read`, rest goes to django I think
+_memory:_ 30% memory is spent on `django`, 20% is spent on `astropy/coordinates/matrix_utilities`, 10% on importing other modules, rest is fragmented quite small
+
+Note that I didn't include the generation of the `images/*/measurements.parquet` or other files in these profiles.
diff --git a/README.md b/README.md
@@ -105,6 +105,8 @@ After creating the folders with the defaults values your directory tree should l
 ├── <b>static</b>
 ├── <b>templates</b>
 ├── <b>pipeline-runs</b>
+├── <b>raw-images</b>
+├── <b>reference-surveys</b>
 ├── <b>vast_pipeline</b>
 └── <b>webinterface</b>
 </code></pre>
@@ -134,7 +136,7 @@ Output:
 ```bash
  ...
 
-[pipeline]
+[vast_pipeline]
   clearpiperun
   importsurvey
   initpiperun

diff --git a/vast_pipeline/pipeline/association.py b/vast_pipeline/pipeline/association.py
@@ -111,7 +111,7 @@ def one_to_many_basic(
 
     if duplicated_skyc2.empty:
         logger.debug('No one-to-many associations.')
-        return sources_df, skyc2_srcs
+        return skyc2_srcs, sources_df
 
     logger.info(
         'Detected #%i double matches, cleaning...',
@@ -328,10 +328,6 @@ def one_to_many_advanced(
         Updated temp_srcs and sources_df with all one_to_many relation
         information added.
     '''
-    if method not in ['advanced', 'deruiter']:
-        raise Exception(
-            f"Advanced association method '{method}' not recongised!"
-        )
     # use only these columns for easy debugging of the dataframe
     cols = [
         'index_old_skyc1', 'id_skyc1', 'source_skyc1', 'd2d_skyc1',
@@ -598,7 +594,7 @@ def many_to_many_advanced(temp_srcs: pd.DataFrame, method: str) -> pd.DataFrame:
         The temporary associtation dataframe used through the advanced
         association process.
     method : str
-        Can be either 'advanced' or 'dr' to represent the advanced association
+        Can be either 'advanced' or 'deruiter' to represent the advanced association
         method being used.
 
     Returns
@@ -790,7 +786,7 @@ def basic_association(
 
     # must check for double matches in the acceptable matches just made
     # this would mean that multiple sources in skyc2 have been matched
-    #  to the same base source we want to keep closest match and move
+    # to the same base source we want to keep closest match and move
     # the other match(es) back to having a -1 src id
     skyc2_srcs, sources_df = one_to_many_basic(skyc2_srcs, sources_df)
 

diff --git a/vast_pipeline/tests/pipeline-runs/advanced-association-with-forced-extraction/config.py b/vast_pipeline/tests/pipeline-runs/advanced-association-with-forced-extraction/config.py
@@ -12,37 +12,37 @@
 # IMAGE_FILES[0] image matches SELAVY_FILES[0] file
 IMAGE_FILES = [
     # insert images file path(s) here
-    './pipeline/tests/data/epoch01.fits',
-    './pipeline/tests/data/epoch02.fits',
-    './pipeline/tests/data/epoch03.fits',
-    './pipeline/tests/data/epoch04.fits',
+    './vast_pipeline/tests/data/epoch01.fits',
+    './vast_pipeline/tests/data/epoch02.fits',
+    './vast_pipeline/tests/data/epoch03.fits',
+    './vast_pipeline/tests/data/epoch04.fits',
 ]
 
 # Selavy catalogue files
 SELAVY_FILES = [
     # insert Selavy file path(s) here
-    './pipeline/tests/data/epoch01.selavy.components.txt',
-    './pipeline/tests/data/epoch02.selavy.components.txt',
-    './pipeline/tests/data/epoch03.selavy.components.txt',
-    './pipeline/tests/data/epoch04.selavy.components.txt',
+    './vast_pipeline/tests/data/epoch01.selavy.components.txt',
+    './vast_pipeline/tests/data/epoch02.selavy.components.txt',
+    './vast_pipeline/tests/data/epoch03.selavy.components.txt',
+    './vast_pipeline/tests/data/epoch04.selavy.components.txt',
 ]
 
 # Noise or RMS files
 NOISE_FILES = [
     # insert RMS file path(s) here
-    './pipeline/tests/data/epoch01.noiseMap.fits',
-    './pipeline/tests/data/epoch02.noiseMap.fits',
-    './pipeline/tests/data/epoch03.noiseMap.fits',
-    './pipeline/tests/data/epoch04.noiseMap.fits',
+    './vast_pipeline/tests/data/epoch01.noiseMap.fits',
+    './vast_pipeline/tests/data/epoch02.noiseMap.fits',
+    './vast_pipeline/tests/data/epoch03.noiseMap.fits',
+    './vast_pipeline/tests/data/epoch04.noiseMap.fits',
 ]
 
 # background map files
 BACKGROUND_FILES = [
     # insert background map file path(s) here
-    './pipeline/tests/data/epoch01.meanMap.fits',
-    './pipeline/tests/data/epoch02.meanMap.fits',
-    './pipeline/tests/data/epoch03.meanMap.fits',
-    './pipeline/tests/data/epoch04.meanMap.fits',
+    './vast_pipeline/tests/data/epoch01.meanMap.fits',
+    './vast_pipeline/tests/data/epoch02.meanMap.fits',
+    './vast_pipeline/tests/data/epoch03.meanMap.fits',
+    './vast_pipeline/tests/data/epoch04.meanMap.fits',
 ]
 
 ###

diff --git a/vast_pipeline/tests/pipeline-runs/advanced-association/config.py b/vast_pipeline/tests/pipeline-runs/advanced-association/config.py
@@ -12,37 +12,37 @@
 # IMAGE_FILES[0] image matches SELAVY_FILES[0] file
 IMAGE_FILES = [
     # insert images file path(s) here
-    './pipeline/tests/data/epoch01.fits',
-    './pipeline/tests/data/epoch02.fits',
-    './pipeline/tests/data/epoch03.fits',
-    './pipeline/tests/data/epoch04.fits',
+    './vast_pipeline/tests/data/epoch01.fits',
+    './vast_pipeline/tests/data/epoch02.fits',
+    './vast_pipeline/tests/data/epoch03.fits',
+    './vast_pipeline/tests/data/epoch04.fits',
 ]
 
 # Selavy catalogue files
 SELAVY_FILES = [
     # insert Selavy file path(s) here
-    './pipeline/tests/data/epoch01.selavy.components.txt',
-    './pipeline/tests/data/epoch02.selavy.components.txt',
-    './pipeline/tests/data/epoch03.selavy.components.txt',
-    './pipeline/tests/data/epoch04.selavy.components.txt',
+    './vast_pipeline/tests/data/epoch01.selavy.components.txt',
+    './vast_pipeline/tests/data/epoch02.selavy.components.txt',
+    './vast_pipeline/tests/data/epoch03.selavy.components.txt',
+    './vast_pipeline/tests/data/epoch04.selavy.components.txt',
 ]
 
 # Noise or RMS files
 NOISE_FILES = [
     # insert RMS file path(s) here
-    './pipeline/tests/data/epoch01.noiseMap.fits',
-    './pipeline/tests/data/epoch02.noiseMap.fits',
-    './pipeline/tests/data/epoch03.noiseMap.fits',
-    './pipeline/tests/data/epoch04.noiseMap.fits',
+    './vast_pipeline/tests/data/epoch01.noiseMap.fits',
+    './vast_pipeline/tests/data/epoch02.noiseMap.fits',
+    './vast_pipeline/tests/data/epoch03.noiseMap.fits',
+    './vast_pipeline/tests/data/epoch04.noiseMap.fits',
 ]
 
 # background map files
 BACKGROUND_FILES = [
     # insert background map file path(s) here
-    './pipeline/tests/data/epoch01.meanMap.fits',
-    './pipeline/tests/data/epoch02.meanMap.fits',
-    './pipeline/tests/data/epoch03.meanMap.fits',
-    './pipeline/tests/data/epoch04.meanMap.fits',
+    './vast_pipeline/tests/data/epoch01.meanMap.fits',
+    './vast_pipeline/tests/data/epoch02.meanMap.fits',
+    './vast_pipeline/tests/data/epoch03.meanMap.fits',
+    './vast_pipeline/tests/data/epoch04.meanMap.fits',
 ]
 
 ###

diff --git a/vast_pipeline/tests/pipeline-runs/basic-association-with-forced-extraction/config.py b/vast_pipeline/tests/pipeline-runs/basic-association-with-forced-extraction/config.py
@@ -12,37 +12,37 @@
 # IMAGE_FILES[0] image matches SELAVY_FILES[0] file
 IMAGE_FILES = [
     # insert images file path(s) here
-    './pipeline/tests/data/epoch01.fits',
-    './pipeline/tests/data/epoch02.fits',
-    './pipeline/tests/data/epoch03.fits',
-    './pipeline/tests/data/epoch04.fits',
+    './vast_pipeline/tests/data/epoch01.fits',
+    './vast_pipeline/tests/data/epoch02.fits',
+    './vast_pipeline/tests/data/epoch03.fits',
+    './vast_pipeline/tests/data/epoch04.fits',
 ]
 
 # Selavy catalogue files
 SELAVY_FILES = [
     # insert Selavy file path(s) here
-    './pipeline/tests/data/epoch01.selavy.components.txt',
-    './pipeline/tests/data/epoch02.selavy.components.txt',
-    './pipeline/tests/data/epoch03.selavy.components.txt',
-    './pipeline/tests/data/epoch04.selavy.components.txt',
+    './vast_pipeline/tests/data/epoch01.selavy.components.txt',
+    './vast_pipeline/tests/data/epoch02.selavy.components.txt',
+    './vast_pipeline/tests/data/epoch03.selavy.components.txt',
+    './vast_pipeline/tests/data/epoch04.selavy.components.txt',
 ]
 
 # Noise or RMS files
 NOISE_FILES = [
     # insert RMS file path(s) here
-    './pipeline/tests/data/epoch01.noiseMap.fits',
-    './pipeline/tests/data/epoch02.noiseMap.fits',
-    './pipeline/tests/data/epoch03.noiseMap.fits',
-    './pipeline/tests/data/epoch04.noiseMap.fits',
+    './vast_pipeline/tests/data/epoch01.noiseMap.fits',
+    './vast_pipeline/tests/data/epoch02.noiseMap.fits',
+    './vast_pipeline/tests/data/epoch03.noiseMap.fits',
+    './vast_pipeline/tests/data/epoch04.noiseMap.fits',
 ]
 
 # background map files
 BACKGROUND_FILES = [
     # insert background map file path(s) here
-    './pipeline/tests/data/epoch01.meanMap.fits',
-    './pipeline/tests/data/epoch02.meanMap.fits',
-    './pipeline/tests/data/epoch03.meanMap.fits',
-    './pipeline/tests/data/epoch04.meanMap.fits',
+    './vast_pipeline/tests/data/epoch01.meanMap.fits',
+    './vast_pipeline/tests/data/epoch02.meanMap.fits',
+    './vast_pipeline/tests/data/epoch03.meanMap.fits',
+    './vast_pipeline/tests/data/epoch04.meanMap.fits',
 ]
 
 ###
-Original file line number
+Diff line change
@@ Expand Up / @@ -109,3 +109,4 @@ Dockerfile @@
     docker/
     .dockerignore
     docker-compose.yml
+    .idea/