Added support for galsim_hub (#117)

* Added support for galsim_hub * Moved galaxy sampling to render_minibatch * Moved generation to whole batch and added possibility to change parameters * refactor to avoid repeating code * Better code for handling the extra images * Added documentation for galsim_hub and new sampling function * Added a test for galsim_hub * Completed documentation * Remove merge comment * Added tutorial notebook * Fixed test and moved import * update lock * added tensorflow and galsim_hub * updates * just updating but still the problem is not resolved * using python 3.7 * added tensorflow and galsim_hub as optional dependencies * try install galsim_hub with pip * small check * eigency dependency? * install eigency first * install galsim first * try conda * try again * some updatest * a new try :) * error in action * weird errro * another try * now i get it * the doc for conda action is misleading * doesn't activate environment * old conda * give up and try source activate * need to specify bash * works???! * lint should work now * misconfiguration * need lint * update docs * Add missing psf convolution * no more requirements.txt * galsim hub no multiprocessing keyword * update poetry Co-authored-by: Thomas Sainrat <[email protected]> Co-authored-by: Ismael Mendoza <[email protected]>
LSSTDESC · Mar 31, 2021 · ab87076 · ab87076
1 parent 46d660f
commit ab87076
Show file tree

Hide file tree

Showing 14 changed files with 1,243 additions and 1,178 deletions.
diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
@@ -9,41 +9,37 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.8]
+        python-version: [3.7]
 
     steps:
       - name: Checkout github repo
         uses: actions/checkout@v2
         with:
           fetch-depth: 0
 
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
+      - name: Install miniconda and environment
+        uses: conda-incubator/setup-miniconda@v2
         with:
+          miniconda-version: "latest"
+          activate-environment: btk_py37
           python-version: ${{ matrix.python-version }}
 
-
-      - name: Install Poetry
-        run: |
-          pip install --upgrade pip
-          pip install poetry
-
-      - name: Poetry cache
-        uses: actions/cache@v2
-        with:
-          path: |
-            ~/.cache/pypoetry
-          key: ${{ runner.os }}-${{ hashFiles('./poetry.lock') }}
-
-      - name: Instal FFTW + Eigen
+      - name: Install conda dependencies
+        shell: bash -l {0} # setup and activate conda environment.
         run: |
-          sudo apt-get install libfftw3-dev libeigen3-dev
+          conda install fftw 
+          conda install -c conda-forge eigen
+          conda install -c conda-forge galsim
+          conda install tensorflow==1.15
+          conda install -c conda-forge poetry
 
       - name: Install poetry dependencies
+        shell: bash -l {0}
         run: |
-          poetry install
+          poetry install --extras "galsim-hub"
 
       - name: build docs
+        shell: bash -l {0}
         run: |
           cd docs
           poetry run make html

diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
@@ -14,7 +14,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.8]
+        python-version: [3.7]
 
     steps:
       - name: Checkout github repo
@@ -23,31 +23,29 @@ jobs:
           lfs: true
           fetch-depth: 0
 
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
+      - name: Install miniconda and environment
+        uses: conda-incubator/setup-miniconda@v2
         with:
+          miniconda-version: "latest"
+          activate-environment: btk_py37
           python-version: ${{ matrix.python-version }}
 
-      - name: Install Poetry
+      - name: Install conda dependencies
+        shell: bash -l {0} # setup and activate conda environment.
         run: |
-          pip install --upgrade pip
-          pip install poetry
-
-      - name: Poetry cache
-        uses: actions/cache@v2
-        with:
-          path: |
-            ~/.cache/pypoetry
-          key: ${{ runner.os }}-${{ hashFiles('./poetry.lock') }}
-
-      # galsim needs this.
-      - name: Instal FFTW + Eigen
-        run: |
-          sudo apt-get install libfftw3-dev libeigen3-dev
+          conda install fftw 
+          conda install -c conda-forge eigen
+          conda install -c conda-forge galsim
+          conda install tensorflow==1.15
+          conda install -c conda-forge poetry
 
       - name: Install poetry dependencies
+        shell: bash -l {0}
         run: |
-          poetry install
+          poetry install --extras "galsim-hub"
 
       - name: Run pre-commit checks
-        uses: pre-commit/[email protected]
+        shell: bash -l {0}
+        run: |
+          pre-commit install
+          pre-commit run --all-files
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -14,7 +14,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.8]
+        python-version: [3.7]
 
     steps:
       - name: Checkout github repo (+ download lfs dependencies)
@@ -23,33 +23,29 @@ jobs:
           lfs: true
           fetch-depth: 0
 
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
+      - name: Install miniconda and environment
+        uses: conda-incubator/setup-miniconda@v2
         with:
+          miniconda-version: "latest"
+          activate-environment: btk_py37
           python-version: ${{ matrix.python-version }}
 
-      - name: Install Poetry
+      - name: Install conda dependencies
+        shell: bash -l {0} # setup and activate conda environment.
         run: |
-          pip install --upgrade pip
-          pip install poetry
-
-      - name: Poetry cache
-        uses: actions/cache@v2
-        with:
-          path: |
-            ~/.cache/pypoetry
-          key: ${{ runner.os }}-${{ hashFiles('./poetry.lock') }}
-
-      # galsim needs this.
-      - name: Instal FFTW + Eigen
-        run: |
-          sudo apt-get install libfftw3-dev libeigen3-dev
+          conda install fftw 
+          conda install -c conda-forge eigen
+          conda install -c conda-forge galsim
+          conda install tensorflow==1.15
+          conda install -c conda-forge poetry
 
       - name: Install poetry dependencies
+        shell: bash -l {0}
         run: |
-          poetry install
+          poetry install --extras "galsim-hub"
 
       - name: Run Tests
+        shell: bash -l {0}
         run: |
           poetry run pytest --cov=./btk --cov-report=xml --durations=0
 

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -60,14 +60,14 @@ The following sections are only relevant for the maintainer of BTK (currently Is
 
 1. We use [poetry](https://python-poetry.org) as python package manager for BTK. It guarantees all developers are sharing the same python environment, makes it really easy to update dependencies, and publish to [pypi](https://pypi.org). Given some of the complications with installing `galsim` via `poetry`, we follow a hybrid approach of `conda`+`poetry`. 
 
-2. It is recommended to create a `conda` virtual environment (using `python3.8`) from scratch and use it to install all required dependencies. After having installed `conda`, please follow the following series of steps:
+2. It is recommended to create a `conda` virtual environment (using `python3.7`) from scratch and use it to install all required dependencies. After having installed `conda`, please follow the following series of steps:
 
 ```
 # enter to the local repo
 cd BlendingToolKit
 
 # create virtual environment.
-conda create -n btk python=3.8
+conda create -n btk python=3.7
 conda activate btk
 
 # make sure you can import galsim after this step before moving on.