Refactor v1 (#412)

* refactor draw blends * refactor measure * atomic metrics * start new metrics classes * survey refactor * delete unnecessary utils * matching functions * posted matching ufnction that does nothing * more progress on metrics and added iou draft * better name * small correction * slight linting * update pdm * measure is now deblend * measure is now deblend * propagate naming consistency * quickstart notebook * comments from axel * small fix to __repr__ functions * one more name change * matching refactored * clean up * ClosestSkyNeighbourMatching added * minor matching bug fixed * matching and rearanging for deblended batch added * linting * add good names for pylint * lint * more good names * ignore this error * disable some pylint errors * linting * more linting * more linting * better to be explicit with catalog_list * match returns new class * first idea for turning BlendBatch into list structure * refactor black * idea: implement multi-resolution batch separate from BlendBatch * new file storing blend dataclasses * refactor * add deblend batches * remove survey name * small correction after update * mr deblender draft * several corrections and validation * punctuation * add easy access to psf and reogranize methods * add dependencies * delete lock for now * slowing finalizing metrics * first draft of scarlet deblender * small comments * reformat add scarlet * add additional file specifying venv to ignore * whitespace * added scarlet * typing change * switch to poetry * finish poetry * remove unnecessary bandpass * no more pandas or seaborn * no more bandpass * disable error * measure start * black * move to measure * draft missing metrics * changed name for symmetry * saving and loading are now class methods * blend_list to catalog_list, better name and normalize * two more ok names * finish measure * organizing metrics utils * fix imports * poetry update * functions for metrics * segmentatino metric implemented, use nan to keep track of empty ones * metrics in folder now * new init * init for metrics * metric recon draft * most measure functions draft implemented * base metric * typos * cleanup Table, matching is now implemented in MatchInfo * small fixes in class * fix Table * new matching functionalty and class and class to store matching information and do matching * type hinting * use correct truth value * finish detection metrics * fix augment data bug * don't need that * update init * minor fixes to saveing and loading * minor fixes to deblenders * saving and loading for deblended batch fixed * started testing functionality * update precommit * add larger catalog with 200k galaxies * use njobs not cpus * correct name convention * call does things in batches (default everywhere) * split lines for clarity * move to notebooks * quickstart continue with Deblender and new format * rgb images in linear * improve names * simplify * new name * add customizable mag cuts * use detected and matched arrays to compute metrics * option to disable progress bar * fix snr * small bug in match stats * add metrics and measurement * new catalog working * update with new catalog * correction when matching arrays * correction in reconstruction (need to update) * numpy array has no .sqrt member * quickstart notebook for tutorial today * test matching * new name * type hints and other fixes for clarity: no more 'ref_mag' * blend generator new name and no more shifts + type hints * clarify private methods and add type hints * type hints and docstrings, more clarity in arguments and mag cut name * poetry update * type hints, docstrings, private functions, remove shifts/indices, extra_data, clarity in inputs * type hinting * type hinting * a few type hints * use true survey rather than just name * directly use survey in BlendBatch in case it's modified * delete old notebooks for now * delete temporary notebooks * update notebook README * better docstring for add pixel columns * fix preprocess catalog * add more good names * add pair sampling and better function for out of bounds * precommit no pdm * use poetry in testing in ci * add docstring to metrics * allow kwargs in deblenders, better sep documentation, fixing Scarlet class based on Prakruth's suggestion * added n_bands and improved error messages, some optional parameters in dataclass * add max_n_sources to deblenders to standarized in case more sources than blend_batch are returned (can happen) * fix repr * spacing * catch rare exceptions and fix errors in Scarlet * handle empty catalogs * fix quickstart to use multiple band in DeblendExample * scarlet tutorial wth better ellipticity estimates * add extra data to Deblend Batch and Example classes; add scarlet sources extra data in Scarlet Deblender. * no more pre-commit * update poetry * didn't need this * update license * don't need requirements * fix workflows except docs * skip init files when isorting * nice init * new line * chagne default seed location to avoid cyclic import * ok for each blend * pylint errors to fix later * fix typing * fix conftest * accomodate one input returning one input when matching (not a tuple with 1 element) * minimal set of tests with new codebase implemented * prefer pylintrc (cleaner and can add coments) * small fixes * one more small notebook fix * fix docstrings * add back minimal pre-commit * add back precomimt * update maintainer to poetry * add scarlet install to lint * not a notebook * minimal new README * add pybind11 * new diagram * one more try * peigen * add scarlet to pytest too * update versions * update versions * update packages * remove filter warnings * update docs with new packages --------- Co-authored-by: Andrii Torchylo <[email protected]> Co-authored-by: Prakruth Adari <[email protected]>
LSSTDESC · Aug 11, 2023 · d53e83d · d53e83d
1 parent ebc7365
commit d53e83d
Show file tree

Hide file tree

Showing 83 changed files with 38,664 additions and 12,092 deletions.
diff --git a/.flake8 b/.flake8
@@ -2,4 +2,4 @@
 max-line-length=100
 max-complexity=18
 ignore=E203, W503
-exclude=btk/__init__.py,docs/build/
+exclude=btk/__init__.py,docs/build/, btk/metrics/__init__.py
diff --git a/MAINTAINER.md → .github/MAINTAINER.md b/MAINTAINER.md → .github/MAINTAINER.md
@@ -2,39 +2,35 @@
 
 ## Setup environment
 
-1. We use [pdm](https://pdm.fming.dev/latest/) as python package manager for BTK. It guarantees all developers are sharing the same python environment, makes it really easy to update dependencies, and publish to [pypi](https://pypi.org).
+1. We use [poetry](https://python-poetry.org/docs/) as python package manager for BTK. It guarantees all developers are sharing the same python environment, makes it really easy to update dependencies, and publish to [pypi](https://pypi.org).
 
-2. It is recommended to create a `pdm` virtual environment from scratch and use it to install all required dependencies. Please follow the following series of steps:
+2. For development, it is recommended to create a `poetry` virtual environment from scratch and use it to install all required dependencies. Please follow the following series of steps:
 
 ```bash
 # enter to the local repo
 cd BlendingToolKit
 
-# install pdm in mac os
-brew install pdm
+# install poetry
+curl -sSL https://install.python-poetry.org | python3 -
 
 # install all python dependencies from pyproject.toml file
 # and create lock file
-pdm update
+poetry update
 
 # activate pdm environment
-eval $(pdm venv activate in-project)
+poetry shell
 
 # install the git hook scripts
 pre-commit install
 ```
 
-Remember that `galsim` has additional dependencies that you might need to install. Please follow the instructions [here](https://galsim-developers.github.io/GalSim/_build/html/install.html).
+Remember that `galsim` has additional dependencies that you might need to install prior to running `poetry update`. Please follow the instructions [here](https://galsim-developers.github.io/GalSim/_build/html/install.html).
 
-## Updating packages
-
-1. If any of the dependencies requires an update, you can simply run `pdm update` inside your local repo to automatically update and install them. Feel free to push the changes of the `pyproject.toml` or `pdm.lock` file to the PR you are working on.
+One workaround in case of problems is to setup a `conda` environment, install `galsim`, install `poetry` with `conda`, and then run `poetry update`.
 
-2. You might also want to update the `requirements.txt` anytime you edit the `pyproject.toml` file.
+## Updating packages
 
-```bash
-pdm export -f requirements --without-hashes --pyproject --prod > requirements.txt
-```
+If any of the dependencies requires an update, you can simply run `poetry update` inside your local repo to automatically update and install them. Feel free to push the changes of the `poetry.lock` file to the PR you are working on.
 
 ## Making new Releases
 

diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
@@ -16,45 +16,31 @@ jobs:
       - name: Checkout github repo
         uses: actions/checkout@v2
 
+      - name: Install poetry
+        run: pipx install poetry
+
       - name: Install pandoc
         run: sudo apt-get install pandoc
 
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v3
         with:
           python-version: ${{ matrix.python-version }}
+          cache: "poetry"
 
-      - name: Setup PDM
-        uses: pdm-project/setup-pdm@v3
-        with:
-          python-version: ${{ matrix.python-version }}
-          enable-pep582: true
-          cache: true
-
-      # galsim needs this.
       - name: Install FFTW
         run: |
           sudo apt-get install libfftw3-dev
 
       - name: Install python packages
         run: |
-          pdm install
-
-      - name: Convert notebooks
-        run: |
-          cd docs/source
-          cat tutorials_header.rst > tutorials.rst
-          echo "" >> tutorials.rst
-          pdm run jupyter nbconvert --to rst ../../notebooks/00-intro.ipynb --FilesWriter.build_directory="./"
-          cat 00-intro.rst >> tutorials.rst
-          echo "" >> tutorials.rst
-          cat tutorials_footer.rst >> tutorials.rst
+          poetry install
 
       - name: build docs
         shell: bash -l {0}
         run: |
           cd docs
-          pdm run make html
+          poetry run make html
           if [ $? = 2 ]; then exit 1;fi
           cd build/html
           touch .nojekyll

diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
@@ -21,28 +21,46 @@ jobs:
       - name: Checkout github repo
         uses: actions/checkout@v2
 
+      - name: Install poetry
+        run: pipx install poetry
+
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v3
         with:
           python-version: ${{ matrix.python-version }}
-
-      - name: Setup PDM
-        uses: pdm-project/setup-pdm@v3
-        with:
-          python-version: ${{ matrix.python-version }}
-          enable-pep582: true
-          cache: true
+          cache: "poetry"
 
       # galsim needs this.
-      - name: Instal FFTW
+      - name: Install FFTW
         run: |
           sudo apt-get install libfftw3-dev
 
-      - name: Install python packages
+      - name: Install poetry dependencies
+        run: |
+          poetry install --with scarlet
+
+      - name: Install Scarlet
+        run: |
+          pip install "pybind11[global]" peigen
+          git clone https://github.com/pmelchior/scarlet.git
+          cd scarlet; python setup.py install; cd ..
+
+      - name: Run black
+        run: |
+          poetry run black --check btk
+
+      - name: Run isort
+        run: |
+          poetry run isort --check btk
+
+      - name: Run pydocstyle
+        run: |
+          poetry run pydocstyle btk
+
+      - name: Run flake8
         run: |
-          pdm install
+          poetry run flake8 btk
 
-      - name: Run pre-commit checks
+      - name: Run pylint
         run: |
-          pdm run pre-commit install
-          pdm run pre-commit run --all-files
+          poetry run pylint btk
diff --git a/.github/workflows/notebooks.yml b/.github/workflows/notebooks.yml
@@ -13,38 +13,31 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.8, 3.9, "3.10"]
+        python-version: [3.9, "3.10", "3.11"]
 
     steps:
       - name: Checkout github repo (+ download lfs dependencies)
         uses: actions/checkout@v2
-        with:
-          fetch-depth: 0
 
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
-        with:
-          python-version: ${{ matrix.python-version }}
+      - name: Install poetry
+        run: pipx install poetry
 
-      - name: Setup PDM
-        uses: pdm-project/setup-pdm@v3
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v3
         with:
           python-version: ${{ matrix.python-version }}
-          enable-pep582: true
-          cache: true
-
-      - name: Create environment variables
-        run: echo "BTK_HOME=$GITHUB_WORKSPACE" >> $GITHUB_ENV
+          cache: "poetry"
 
       # galsim needs this.
-      - name: Instal FFTW
+      - name: Install FFTW
         run: |
           sudo apt-get install libfftw3-dev
 
-      - name: Install python packages
+      - name: Install poetry dependencies
         run: |
-          pdm install
+          poetry install
 
+      # just checks if they run
       - name: Run notebooks
         run: |
-          pdm run pytest --nbmake notebooks --ignore "notebooks/01b-scarlet-measure.ipynb"
+          poetry run pytest --nbmake notebooks
diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml
@@ -9,33 +9,31 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.9"]
+        python-version: ["3.10"]
 
     steps:
       - name: Checkout github repo
         uses: actions/checkout@v2
 
+      - name: Install poetry
+        run: pipx install poetry
+
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v3
         with:
           python-version: ${{ matrix.python-version }}
+          cache: "poetry"
 
-      - name: Setup PDM
-        uses: pdm-project/setup-pdm@v3
-        with:
-          python-version: ${{ matrix.python-version }}
-          enable-pep582: true
-          cache: true
-
-      # galsim needs this.
-      - name: Instal FFTW
+      - name: Install FFTW
         run: |
           sudo apt-get install libfftw3-dev
 
-      - name: Install python packages
+      - name: Install poetry dependencies
         run: |
-          pdm install
+          poetry install
 
       - name: Build and publish
         run: |
-          pdm publish -r pypi --username ${{ secrets.PYPI_USERNAME }} --password ${{ secrets.PYPI_PASSWORD }}
+          poetry version $(git describe --tags --abbrev=0)
+          poetry build
+          poetry publish --username __token__ --password ${{ secrets.PYPI_TOKEN }}
diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml
@@ -15,36 +15,39 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.8", "3.9", "3.10"]
+        python-version: ["3.9", "3.10", "3.11"]
 
     steps:
       - name: Checkout github repo
         uses: actions/checkout@v2
 
+      - name: Install poetry
+        run: pipx install poetry
+
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v3
         with:
           python-version: ${{ matrix.python-version }}
-
-      - name: Setup PDM
-        uses: pdm-project/setup-pdm@v3
-        with:
-          python-version: ${{ matrix.python-version }}
-          enable-pep582: true
-          cache: true
+          cache: "poetry"
 
       # galsim needs this.
-      - name: Instal FFTW
+      - name: Install FFTW
         run: |
           sudo apt-get install libfftw3-dev
 
-      - name: Install python packages
+      - name: Install poetry dependencies
+        run: |
+          poetry install
+
+      - name: Install Scarlet
         run: |
-          pdm install
+          pip install "pybind11[global]" peigen
+          git clone https://github.com/pmelchior/scarlet.git
+          cd scarlet; python setup.py install; cd ..
 
       - name: Run Tests
         run: |
-          pdm run pytest --cov=./btk --cov-report=xml --durations=0
+          poetry run pytest --cov=./btk --cov-report=xml --durations=0
 
       - name: Upload coverage to Codecov
         uses: codecov/codecov-action@v1

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -13,43 +13,3 @@ repos:
       - id: check-merge-conflict
       - id: end-of-file-fixer
         exclude: ^.*fits
-  - repo: https://github.com/psf/black
-    rev: 23.1.0
-    hooks:
-      - id: black
-        language_version: python3
-        args:
-          - "--line-length=100"
-  - repo: https://github.com/pycqa/isort
-    rev: 5.12.0
-    hooks:
-      - id: isort
-        name: isort (python)
-  - repo: https://github.com/pycqa/flake8
-    rev: 6.0.0
-    hooks:
-      - id: flake8
-  - repo: https://github.com/pycqa/pydocstyle
-    rev: 6.3.0
-    hooks:
-      - id: pydocstyle
-        args:
-          - --convention=google
-          - --match="(?!test_).*\.py"
-        exclude: |
-          (?x)(
-              ^docs/source/conf.py|
-              ^conftest.py|
-              ^tests/
-          )
-  - repo: https://github.com/kynan/nbstripout
-    rev: 0.6.1
-    hooks:
-      - id: nbstripout
-        args:
-          - --keep-output
-  - repo: https://github.com/floatingpurr/sync_with_pdm
-    rev: "0.3.0"
-    hooks:
-      - id: sync_with_pdm
-        args: [] # optional args
diff --git a/.pylintrc b/.pylintrc
@@ -0,0 +1,29 @@
+[TYPECHECK]
+
+
+disable=
+	# Not sure how to fix this for Metrics specifically
+	arguments-differ,
+
+	# we have too a lot of short variable names
+	invalid-name,
+
+	# doesn't seem helpful
+	too-few-public-methods,
+
+	# not sure it's advisable to "fix" these
+	too-many-arguments,
+	too-many-locals,
+	too-many-instance-attributes,
+
+	# when overriding methods you don't get to pick your arguments
+	unused-argument,
+
+    # sep raises this 
+    c-extension-no-member,
+
+
+[SIMILARITIES]
+
+# Minimum lines number of a similarity to report duplicate-code
+min-similarity-lines=11
diff --git a/LICENSE b/LICENSE
@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2014-2021 by the BlendingToolKit contributors on Github at <https://github.com/LSSTDESC/BlendingToolKit>. All rights reserved.
+Copyright (c) 2014-2023 by the BlendingToolKit contributors on Github at <https://github.com/LSSTDESC/BlendingToolKit>. All rights reserved.
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal