Added abstract base class for DCT #25

tdmorello · 2022-01-14T23:55:44Z

Add abstract base class for DCT to provide a common interface to each DCT backend.

tdmorello · 2022-01-15T00:16:17Z

setup.cfg

@@ -1,35 +1,35 @@
 [metadata]
 name = pybasic
 version = 0.0.1
-author = Lorenz Lamm, Tingying Peng, Mohammad Mirkazemi
-author_email = [email protected], [email protected]
 description = A python package for background and shading correction of optical microscopy images
 long_description = file: README.md
 long_description_content_type = text/markdown
 url = https://github.com/peng-lab/PyBaSiC


Also, add yourself as an author now that you've contributed :)

tdmorello · 2022-01-15T00:17:24Z

setup.cfg

 keywords = background shading flatfield darkfield biology optical microscopy image  # TODO add illumination
+project_urls =
+    Bug Tracker = https://github.com/peng-lab/PyBaSiC/issues


tdmorello · 2022-01-15T00:17:24Z

setup.cfg

@@ -39,7 +39,11 @@ console_scripts =
    pybasic = pybasic.__main__:main


Change package name

tdmorello · 2022-01-15T00:19:08Z

src/pybasic/tools/dct2d_tools.py

+DCT_BACKENDS = {"JAX": JaxDCT, "OPENCV": OpenCVDCT, "SCIPY": SciPyDCT}
+
+
+dct = DCT_BACKENDS[os.environ["DCT_BACKEND"] or DEFAULT_BACKEND]


Change to try except KeyError

For the environment variable, I recommend adding a package name prefix such as BASIC_DCT_BACKEND.

tdmorello · 2022-01-15T00:19:39Z

src/pybasic/tools/dct2d_tools.py

+        ...
+
+
+DCT_BACKENDS = {"JAX": JaxDCT, "OPENCV": OpenCVDCT, "SCIPY": SciPyDCT}


Instantiate classes to make sure they follow ABC requirements

Nicholas-Schaub

Overall I think this looks excellent. There are a few things to resolve/test to make sure optional dependencies are actually optional (the code will run without the optional dependencies). Other than that it looks good.

If you want to take a stab at writing benchmarks, excellent. Otherwise, could you open up a new issue to write benchmarks for this? You should be able to find an example of benchmarks in the benchmarks folder.

Nicholas-Schaub · 2022-01-15T19:33:32Z

src/pybasic/tools/dct2d_tools.py

+if os.environ.get("BASIC_DCT_BACKEND"):
+    try:
+        dct = DCT_BACKENDS[os.environ["BASIC_DCT_BACKEND"]]
+    except KeyError as e:
+        # TODO change to logging or warning
+        raise e
+        # # OR when this fails, revert to default
+        # print(
+        #     f"unrecognized dct backend {os.environ['BASIC_DCT_BACKEND']}, "
+        #     f"defaulting to {DEFAULT_BACKEND}"
+        # )
+        # dct = DCT_BACKENDS[DEFAULT_BACKEND]
+else:
+    dct = DCT_BACKENDS[DEFAULT_BACKEND]


Maybe replace this with:

backend = os.environ.get("BASIC_DCT_BACKEND") dct = DCT_BACKENDS.get(backend,DCT_BACKENDS[DEFAULT_BACKEND])

This will reduce the amount of code, but will also grab DEFAULT_BACKEND if the supplied backend is invalid.

When we add logs, we should further enhance this by adding a debug message to show what backend was supplied versus what ends up being used, and display a warning if we grab the default when a different backend was specified.

I took this suggestion. It is tagged with a TODO to add logging.

I need to troubleshoot asv because it is not working on my computer. I'll make an issue to write the benchmarks since this merged, but I'm going to try and get it working.

Nicholas-Schaub · 2022-01-15T19:36:37Z

src/pybasic/tools/dct2d_tools.py

+try:
+    import cv2
+except ImportError:
+    print("unable to import cv2")
+try:
+    import scipy.fft
+except ImportError:
+    print("unable to import scipy")


I think it might be a good idea to include package existence variables inside of each try blocks such as has_cv2. Then, when we define classes, we can put each class into an if has_(package) block. Otherwise, I think the way the code is currently written will throw an error if the packages were not imported.

This will also allow us to easily automate testing for packages that someone does actually have installed, because then we can just get all defined subclasses of DCT and run the same function in the unit tests.

Good point. I incorporated these changes.

Nicholas-Schaub · 2022-01-15T19:50:54Z

src/pybasic/tools/dct2d_tools.py

+        return scipy.fft.idct(scipy.fft.idct(arr.T, norm="ortho").T, norm="ortho")
+
+
+DCT_BACKENDS = {"JAX": JaxDCT(), "OPENCV": OpenCVDCT(), "SCIPY": SciPyDCT()}


I think this will throw errors if not all dependencies are installed.

One way to resolve this would be:

DCT_BACKENDS = {} for sc in DCT.__subclasses__(): name = sc.__name__.upper()[:-3] DCT_BACKENDS[name] = sc()

- optimise SVD choice - small other cleanup

…aSiCPy into dev

@yfukai

* Add badges to README * Update .gitignore * Add noxfile * Move files to src * Add config files * Placeholder dct functions * Update gitignore * Add class for settings * Update setup.cfg * Update requirements.txt * Add docstring to settings class * Add validation to Settings input * Update github CI * Add gpu option to init * Update README formatting * Rename 'figures' folder to 'resources' * Update gitignore * Update gitignore * Move API from README to docs * Refactor, update docstrings * Change folder ExampleData to data * Remove environment file * API development * Read the docs config * Remove png from output formats * Update Profile docs * Update docstrings * Fix imports * Fix docs... again * Update tests * Update workflow * Fix workflow * Fix workflows * Fix tests * Delete requirements.txt * Added benchmark library asv * Revert "Delete CONTRIBUTING.rst" This reverts commit 01d8fe1. * Revert "Added benchmark library asv" This reverts commit d067f04. * Deleted CONTRIBUTING.rst * Create VERSION * Update README.md * Settings becomes pydantic dataclass * Update dependencies * Update types * Update pre-commit * Update types, variable names * Rename variables * Add properties to Profile class * Add properties for getting/setting BaSiC settings * Add dct types to docs * Add abstract base class for DCT * Feat/benchmarks (#7) * Added benchmark library asv * Revert "Delete CONTRIBUTING.rst" This reverts commit 01d8fe1. * Delete CONTRIBUTING.rst * Changed project/repository names * Changed to BaSiCPy * Update pre-commit * Add dct backends and tests * Update setup.cfg * Quickfix for failing tests * Converted `BaSiC` to a pydantic class (#23) * Converted BaSiC to pydantic class. * Changed get_dark to get_darkfield * Added CLI parsing (#26) * Converted BaSiC to pydantic class. * Created command line inputs. * Added metavar indicator to input arguments. * Check package installed before making DCT class * Remove unused import * Added abstract base class for DCT (#25) * Add abstract base class for DCT * Update pre-commit * Add dct backends and tests * Update setup.cfg * Quickfix for failing tests * Update inexact_alm_rspca_l1.py (#1) - optimise SVD choice - small other cleanup * Check package installed before making DCT class * Remove unused import Co-authored-by: Nicholas-Schaub <[email protected]> Co-authored-by: Mahdi Lamb <[email protected]> * Added predict method, does not include baseline estimation (#29) * Implemented shading model fit algorithm from main branch (#27) * Implemented shading model fit algorithm from main branch * Fixed idct in inexact_alm.py * Fix/fit (#34) * Added benchmark library asv * Revert "Delete CONTRIBUTING.rst" This reverts commit 01d8fe1. * Revert "Added benchmark library asv" This reverts commit d067f04. * Deleted CONTRIBUTING.rst * Set score values to object attributes * Added caching to `predict` method (#35) * Added benchmark library asv * Revert "Delete CONTRIBUTING.rst" This reverts commit 01d8fe1. * Revert "Added benchmark library asv" This reverts commit d067f04. * Deleted CONTRIBUTING.rst * Set score values to object attributes * Added caching for resizing of flatfield/darkfield * Feat/autodoc pydantic (#38) * Add abstract base class for DCT * Update pre-commit * Add dct backends and tests * Update setup.cfg * Quickfix for failing tests * Update inexact_alm_rspca_l1.py (#1) - optimise SVD choice - small other cleanup * Check package installed before making DCT class * Remove unused import * Implement autodoc_pydantic Co-authored-by: Nicholas-Schaub <[email protected]> Co-authored-by: Mahdi Lamb <[email protected]> * Feat/cleanup (#39) * Add abstract base class for DCT * Update pre-commit * Add dct backends and tests * Update setup.cfg * Quickfix for failing tests * Update inexact_alm_rspca_l1.py (#1) - optimise SVD choice - small other cleanup * Check package installed before making DCT class * Remove unused import * Remove Profiles class and methods from basic Co-authored-by: Nicholas-Schaub <[email protected]> Co-authored-by: Mahdi Lamb <[email protected]> * Convert relative to absolute imports * Add pytest-benchmark * Update noxfile for pytest-benchmark * ignored python-version * Update .gitignore * switched to commented line * refactor name to basicpy * further refactored... * changed dir name * refactor `predict` to `transform` * Update README.md change PyBaSiC to BaSiCPy * Implement logging (#44) * added logging package to all * trying to fix logging in main * initial attempt to add logger logging to each files * fixed typo * fixed main logger typo * fixed import style * added actual logs in the calculation * changed logger to class-level * added log for inexact_alm * fixed main.py logging level and added warning in dct2d_tools. * made BaSiC logger to file-level * Feat/jax dct (#46) * Add custom JAX idct function * Refactor idct * Fix tests Co-authored-by: Fukai Yohsuke <[email protected]> * Add save and load methods (#43) * Add save and load methods * Update docstring * Update load_model test * Add overwrite option and tests to save_model * Update save/load tests Co-authored-by: Fukai Yohsuke <[email protected]> * Update README.md (#49) * Update README.md * Update README.md * removed dropbox link from the link * Update README.md Co-authored-by: Fukai Yohsuke <[email protected]> * fixed pydantic version lower bound to get test_basic_save_model succeeded * set the license to MIT * removed pydantic version * Testing fit method with synthetic data (#53) * added logging package to all * trying to fix logging in main * initial attempt to add logger logging to each files * fixed typo * fixed main logger typo * fixed import style * added actual logs in the calculation * changed logger to class-level * added log for inexact_alm * added synthesized fit test * changed threshold for error * fixed typo for sign * added pooch for Zenodo datasets * some fix... * fixed pooch data bug * Revert "added pooch for Zenodo datasets" This reverts commit 4fdf0d7. * removed logger from class * refactor and added code for debug plotting in test_basic_fit_synthesized * Data in package (#57) * added files * added fetch code to data module * formatted data... * added data download function * added docstring * updated comment * added docstring for all * added test download * added pooch for dependency * typo fix... * fixed typo... * parametrized tests * disabled hash checking * fixed bug * Revert "fixed bug" This reverts commit 6b24503. * Revert "disabled hash checking" This reverts commit d0dbd8a. * added data and tests * changed data url to the dev branch * added tests * added pydantic version to get test_basic_save_model succeed * added tests for original dataset * added code for original fetch * added exception for invalid data cases * reference commit with * switched to dev branch * Fixed typo for data.py comments * Major update with LADMAP implementation (#62) * added logging package to all * trying to fix logging in main * initial attempt to add logger logging to each files * fixed typo * fixed main logger typo * Add custom JAX idct function * Refactor idct * fixed import style * added actual logs in the calculation * changed logger to class-level * added log for inexact_alm * added synthesized fit test * changed threshold for error * fixed typo for sign * added pooch for Zenodo datasets * some fix... * fixed pooch data bug * Revert "added pooch for Zenodo datasets" This reverts commit 4fdf0d7. * wrote tests for experimental data * added files * added fetch code to data module * formatted data... * added data download function * added docstring * updated comment * added docstring for all * added test download * added pooch for dependency * typo fix... * fixed typo... * parametrized tests * disabled hash checking * fixed bug * Revert "fixed bug" This reverts commit 6b24503. * Revert "disabled hash checking" This reverts commit d0dbd8a. * added jax experimental code and tested tingying's implementation * experimental implementation wroking to approximate darkfield * at least jit version is working * deleted old version codes * refactor to include D_R and D_Z * added darkfield 1 * experimental code working * loop jit * fixed package name * merged Tim's idct * jax impl working with Tim's idct * ladmap code is fed into the basicpy class (still a bit slow?) * converted _fit_ladmap_single to the member method * class method working... * refactored jnp.newaxis * in-class version working without jit * trying to add pytree property to the model but aborting this approach * writing jitted routine with new pydantic classes * added fields to jax routines * jit working * all jax working!! * apprioximate method not working with jax * confirmed approximate fit working in jax * organized files * import organize * single fit is working, but reweighting iteration is not working * reweighting loops is working! * still working example * still running... * resize implemented * fixed resize in transform * added _step_only_baseline for approximate fit * custom weight calc working * removed logger from class * approximate fit optimization changed to original imp. * working version with custom weight * working before baseline calculation * baseline calc working except approximate cases * trying to get baseline work for approximate fit * dropped python 3.7 * still approximate version not working... * some working version * approximate version working except baseline * baseline calc not working for approximate cases * sort_intensity option working for approximate fit! * removed comments * analyzing difference b/w original implementation * refactor and added code for debug plotting in test_basic_fit_synthesized * added data and tests * changed data url to the dev branch * added tests * added pydantic version to get test_basic_save_model succeed * added tests for original dataset * added code for original fetch * added exception for invalid data cases * reference commit with * switched to dev branch * added comparison w/ reference implementation at this time * Fixed typo for data.py comments * updated notebook * at least original impl is reproduced to some extent * slightly changed transform tests to get them working * Revert "Merge branch 'data_in_package' into jax_implementation" This reverts commit 9d30051, reversing changes made to a3d35ff. * deleted experimental note * added relative fitting weight argment * added fitting results at commit 11b0747 for testing * test rewriting but still failing * test working * added test with experimental data * added pytest-datadir in nox * migrated to pytest-datadir * made darkfield sparsity coef tunable * Revert "made darkfield sparsity coef tunable" This reverts commit 2b8cd5d. * loosen the isclose torelance * fixed typo in tests * deleted reference results * removed reference results * added additional darkfield coef and test passing * trying to modify dimensions * rewriting notebooks * deleted previous notebooks * notebook working * readme updated * example notebook added * fixed type * at least approximate method is working! * fixed some bugs * tests are running with loosened checks for experimental data * added 3D DCT * loosen the torelance for approximate fit * loosen darkfield torelance for approximate fit * changed DCT from 2D to 3D in ladmap fit * loosen torelance for ladmap * small modification * added constraint for S mean * get the test working * before adding positivity constraint * changed initial config * added naive mean S regularization * rewrote organizing test data notebook * added value_diff for update torelance * fixed convergence bug * ladmap working with modified stopping cond * working with modified stopping cond. (still harmonic ones not added) * added test data * removed cell output for jupyter notebook * removed cell output for jupyter notebook * updated test * loosened the darkfield test constraint * added guard for S==0 * updated example for lambda_flatfield_coef=100 * lambda_flatfield_coef=100 example update * updated flatfield and darkfield dimension for 3d images * updating organize test notebook * updated notebook path * notebook updated * re-added python 3.7 * added 3D test * refactored synthetic test * fixed tests * reduced z size in test * added rescue for RESOURCE_EXHAUSTED: Out of memory * updated enum type to str, Enum Co-authored-by: Tim Morello <[email protected]> Co-authored-by: Nicholas-Schaub <[email protected]> * Update contribution notes (#66) * initial draft * added git version for the original * 👥 Add @yfukai as a contributor * 👥 Update @yfukai as a contributor * 👥 Update @yfukai as a contributor * 👥 Update @yfukai as a contributor * 👥 Update @yfukai as a contributor * 👥 Add @tying84 as a contributor * 👥 Update @tying84 as a contributor * 👥 Update @tying84 as a contributor * 👥 Add @tdmorello as a contributor * 👥 Update @tdmorello as a contributor * 👥 Update @tdmorello as a contributor * 👥 Add @Nicholas-Schaub as a contributor * 👥 Update @Nicholas-Schaub as a contributor * 👥 Update @Nicholas-Schaub as a contributor * 👥 Update @Nicholas-Schaub as a contributor * 👥 Update @Nicholas-Schaub as a contributor * trying to see how all-contributors works * reordered contributors * regenerated readme * commenting on contrinbution * organized README * updated nick email, added contributions to Tingying * removed experimental_code reference * updated jaxlib version to avoid importerror (#69) * Merge Yu's docs * deleting files not used at this point * enab * re-added autoflake and isort * removed opencv * Feat/release (#72) * setup.cfg * Create release pipeline * Updated config and added new details to README * Fixed build pipeline, added MANIFEST * Fixed formatting issues * Fixed typo in README * Moved bumpversion config to separate file * Fixed whitespace in bumpversion config * Exclude bumpversion config from whitespace checks * Bump version: 0.1.0-dev0 → 0.2.0-dev0 * Fix/release (#75) * setup.cfg * Create release pipeline * Updated config and added new details to README * Fixed build pipeline, added MANIFEST * Fixed formatting issues * Fixed typo in README * Moved bumpversion config to separate file * Fixed whitespace in bumpversion config * Exclude bumpversion config from whitespace checks * Bump version: 0.1.0-dev0 → 0.2.0-dev0 * Modified config, updated VERSION * Bump version: 0.2.0-dev0 → 0.2.0-dev1 * Update secrets environment * Bump version: 0.2.0-dev1 → 0.2.0-dev2 * Updated twine upload command * Bump version: 0.2.0-dev2 → 0.2.0-dev3 * Updated CI pipeline * Bump version: 0.2.0-dev3 → 0.2.0-dev4 * Use pypa publishing container * Bump version: 0.2.0-dev4 → 0.2.0-dev5 * Update build * Bump version: 0.2.0-dev5 → 0.2.0-dev6 * 👥 Update @Nicholas-Schaub as a contributor * 👥 Update @tdmorello as a contributor * 👥 Add @YuLiu-web as a contributor * update readme * 👥 Update @tying84 as a contributor * add tingying organizing meetings * Fix `lambda flatfield` bug (#74) * fixed bug for 3d images and lambda_flatfield calc * reorganized test data * Bump version: 0.2.0-dev6 → 0.2.1-dev0 * fixed save bug for 3D images and added tests (#77) * updated jaxlib dependency * updated notebooks * 👥 Update @tdmorello as a contributor * some update * fixed typos * Update docs * fixed typos * changed default parameters * updated examples * Add comments for notebook gallery * Update notebooks * compared lagrangian with the origin implementation * notebook update * formatted notebooks with black * added markdowns to notebooks * ifurther updated notebooks * Add contributing guide to docs * Add autoenum to docs * updated notebooks * updated thumbnail data for docs * updated readme * ignored package lock file for yarm * fixed type * added windows instruction * 👥 Update @YuLiu-web as a contributor * fixed notebook folder * fixed typo * added macos and windows tests * test falling back to ubuntu * added jax installation in noxfile for windows * migrated contribution details to the docs * update badges * fixed badges * a bit more update... * fixed readme * fixed badgee stype * further fix... * Timelapse option in `transform` (#80) * udpated transform method * fixed tests * fixed typo * solving problems with tests * fixed transform bug * removed tests for rescaling-fitting * removed resized transform example * added dask test * set baseline type to optinal * updated testdata * deleted notebook outputs * Update timelapse * re-organized transform part * tests running... * made the resize method changable * resize method tested * tested resize method * added comments for the transform function * minor fix * fixed fitting_weight shape bug Co-authored-by: Tim Morello <[email protected]> * Update example data (#92) * updating notebooks * updated rescaled data * updated test data * update notebooks * Fixed most unit tests * re-added reference comparison notebook * re-added test data Co-authored-by: Nicholas-Schaub <[email protected]> * Fix `fit` bug for dask array inputs (#93) * udpated transform method * fixed tests * fixed typo * solving problems with tests * fixed transform bug * removed tests for rescaling-fitting * removed resized transform example * added dask test * set baseline type to optinal * updated testdata * deleted notebook outputs * Update timelapse * re-organized transform part * tests running... * made the resize method changable * resize method tested * tested resize method * added comments for the transform function * minor fix * fixed fitting_weight shape bug * fixed dask-related bug Co-authored-by: Tim Morello <[email protected]> * Update hashes (#95) * Minor change before beta release (#96) * DCT tools update (#98) * updated hashes to newer ones * changed source to main * ci experiment with dev branch * updated branch * added tests for wrong argments * updated classifier to beta * updated copyright * just added spaces * updated downsampled data to Zenodo * renamed data to datasets * typo fix... * deleted depricated tools * fixed package names ... * import fix * 👥 Update @yfukai as a contributor * added my contribution for ci... * removed deprecated routines from old DCT2d * updated tests and fixed jax version to solve problems * Update coef definitions (#103) * updated hashes to newer ones * changed source to main * ci experiment with dev branch * updated branch * added tests for wrong argments * updated classifier to beta * updated copyright * just added spaces * updated downsampled data to Zenodo * renamed data to datasets * typo fix... * deleted depricated tools * fixed package names ... * import fix * 👥 Update @yfukai as a contributor * added my contribution for ci... * removed deprecated routines from old DCT2d * trying to add darkfield weight and changing weight for ladmap * fixed typo * added attribute * updated tests and fixed jax version to solve problems * rewrite ladmap to normalize B * working ver? * updated test data * udpated weight for darkfield * updated test data * removed print * fixed normalization const * added test to show basic does not depend on the intensity scale * small update * Update config names (#106) * updated hashes to newer ones * changed source to main * ci experiment with dev branch * updated branch * added tests for wrong argments * updated classifier to beta * updated copyright * just added spaces * updated downsampled data to Zenodo * renamed data to datasets * typo fix... * deleted depricated tools * fixed package names ... * import fix * 👥 Update @yfukai as a contributor * added my contribution for ci... * removed deprecated routines from old DCT2d * trying to add darkfield weight and changing weight for ladmap * fixed typo * added attribute * updated tests and fixed jax version to solve problems * rewrite ladmap to normalize B * working ver? * updated test data * udpated weight for darkfield * updated test data * removed print * fixed normalization const * added test to show basic does not depend on the intensity scale * small update * updated coefs * updated field names * fixed typo in log * updated the definition for self._sparse_cost_darkfield in approximate * Update documents (#107) * updated hashes to newer ones * changed source to main * ci experiment with dev branch * updated branch * added tests for wrong argments * updated classifier to beta * updated copyright * just added spaces * updated downsampled data to Zenodo * renamed data to datasets * typo fix... * deleted depricated tools * fixed package names ... * import fix * 👥 Update @yfukai as a contributor * added my contribution for ci... * removed deprecated routines from old DCT2d * trying to add darkfield weight and changing weight for ladmap * fixed typo * added attribute * updated tests and fixed jax version to solve problems * rewrite ladmap to normalize B * working ver? * updated test data * udpated weight for darkfield * updated test data * removed print * fixed normalization const * added test to show basic does not depend on the intensity scale * small update * updated coefs * updated field names * fixed typo in log * updated parameters for notebooks * added docs_build * added ci for docs * changed the notebooks so that it only installs basicpy for colab * updated docs * added pandoc in ci * fixed error typo * Cleaned up code and logging. * Changed error message comment * Add single-step convergence warning after the loop * Cleanup (#109) * Added benchmark library asv * Revert "Delete CONTRIBUTING.rst" This reverts commit 01d8fe1. * Revert "Added benchmark library asv" This reverts commit d067f04. * Deleted CONTRIBUTING.rst * Cleaned up code and logging. * Changed error message comment * Add single-step convergence warning after the loop Co-authored-by: Yohsuke T. Fukai <[email protected]> * Change default log level to warning * removed device variable from the basic object (#110) Co-authored-by: Nicholas-Schaub <[email protected]> * fixed requirements for gallery fix (#112) * Bump version: 0.2.1-dev0 → 1.0.0 * Bump version: 0.2.1-dev0 → 1.0.0 * Updated docs version and bumpversion config * Bump version: 1.0.0 → 1.0.1 Co-authored-by: Tim Morello <[email protected]> Co-authored-by: Tim Morello <[email protected]> Co-authored-by: Nicholas-Schaub <[email protected]> Co-authored-by: Nicholas-Schaub <[email protected]> Co-authored-by: Mahdi Lamb <[email protected]> Co-authored-by: Tingying Peng <[email protected]> Co-authored-by: YuLiu-web <[email protected]>

@yfukai

* Add badges to README * Update .gitignore * Add noxfile * Move files to src * Add config files * Placeholder dct functions * Update gitignore * Add class for settings * Update setup.cfg * Update requirements.txt * Add docstring to settings class * Add validation to Settings input * Update github CI * Add gpu option to init * Update README formatting * Rename 'figures' folder to 'resources' * Update gitignore * Update gitignore * Move API from README to docs * Refactor, update docstrings * Change folder ExampleData to data * Remove environment file * API development * Read the docs config * Remove png from output formats * Update Profile docs * Update docstrings * Fix imports * Fix docs... again * Update tests * Update workflow * Fix workflow * Fix workflows * Fix tests * Delete requirements.txt * Create VERSION * Update README.md * Settings becomes pydantic dataclass * Update dependencies * Update types * Update pre-commit * Update types, variable names * Rename variables * Add properties to Profile class * Add properties for getting/setting BaSiC settings * Add dct types to docs * Add abstract base class for DCT * Feat/benchmarks (#7) * Added benchmark library asv * Revert "Delete CONTRIBUTING.rst" This reverts commit 01d8fe1. * Delete CONTRIBUTING.rst * Changed project/repository names * Changed to BaSiCPy * Update pre-commit * Add dct backends and tests * Update setup.cfg * Quickfix for failing tests * Converted `BaSiC` to a pydantic class (#23) * Converted BaSiC to pydantic class. * Changed get_dark to get_darkfield * Added CLI parsing (#26) * Converted BaSiC to pydantic class. * Created command line inputs. * Added metavar indicator to input arguments. * Check package installed before making DCT class * Remove unused import * Added abstract base class for DCT (#25) * Add abstract base class for DCT * Update pre-commit * Add dct backends and tests * Update setup.cfg * Quickfix for failing tests * Update inexact_alm_rspca_l1.py (#1) - optimise SVD choice - small other cleanup * Check package installed before making DCT class * Remove unused import Co-authored-by: Nicholas-Schaub <[email protected]> Co-authored-by: Mahdi Lamb <[email protected]> * Added predict method, does not include baseline estimation (#29) * Implemented shading model fit algorithm from main branch (#27) * Implemented shading model fit algorithm from main branch * Fixed idct in inexact_alm.py * Fix/fit (#34) * Added benchmark library asv * Revert "Delete CONTRIBUTING.rst" This reverts commit 01d8fe1. * Revert "Added benchmark library asv" This reverts commit d067f04. * Deleted CONTRIBUTING.rst * Set score values to object attributes * Added caching to `predict` method (#35) * Added benchmark library asv * Revert "Delete CONTRIBUTING.rst" This reverts commit 01d8fe1. * Revert "Added benchmark library asv" This reverts commit d067f04. * Deleted CONTRIBUTING.rst * Set score values to object attributes * Added caching for resizing of flatfield/darkfield * Feat/autodoc pydantic (#38) * Add abstract base class for DCT * Update pre-commit * Add dct backends and tests * Update setup.cfg * Quickfix for failing tests * Update inexact_alm_rspca_l1.py (#1) - optimise SVD choice - small other cleanup * Check package installed before making DCT class * Remove unused import * Implement autodoc_pydantic Co-authored-by: Nicholas-Schaub <[email protected]> Co-authored-by: Mahdi Lamb <[email protected]> * Feat/cleanup (#39) * Add abstract base class for DCT * Update pre-commit * Add dct backends and tests * Update setup.cfg * Quickfix for failing tests * Update inexact_alm_rspca_l1.py (#1) - optimise SVD choice - small other cleanup * Check package installed before making DCT class * Remove unused import * Remove Profiles class and methods from basic Co-authored-by: Nicholas-Schaub <[email protected]> Co-authored-by: Mahdi Lamb <[email protected]> * Convert relative to absolute imports * Add pytest-benchmark * Update noxfile for pytest-benchmark * ignored python-version * Update .gitignore * switched to commented line * refactor name to basicpy * further refactored... * changed dir name * refactor `predict` to `transform` * Update README.md change PyBaSiC to BaSiCPy * Implement logging (#44) * added logging package to all * trying to fix logging in main * initial attempt to add logger logging to each files * fixed typo * fixed main logger typo * fixed import style * added actual logs in the calculation * changed logger to class-level * added log for inexact_alm * fixed main.py logging level and added warning in dct2d_tools. * made BaSiC logger to file-level * Feat/jax dct (#46) * Add custom JAX idct function * Refactor idct * Fix tests Co-authored-by: Fukai Yohsuke <[email protected]> * Add save and load methods (#43) * Add save and load methods * Update docstring * Update load_model test * Add overwrite option and tests to save_model * Update save/load tests Co-authored-by: Fukai Yohsuke <[email protected]> * Update README.md (#49) * Update README.md * Update README.md * removed dropbox link from the link * Update README.md Co-authored-by: Fukai Yohsuke <[email protected]> * fixed pydantic version lower bound to get test_basic_save_model succeeded * set the license to MIT * removed pydantic version * Testing fit method with synthetic data (#53) * added logging package to all * trying to fix logging in main * initial attempt to add logger logging to each files * fixed typo * fixed main logger typo * fixed import style * added actual logs in the calculation * changed logger to class-level * added log for inexact_alm * added synthesized fit test * changed threshold for error * fixed typo for sign * added pooch for Zenodo datasets * some fix... * fixed pooch data bug * Revert "added pooch for Zenodo datasets" This reverts commit 4fdf0d7. * removed logger from class * refactor and added code for debug plotting in test_basic_fit_synthesized * Data in package (#57) * added files * added fetch code to data module * formatted data... * added data download function * added docstring * updated comment * added docstring for all * added test download * added pooch for dependency * typo fix... * fixed typo... * parametrized tests * disabled hash checking * fixed bug * Revert "fixed bug" This reverts commit 6b24503. * Revert "disabled hash checking" This reverts commit d0dbd8a. * added data and tests * changed data url to the dev branch * added tests * added pydantic version to get test_basic_save_model succeed * added tests for original dataset * added code for original fetch * added exception for invalid data cases * reference commit with * switched to dev branch * Fixed typo for data.py comments * Major update with LADMAP implementation (#62) * added logging package to all * trying to fix logging in main * initial attempt to add logger logging to each files * fixed typo * fixed main logger typo * Add custom JAX idct function * Refactor idct * fixed import style * added actual logs in the calculation * changed logger to class-level * added log for inexact_alm * added synthesized fit test * changed threshold for error * fixed typo for sign * added pooch for Zenodo datasets * some fix... * fixed pooch data bug * Revert "added pooch for Zenodo datasets" This reverts commit 4fdf0d7. * wrote tests for experimental data * added files * added fetch code to data module * formatted data... * added data download function * added docstring * updated comment * added docstring for all * added test download * added pooch for dependency * typo fix... * fixed typo... * parametrized tests * disabled hash checking * fixed bug * Revert "fixed bug" This reverts commit 6b24503. * Revert "disabled hash checking" This reverts commit d0dbd8a. * added jax experimental code and tested tingying's implementation * experimental implementation wroking to approximate darkfield * at least jit version is working * deleted old version codes * refactor to include D_R and D_Z * added darkfield 1 * experimental code working * loop jit * fixed package name * merged Tim's idct * jax impl working with Tim's idct * ladmap code is fed into the basicpy class (still a bit slow?) * converted _fit_ladmap_single to the member method * class method working... * refactored jnp.newaxis * in-class version working without jit * trying to add pytree property to the model but aborting this approach * writing jitted routine with new pydantic classes * added fields to jax routines * jit working * all jax working!! * apprioximate method not working with jax * confirmed approximate fit working in jax * organized files * import organize * single fit is working, but reweighting iteration is not working * reweighting loops is working! * still working example * still running... * resize implemented * fixed resize in transform * added _step_only_baseline for approximate fit * custom weight calc working * removed logger from class * approximate fit optimization changed to original imp. * working version with custom weight * working before baseline calculation * baseline calc working except approximate cases * trying to get baseline work for approximate fit * dropped python 3.7 * still approximate version not working... * some working version * approximate version working except baseline * baseline calc not working for approximate cases * sort_intensity option working for approximate fit! * removed comments * analyzing difference b/w original implementation * refactor and added code for debug plotting in test_basic_fit_synthesized * added data and tests * changed data url to the dev branch * added tests * added pydantic version to get test_basic_save_model succeed * added tests for original dataset * added code for original fetch * added exception for invalid data cases * reference commit with * switched to dev branch * added comparison w/ reference implementation at this time * Fixed typo for data.py comments * updated notebook * at least original impl is reproduced to some extent * slightly changed transform tests to get them working * Revert "Merge branch 'data_in_package' into jax_implementation" This reverts commit 9d30051, reversing changes made to a3d35ff. * deleted experimental note * added relative fitting weight argment * added fitting results at commit 11b0747 for testing * test rewriting but still failing * test working * added test with experimental data * added pytest-datadir in nox * migrated to pytest-datadir * made darkfield sparsity coef tunable * Revert "made darkfield sparsity coef tunable" This reverts commit 2b8cd5d. * loosen the isclose torelance * fixed typo in tests * deleted reference results * removed reference results * added additional darkfield coef and test passing * trying to modify dimensions * rewriting notebooks * deleted previous notebooks * notebook working * readme updated * example notebook added * fixed type * at least approximate method is working! * fixed some bugs * tests are running with loosened checks for experimental data * added 3D DCT * loosen the torelance for approximate fit * loosen darkfield torelance for approximate fit * changed DCT from 2D to 3D in ladmap fit * loosen torelance for ladmap * small modification * added constraint for S mean * get the test working * before adding positivity constraint * changed initial config * added naive mean S regularization * rewrote organizing test data notebook * added value_diff for update torelance * fixed convergence bug * ladmap working with modified stopping cond * working with modified stopping cond. (still harmonic ones not added) * added test data * removed cell output for jupyter notebook * removed cell output for jupyter notebook * updated test * loosened the darkfield test constraint * added guard for S==0 * updated example for lambda_flatfield_coef=100 * lambda_flatfield_coef=100 example update * updated flatfield and darkfield dimension for 3d images * updating organize test notebook * updated notebook path * notebook updated * re-added python 3.7 * added 3D test * refactored synthetic test * fixed tests * reduced z size in test * added rescue for RESOURCE_EXHAUSTED: Out of memory * updated enum type to str, Enum Co-authored-by: Tim Morello <[email protected]> Co-authored-by: Nicholas-Schaub <[email protected]> * Update contribution notes (#66) * initial draft * added git version for the original * 👥 Add @yfukai as a contributor * 👥 Update @yfukai as a contributor * 👥 Update @yfukai as a contributor * 👥 Update @yfukai as a contributor * 👥 Update @yfukai as a contributor * 👥 Add @tying84 as a contributor * 👥 Update @tying84 as a contributor * 👥 Update @tying84 as a contributor * 👥 Add @tdmorello as a contributor * 👥 Update @tdmorello as a contributor * 👥 Update @tdmorello as a contributor * 👥 Add @Nicholas-Schaub as a contributor * 👥 Update @Nicholas-Schaub as a contributor * 👥 Update @Nicholas-Schaub as a contributor * 👥 Update @Nicholas-Schaub as a contributor * 👥 Update @Nicholas-Schaub as a contributor * trying to see how all-contributors works * reordered contributors * regenerated readme * commenting on contrinbution * organized README * updated nick email, added contributions to Tingying * removed experimental_code reference * updated jaxlib version to avoid importerror (#69) * Merge Yu's docs * deleting files not used at this point * enab * re-added autoflake and isort * removed opencv * Feat/release (#72) * setup.cfg * Create release pipeline * Updated config and added new details to README * Fixed build pipeline, added MANIFEST * Fixed formatting issues * Fixed typo in README * Moved bumpversion config to separate file * Fixed whitespace in bumpversion config * Exclude bumpversion config from whitespace checks * Bump version: 0.1.0-dev0 → 0.2.0-dev0 * Fix/release (#75) * setup.cfg * Create release pipeline * Updated config and added new details to README * Fixed build pipeline, added MANIFEST * Fixed formatting issues * Fixed typo in README * Moved bumpversion config to separate file * Fixed whitespace in bumpversion config * Exclude bumpversion config from whitespace checks * Bump version: 0.1.0-dev0 → 0.2.0-dev0 * Modified config, updated VERSION * Bump version: 0.2.0-dev0 → 0.2.0-dev1 * Update secrets environment * Bump version: 0.2.0-dev1 → 0.2.0-dev2 * Updated twine upload command * Bump version: 0.2.0-dev2 → 0.2.0-dev3 * Updated CI pipeline * Bump version: 0.2.0-dev3 → 0.2.0-dev4 * Use pypa publishing container * Bump version: 0.2.0-dev4 → 0.2.0-dev5 * Update build * Bump version: 0.2.0-dev5 → 0.2.0-dev6 * 👥 Update @Nicholas-Schaub as a contributor * 👥 Update @tdmorello as a contributor * 👥 Add @YuLiu-web as a contributor * update readme * 👥 Update @tying84 as a contributor * add tingying organizing meetings * Fix `lambda flatfield` bug (#74) * fixed bug for 3d images and lambda_flatfield calc * reorganized test data * Bump version: 0.2.0-dev6 → 0.2.1-dev0 * fixed save bug for 3D images and added tests (#77) * updated jaxlib dependency * updated notebooks * 👥 Update @tdmorello as a contributor * some update * fixed typos * Update docs * fixed typos * changed default parameters * updated examples * Add comments for notebook gallery * Update notebooks * compared lagrangian with the origin implementation * notebook update * formatted notebooks with black * added markdowns to notebooks * ifurther updated notebooks * Add contributing guide to docs * Add autoenum to docs * updated notebooks * updated thumbnail data for docs * updated readme * ignored package lock file for yarm * fixed type * added windows instruction * 👥 Update @YuLiu-web as a contributor * fixed notebook folder * fixed typo * added macos and windows tests * test falling back to ubuntu * added jax installation in noxfile for windows * migrated contribution details to the docs * update badges * fixed badges * a bit more update... * fixed readme * fixed badgee stype * further fix... * Timelapse option in `transform` (#80) * udpated transform method * fixed tests * fixed typo * solving problems with tests * fixed transform bug * removed tests for rescaling-fitting * removed resized transform example * added dask test * set baseline type to optinal * updated testdata * deleted notebook outputs * Update timelapse * re-organized transform part * tests running... * made the resize method changable * resize method tested * tested resize method * added comments for the transform function * minor fix * fixed fitting_weight shape bug Co-authored-by: Tim Morello <[email protected]> * Update example data (#92) * updating notebooks * updated rescaled data * updated test data * update notebooks * Fixed most unit tests * re-added reference comparison notebook * re-added test data Co-authored-by: Nicholas-Schaub <[email protected]> * Fix `fit` bug for dask array inputs (#93) * udpated transform method * fixed tests * fixed typo * solving problems with tests * fixed transform bug * removed tests for rescaling-fitting * removed resized transform example * added dask test * set baseline type to optinal * updated testdata * deleted notebook outputs * Update timelapse * re-organized transform part * tests running... * made the resize method changable * resize method tested * tested resize method * added comments for the transform function * minor fix * fixed fitting_weight shape bug * fixed dask-related bug Co-authored-by: Tim Morello <[email protected]> * Update hashes (#95) * Minor change before beta release (#96) * DCT tools update (#98) * updated hashes to newer ones * changed source to main * ci experiment with dev branch * updated branch * added tests for wrong argments * updated classifier to beta * updated copyright * just added spaces * updated downsampled data to Zenodo * renamed data to datasets * typo fix... * deleted depricated tools * fixed package names ... * import fix * 👥 Update @yfukai as a contributor * added my contribution for ci... * removed deprecated routines from old DCT2d * updated tests and fixed jax version to solve problems * Update coef definitions (#103) * updated hashes to newer ones * changed source to main * ci experiment with dev branch * updated branch * added tests for wrong argments * updated classifier to beta * updated copyright * just added spaces * updated downsampled data to Zenodo * renamed data to datasets * typo fix... * deleted depricated tools * fixed package names ... * import fix * 👥 Update @yfukai as a contributor * added my contribution for ci... * removed deprecated routines from old DCT2d * trying to add darkfield weight and changing weight for ladmap * fixed typo * added attribute * updated tests and fixed jax version to solve problems * rewrite ladmap to normalize B * working ver? * updated test data * udpated weight for darkfield * updated test data * removed print * fixed normalization const * added test to show basic does not depend on the intensity scale * small update * added installation doc for M1 chips * updated installation instruction for M1 mac * reverted some merge changes * Update README.md Co-authored-by: Tim Morello <[email protected]> Co-authored-by: Tim Morello <[email protected]> Co-authored-by: Nicholas-Schaub <[email protected]> Co-authored-by: Mahdi Lamb <[email protected]> Co-authored-by: Tingying Peng <[email protected]> Co-authored-by: YuLiu-web <[email protected]>

Add abstract base class for DCT

2f351b5

tdmorello commented Jan 15, 2022

View reviewed changes

tdmorello changed the title ~~Add abstract base class for DCT~~ Added abstract base class for DCT Jan 15, 2022

This was linked to issues Jan 15, 2022

Implement a DCT abstract base class #15

Closed

Implement CV2s version of DCT #16

Closed

tdmorello added 4 commits January 15, 2022 13:18

Update pre-commit

3aa2c03

Add dct backends and tests

9420ce8

Update setup.cfg

c151f4f

Quickfix for failing tests

f0a2c78

tdmorello marked this pull request as ready for review January 15, 2022 19:24

tdmorello requested review from Nicholas-Schaub and yfukai January 15, 2022 19:25

Merge branch 'dev' into dev

b75492b

Nicholas-Schaub requested changes Jan 15, 2022

View reviewed changes

mahdilamb and others added 4 commits January 15, 2022 17:03

Update inexact_alm_rspca_l1.py (peng-lab#1)

f3fcf19

- optimise SVD choice - small other cleanup

Check package installed before making DCT class

149f105

Merge HEAD, branches 'dev' and 'dev' of https://github.com/peng-lab/B…

8267279

…aSiCPy into dev

Remove unused import

1adb2e6

tdmorello requested a review from Nicholas-Schaub January 16, 2022 03:16

tdmorello self-assigned this Jan 16, 2022

Nicholas-Schaub approved these changes Jan 17, 2022

View reviewed changes

Nicholas-Schaub merged commit 3cbeba9 into peng-lab:dev Jan 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added abstract base class for DCT #25

Added abstract base class for DCT #25

tdmorello commented Jan 14, 2022 •

edited

Loading

tdmorello Jan 15, 2022

Nicholas-Schaub Jan 15, 2022

tdmorello Jan 15, 2022

tdmorello Jan 15, 2022

tdmorello Jan 15, 2022

Nicholas-Schaub Jan 15, 2022

tdmorello Jan 15, 2022

Nicholas-Schaub left a comment

Nicholas-Schaub Jan 15, 2022

tdmorello Jan 16, 2022

tdmorello Jan 17, 2022

Nicholas-Schaub Jan 15, 2022 •

edited

Loading

tdmorello Jan 16, 2022

Nicholas-Schaub Jan 15, 2022

tdmorello Jan 16, 2022

		@@ -39,7 +39,11 @@ console_scripts =
		pybasic = pybasic.__main__:main

		DCT_BACKENDS = {"JAX": JaxDCT, "OPENCV": OpenCVDCT, "SCIPY": SciPyDCT}


		dct = DCT_BACKENDS[os.environ["DCT_BACKEND"] or DEFAULT_BACKEND]

		return scipy.fft.idct(scipy.fft.idct(arr.T, norm="ortho").T, norm="ortho")


		DCT_BACKENDS = {"JAX": JaxDCT(), "OPENCV": OpenCVDCT(), "SCIPY": SciPyDCT()}

Added abstract base class for DCT #25

Added abstract base class for DCT #25

Conversation

tdmorello commented Jan 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nicholas-Schaub left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nicholas-Schaub Jan 15, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tdmorello commented Jan 14, 2022 •

edited

Loading

Nicholas-Schaub Jan 15, 2022 •

edited

Loading