Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove galsim hub and tensorflow temporarily #251

Merged
merged 3 commits into from
Jan 14, 2022
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ jobs:
- name: Install poetry dependencies
run: |
poetry run pip install Cython
poetry install --extras "galsim-hub scarlet"
poetry install --extras "scarlet"
poetry run pip install git+https://github.com/pmelchior/scarlet

- name: Run Tests
Expand All @@ -56,7 +56,7 @@ jobs:

- name: Run notebooks
run: |
poetry run pytest --nbval notebooks --sanitize-with tests/sanitize.cfg --ignore "notebooks/01c-galsim_hub_tutorial.ipynb" --ignore "notebooks/01b-scarlet-measure.ipynb"
poetry run pytest --nbval notebooks --sanitize-with tests/sanitize.cfg --ignore "notebooks/01b-scarlet-measure.ipynb"

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v1
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ poetry shell
pre-commit install
```

If you have any problems with the installation, they are probably due to `galsim`. It is recommended that you follow the instructions for [installing galsim](https://galsim-developers.github.io/GalSim/_build/html/install.html) first (inside the `btk37` conda environment), and then try again. Another potential issue might be `tensorflow` and/or `galsim_hub`, if you suspect that might be the problem please create an issue in the BTK github. Note that these two dependencies only work on `python 3.7`.
If you have any problems with the installation, they are probably due to `galsim`. It is recommended that you follow the instructions for [installing galsim](https://galsim-developers.github.io/GalSim/_build/html/install.html) first (inside the `btk37` conda environment), and then try again.

## Pull Requests

Expand Down
111 changes: 3 additions & 108 deletions btk/draw_blends.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
import galsim
import numpy as np
from astropy.table import Column
from astropy.table import Table

from btk import DEFAULT_SEED
from btk.create_blend_generator import BlendGenerator
Expand Down Expand Up @@ -330,10 +329,10 @@ def render_mini_batch(self, blend_list, psf, wcs, survey, seedseq_minibatch, ext
survey (dict): Dictionary containing survey information.
seedseq_minibatch (numpy.random.SeedSequence): Numpy object for generating
random seeds (for the noise generation).
extra_data: This field can be used if some data need to be generated
extra_data: This field can be used if some data needs to be generated
before getting to the step where single galaxies are drawn. It should
have a "shape" of (batch_size,n_blend,...) where n_blend is the number
of objects in a blend. See GalsimHubGenerator for an example of usage.
have a "shape" of (batch_size, n_blend,...) where n_blend is the number
of objects in a blend.

Returns:
`numpy.ndarray` of blend images and isolated galaxy images, along with
Expand Down Expand Up @@ -542,107 +541,3 @@ def render_single(self, entry, filt, psf, survey, extra_data):
gal_conv = gal_conv.shift(entry["ra"], entry["dec"])

return gal_conv.drawImage(nx=pix_stamp_size, ny=pix_stamp_size, scale=survey.pixel_scale)


class GalsimHubGenerator(DrawBlendsGenerator):
"""Implementation of DrawBlendsGenerator for drawing galaxies simulated with galsim_hub.

Galsim Hub (https://github.com/McWilliamsCenter/galsim_hub) is a framework
for generating real-looking galaxies using deep learning models.
"""

compatible_catalogs = ("CosmosCatalog",)

def check_compatibility(self, survey):
"""Checks the compatibility between the catalog and a given survey.

Args:
survey (btk.survey.Survey): Survey to check
"""
if type(self.catalog).__name__ not in self.compatible_catalogs:
raise ValueError(
f"The catalog provided is of the wrong type. The types of "
f"catalogs available for the {type(self).__name__} are {self.compatible_catalogs}"
)

def __init__(
self,
catalog,
sampling_function,
surveys: list,
batch_size=8,
stamp_size=24,
cpus=1,
verbose=False,
add_noise=True,
shifts=None,
indexes=None,
channels_last=False,
galsim_hub_model="hub:Lanusse2020",
param_names=["flux_radius", "mag_auto", "zphot"],
save_path=None,
seed=DEFAULT_SEED,
): # noqa: D417
"""Initializes the GalsimHubGenerator class.

Args:
galsim_hub_model (str): Source of the model to use. Can be
either a distant model or a local one, see the
galsim_hub repo for more information.
param_names (list): list of the parameters with which
the generation is parametrized; this is unique to
each model.
"""
super().__init__(
catalog,
sampling_function,
surveys,
batch_size=batch_size,
stamp_size=stamp_size,
cpus=cpus,
verbose=verbose,
add_noise=add_noise,
shifts=shifts,
indexes=indexes,
channels_last=channels_last,
save_path=save_path,
seed=seed,
)
import galsim_hub

self.galsim_hub_model = galsim_hub.GenerativeGalaxyModel(galsim_hub_model)
self.param_names = param_names

def render_mini_batch(self, blend_list, psf, wcs, survey, seed):
"""Returns isolated and blended images for blend catalogs in blend_list.

Here we generate the images for all galaxies in the batch at the same
time, since galsim_hub is optimized for batch generation.
"""
galsim_hub_params = Table()
for p in self.param_names:
column = Column(np.concatenate([blend[p] for blend in blend_list]), p)
galsim_hub_params.add_column(column)

base_images = self.galsim_hub_model.sample(galsim_hub_params)
base_images_l = []
index = 0
for blend in blend_list:
base_images_l.append(base_images[index : index + len(blend)])
index += len(blend)

return super().render_mini_batch(blend_list, psf, wcs, survey, seed, base_images_l)

def render_single(self, entry, filt, psf, survey, extra_data):
"""Returns the Galsim Image of an isolated galaxy."""
base_image = extra_data
base_image = galsim.Convolve(base_image, psf)
gal_flux = get_flux(entry["ref_mag"], filt, survey)
base_image = base_image.withFlux(gal_flux)
base_image = base_image.shift(entry["ra"], entry["dec"])

pix_stamp_size = int(self.stamp_size / survey.pixel_scale)
galaxy_image = base_image.drawImage(
nx=pix_stamp_size, ny=pix_stamp_size, scale=survey.pixel_scale, dtype=np.float64
)
return galaxy_image
64 changes: 0 additions & 64 deletions btk/sampling_functions.py
Original file line number Diff line number Diff line change
Expand Up @@ -129,70 +129,6 @@ def __call__(self, table, shifts=None, indexes=None):
return blend_table


class DefaultSamplingGalsimHub(SamplingFunction):
"""Default sampling function used for producing blend tables, esp. for galsim_hub."""

def __init__(self, max_number=2, stamp_size=24.0, maxshift=None, seed=DEFAULT_SEED):
"""Initialize default sampling function for galsim_hub.

Args:
max_number (int): Defined in parent class
stamp_size (float): Size of the desired stamp.
maxshift (float): Magnitude of maximum value of shift. If None then it
is set as one-tenth the stamp size. (in arcseconds)
seed (int): Seed to initialize randomness for reproducibility.
"""
super().__init__(max_number, seed)
self.stamp_size = stamp_size
self.maxshift = maxshift if maxshift else self.stamp_size / 10.0

@property
def compatible_catalogs(self):
"""Defined in parent class."""
return "CosmosCatalog"

def __call__(self, table, shifts=None, indexes=None):
"""Method returning a blend sample from table.

This sampling function is almost identical to the default one, except that an
additionnal cut on the flux radius is applied. This is done to avoid having
galaxies too large being selected, as generating large galaxies causes artifacts
in galsim_hub images.

Args:
table (Astropy.table): Table containing entries corresponding to galaxies
from which to sample.
shifts (list): Contains arbitrary shifts to be applied instead of random ones.
Should of the form [dx,dy] where dx and dy are the lists
containing the x and y shifts.
indexes (list): Contains the indexes of the galaxies to use.

Returns:
Astropy.table with entries corresponding to one blend.
"""
number_of_objects = self.rng.integers(1, self.max_number + 1)
(q,) = np.where((table["ref_mag"] <= 25.3) & (table["flux_radius"] <= 24.0))

if indexes is None:
blend_table = table[self.rng.choice(q, size=number_of_objects)]
else:
blend_table = table[indexes]
blend_table["ra"] = 0.0
blend_table["dec"] = 0.0
if shifts is None:
dx, dy = _get_random_center_shift(number_of_objects, self.maxshift, self.rng)
else:
dx, dy = shifts
blend_table["ra"] += dx
blend_table["dec"] += dy

if np.any(blend_table["ra"] > self.stamp_size / 2.0) or np.any(
blend_table["dec"] > self.stamp_size / 2.0
):
warnings.warn("Object center lies outside the stamp")
return blend_table


class BasicSampling(SamplingFunction):
"""Example of basic sampling function features.

Expand Down
8 changes: 0 additions & 8 deletions conf/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,6 @@ max_shift: Null
add_noise: True
channels_last: False

# galsim hub
galsim_hub:
model: "hub:Lanusse2020"
param_names:
- "flux_radius"
- "mag_auto"
- "zphot"

# measure
measure_kwargs: Null
measure_functions:
Expand Down
16 changes: 0 additions & 16 deletions conf/draw_blends/galsim_hub.yaml

This file was deleted.

11 changes: 2 additions & 9 deletions conf/hydra/help/btk_help.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ template: |-
Here are all the available options for the CLI.

* sampling: Specify the sampling function to be used, options:
{default, galsim_hub, group_sampling, group_sampling_numbered}
{default, group_sampling, group_sampling_numbered}

* catalog: Attribute group consisting of two sub-attributes.
* catalog: Name of the BTK catalog class, options: {catsim, cosmos}
Expand All @@ -61,7 +61,7 @@ template: |-
add your own config file to conf/surveys. See the CLI section of the tutorial for
more details.

* draw_blends: Which draw_blend_generator to use, options are {catsim, cosmos, galsim_hub}
* draw_blends: Which draw_blend_generator to use, options are {catsim, cosmos}

* save_path: Absolute path to a (preferably) empty directory where you would like to save
results of running BTK.
Expand All @@ -80,13 +80,6 @@ template: |-

* channels_last: Whether to use have images in channel last format (True) or not (False).

* galsim_hub: Attribute group consisting of options:
* model: Which galsim_hub model to use (default: 'hub:Lanusse2020')
* param_names: list of the parameters with which the generation is parametrized; this
is unique to each model (default: `['flux_radius', 'mag_auto', 'zphot']`).

NOTE: This flag has no effect unless `draw_blends=galsim_hub`

* measure_kwargs: Dictionary or list of dictionaries containing the keyword arguments to be
passed in to each measure_function.

Expand Down
5 changes: 0 additions & 5 deletions conf/sampling/galsim_hub.yaml

This file was deleted.

5 changes: 0 additions & 5 deletions docs/source/catalog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,6 @@ The Cosmic Evolution Survey (COSMOS) is an astronomical survey based on multiple

BTK includes an implementation of COSMOS real galaxy rendering, providing the possibility to get more realistic blends.

Galsim_Hub
-----------
Galsim_Hub is a framework for deep learning image generation models, developped by François Lanusse, which is available `in this page <https://github.com/McWilliamsCenter/galsim_hub>`_ .

BTK supports drawing galaxies generated using any galsim_hub compatible model; the default one generates COSMOS-like galaxies and is described in `this paper <https://arxiv.org/abs/2008.03833>`_, but the BTK implementation should be compatible with any model.

.. Cosmo DC2
.. ---------
Expand Down
10 changes: 0 additions & 10 deletions docs/source/cli.rst
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,6 @@ Here is a list of all the options of BTK you can customize directly from the CLI

- default

- galsim_hub

- group_sampling

- group_sampling_numbered
Expand Down Expand Up @@ -81,8 +79,6 @@ Here is a list of all the options of BTK you can customize directly from the CLI

- cosmos

- galsim_hub

* ``save_path``: Absolute path to a (preferably) empty directory where you would like to save results of running BTK.

* ``cpus``: Number of cpus you would like to use for multiprocessing.
Expand All @@ -99,12 +95,6 @@ Here is a list of all the options of BTK you can customize directly from the CLI

* ``channels_last``: Whether to use have images in channel last format (True) or not (False).

* ``galsim_hub``: Attribute group consisting of options:

- ``model``: Which galsim_hub model to use (default: 'hub:Lanusse2020')

- ``param_names``: list of the parameters with which the generation is parametrized; this is unique to each model (default: ``['flux_radious, 'mag_auto', 'zphot']``). **NOTE**: This flag is ignored unless ``draw_blends=galsim_hub``

* ``measure_kwargs``: Dictionary or list of dictionaries containing the keyword arguments to be passed in to each measure_function.

* ``measure_functions``: List of measure_functions to be ran, options:
Expand Down
6 changes: 1 addition & 5 deletions docs/source/install.rst
Original file line number Diff line number Diff line change
Expand Up @@ -50,14 +50,10 @@ This should install all other missing dependencies if necessary. You can then im

Optional packages
------------------------------
BTK comes with two optional set of dependencies that can be installed: ``galsim_hub`` and ``scarlet``. You can install them alongside ``btk`` as follows:
BTK comes with an optional set of dependencies that can be installed for ``scarlet``. You can install them alongside ``btk`` as follows:

.. code-block::

# install galsim_hub and its dependencies.
# NOTE: it only works on python3.7
pip install blending_toolkit[galsim_hub]

# install latest version of scarlet and its dependencies.
pip install blending_toolkit[scarlet]
pip install git+https://github.com/pmelchior/scarlet
20 changes: 0 additions & 20 deletions docs/source/tutorials.rst
Original file line number Diff line number Diff line change
Expand Up @@ -535,26 +535,6 @@ However, BTK still retrieves the ``flux_radius`` information from this catalog,

4. (optional) One of the two catalogs can contain multiband magnitudes using the format just described.


Galsim_Hub tutorial
--------------------

BTK supports galaxy image generation with ``galsim_hub``; please refer to `this page <https://github.com/McWilliamsCenter/galsim_hub>`_ for more details on ``galsim_hub``. Please note that ``galsim_hub`` only works with ``python 3.7``.

The steps for using the galsim_hub generation are very similar to those from the previous section. Before starting this tutorial, you must install ``galsim_hub``, which can be done using pip.

::

pip install galsim_hub

Alternatively, you can optionally install ``galsim_hub`` along with BTK:

::

pip install btk[galsim-hub]

You can find a notebook version of this tutorial in this `link <https://github.com/LSSTDESC/BlendingToolKit/blob/main/notebooks/01c-galsim_hub_tutorial.ipynb>`_.

SCARLET implementation
-----------------------

Expand Down
Loading