Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add function to load raster tile maps using contextily #2125

Merged
merged 35 commits into from
Mar 2, 2023
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
6f7a4f0
Add function to load raster basemap tiles using contextily
weiji14 Sep 19, 2022
917604d
Format to remove triple dots
weiji14 Sep 19, 2022
66f66cc
Use correct Spherical Mercator coordinates
weiji14 Sep 19, 2022
e99ed4d
Fix coordinates for doctest
weiji14 Sep 19, 2022
3c437c0
Change ll parameter to lonlat
weiji14 Sep 19, 2022
c738375
Reduce number of local variables to fix pylint R0914
weiji14 Sep 19, 2022
30fd9a2
Add contextily to CI build matrix and include it as optional dependency
weiji14 Sep 19, 2022
f913222
Set default bounding box coordinates to be lonlat
weiji14 Sep 20, 2022
1270c3f
Merge branch 'main' into contextily/load_map_tiles
weiji14 Oct 12, 2022
0455bb4
Skip doctest when contextily is not installed
weiji14 Oct 12, 2022
8b5317f
Merge branch 'main' into contextily/load_map_tiles
weiji14 Nov 4, 2022
6142b40
Fix typo
weiji14 Nov 4, 2022
5400710
Add intersphinx link for rasterio
weiji14 Nov 4, 2022
8aa1c7a
Merge branch 'main' into contextily/load_map_tiles
weiji14 Dec 6, 2022
bf1c007
Merge branch 'main' into contextily/load_map_tiles
weiji14 Dec 28, 2022
26dd404
Merge branch 'main' into contextily/load_map_tiles
weiji14 Dec 28, 2022
2959e58
Document wait and max_retries parameters used in contextily.bounds2img
weiji14 Dec 28, 2022
241aec1
Merge branch 'main' into contextily/load_map_tiles
weiji14 Jan 7, 2023
eeda973
Reorder deps and add a fullstop
weiji14 Jan 7, 2023
cc86e7d
Merge branch 'main' into contextily/load_map_tiles
weiji14 Jan 24, 2023
67b17fc
Use PyGMT's convention for default values in docstrings
weiji14 Jan 24, 2023
5a446fb
Merge branch 'main' into contextily/load_map_tiles
weiji14 Feb 26, 2023
3dc8846
Rename load_map_tiles to load_tile_map
weiji14 Feb 26, 2023
5c61779
Lint to change coords dict() to {}
weiji14 Feb 26, 2023
d06e2c0
Add zoom parameter and remove kwargs
weiji14 Feb 26, 2023
6b2d5cd
Revert "Reduce number of local variables to fix pylint R0914"
weiji14 Feb 26, 2023
b1ad795
Add contextily to docs build CI requirements
weiji14 Feb 27, 2023
484b6c8
Add contextily to pygmt.show_versions() dependency list
weiji14 Feb 27, 2023
cf7ab34
Apply suggestions from code review
weiji14 Feb 27, 2023
8563ecf
Lint to reduce length of comment line
weiji14 Feb 27, 2023
aa2d5fb
Document the three possible source options thoroughly
weiji14 Mar 2, 2023
d74f878
Remove extra fullstops and fix typos
weiji14 Mar 2, 2023
9f14d31
Add more detail about the zoom level of detail
weiji14 Mar 2, 2023
b02b50c
Remove another fullstop
weiji14 Mar 2, 2023
10af781
Wrap integers in backticks and fix some typos
weiji14 Mar 2, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci_tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ jobs:
optional-packages: ''
- python-version: '3.10'
numpy-version: '1.23'
optional-packages: 'geopandas ipython'
optional-packages: 'contextily geopandas ipython'
timeout-minutes: 30
defaults:
run:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci_tests_dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ jobs:
geopandas ghostscript libnetcdf hdf5 zlib curl pcre make
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Off-topic: All Python packages are installed using pip, except that geopandas is installed using mamba. What's the reason?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at #1290 (comment), I think it was to get compatible GDAL versions between gmt, geopandas and fiona at the time. We could try doing pip install --pre geopandas, but best do it in a separate PR.

pip install --pre --prefer-binary \
numpy pandas xarray netCDF4 packaging \
build dvc ipython 'pytest>=6.0' pytest-cov \
contextily build dvc ipython 'pytest>=6.0' pytest-cov \
weiji14 marked this conversation as resolved.
Show resolved Hide resolved
pytest-doctestplus pytest-mpl sphinx-gallery

# Pull baseline image data from dvc remote (DAGsHub)
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci_tests_legacy.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ jobs:
- name: Install dependencies
run: |
mamba install gmt=${{ matrix.gmt_version }} numpy \
pandas xarray netCDF4 packaging geopandas \
pandas xarray netCDF4 packaging contextily geopandas \
weiji14 marked this conversation as resolved.
Show resolved Hide resolved
build dvc make pytest>=6.0 \
pytest-cov pytest-doctestplus pytest-mpl sphinx-gallery

Expand Down
1 change: 1 addition & 0 deletions doc/api/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,7 @@ and store them in the GMT cache folder.
datasets.load_fractures_compilation
datasets.load_hotspots
datasets.load_japan_quakes
datasets.load_map_tiles
weiji14 marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm unsure about the name. Should it be called load_map_tiles or load_tile_maps?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about the 2nd part of #2115, and had an idea about making a fig.tilemap method instead of complicating fig.basemap further. In that case, load_tile_maps might actually work better?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are both "map tiles" and "tile maps" grammatically correct?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think both should be ok. At least I know there's Web Map Tile Service (WMTS) and Tile Map Service (TMS)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm starting to lean towards load_tile_maps or load_tile_map. Should there be an s or no s?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer load_tile_map without s.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, renamed to load_tile_map in 3dc8846

datasets.load_mars_shape
datasets.load_ocean_ridge_points
datasets.load_sample_bathymetry
Expand Down
4 changes: 3 additions & 1 deletion doc/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,13 @@

# intersphinx configuration
intersphinx_mapping = {
"python": ("https://docs.python.org/3/", None),
"contextily": ("https://contextily.readthedocs.io/en/stable/", None),
"geopandas": ("https://geopandas.org/en/stable/", None),
"numpy": ("https://numpy.org/doc/stable/", None),
"python": ("https://docs.python.org/3/", None),
"pandas": ("https://pandas.pydata.org/pandas-docs/stable/", None),
"xarray": ("https://xarray.pydata.org/en/stable/", None),
"xyzservices": ("https://xyzservices.readthedocs.io/en/stable", None),
}

# options for sphinx-copybutton
Expand Down
1 change: 1 addition & 0 deletions doc/install.rst
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,7 @@ PyGMT requires the following libraries to be installed:
The following are optional dependencies:

* `IPython <https://ipython.org>`__: For embedding the figures in Jupyter notebooks (recommended).
* `Contextily <https://contextily.readthedocs.io>`__: For retrieving tile maps from the internet.
* `GeoPandas <https://geopandas.org>`__: For using and plotting GeoDataFrame objects.

Installing GMT and other dependencies
Expand Down
1 change: 1 addition & 0 deletions environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ dependencies:
- netCDF4
- packaging
# Optional dependencies
- contextily
weiji14 marked this conversation as resolved.
Show resolved Hide resolved
- geopandas
# Development dependencies (general)
- build
Expand Down
1 change: 1 addition & 0 deletions pygmt/datasets/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

from pygmt.datasets.earth_age import load_earth_age
from pygmt.datasets.earth_relief import load_earth_relief
from pygmt.datasets.map_tiles import load_map_tiles
from pygmt.datasets.samples import (
list_sample_data,
load_fractures_compilation,
Expand Down
108 changes: 108 additions & 0 deletions pygmt/datasets/map_tiles.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
"""
Function to load raster basemap tiles from XYZ tile providers, and load as
:class:`xarray.DataArray`.
"""

try:
import contextily
except ImportError:
contextily = None

import numpy as np
import xarray as xr


def load_map_tiles(region, source=None, lonlat=False, **kwargs):
"""
Load a georeferenced raster basemap from XYZ tile providers.

The tiles that compose the map are merged and georeferenced into an
:class:`xarray.DataArray` image with 3 bands (RGB). Note that the returned
image is in a Spherical Mercator (EPSG:3857) coordinate reference system.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we should convert the returned image to longitude/latitude because it's more commonly used than the Spherical Mercator coordinate system, following https://contextily.readthedocs.io/en/latest/warping_guide.html.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most tiles are originally served as Web Mercator (EPSG:3857), and reprojecting to latlon (EPSG:4326) would result in distortion. I'd prefer the reprojection to be a user controlled step (e.g. using rioxarray's .rio.to_crs), because if latlon isn't what the user wants, there would be double reprojection (EPSG:3857 ->EPSG:4326 ->EPSG:????) which is less accurate than single reprojection (EPSG:3857 ->EPSG:????).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer the reprojection to be a user controlled step (e.g. using rioxarray's .rio.to_crs)

What about having an option so that users can decide on the desired projection and don't have to learn the syntax of the rioxarray package?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe in a separate PR?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe in a separate PR?

Sounds good to me.


Parameters
----------
region : list
The bounding box of the map in the form of a list [*xmin*, *xmax*,
*ymin*, *ymax*]. These coordinates should be in Spherical Mercator
(EPSG:3857) if ``lonlat=False``, or longitude/latitude if
``lonlat=True``.
weiji14 marked this conversation as resolved.
Show resolved Hide resolved

source : xyzservices.TileProvider or str
weiji14 marked this conversation as resolved.
Show resolved Hide resolved
[Optional. Default: Stamen Terrain web tiles] The tile source: web tile
provider or path to local file. The web tile provider can be in the
weiji14 marked this conversation as resolved.
Show resolved Hide resolved
form of a :class:`xyzservices.TileProvider` object or a URL. The
placeholders for the XYZ in the URL need to be {x}, {y}, {z},
respectively. For local file paths, the file is read with rasterio and
weiji14 marked this conversation as resolved.
Show resolved Hide resolved
all bands are loaded into the basemap. IMPORTANT: tiles are assumed to
be in the Spherical Mercator projection (EPSG:3857).

lonlat : bool
[Optional. Default: False]. If True, coordinates in ``region`` are
assumed to be lon/lat as opposed to Spherical Mercator.

kwargs : dict
Extra keyword arguments to pass to :func:`contextily.bounds2img`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like there are only a few contextily.bounds2img parameters that are not wrapped (wait and max_retries). Why are you leaving that as a kwarg option instead of adding default values?

Copy link
Member Author

@weiji14 weiji14 Dec 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using kwargs means that if the API of contextily.bounds2img changes for any reason (e.g. new parameters added, or parameters get deleted or renamed), the pygmt.datasets.load_map_tiles function would still work and be cross-compatible with different contextily versions.

That said, the API does seem rather stable looking at https://github.com/geopandas/contextily/blame/v1.2.0/contextily/tile.py#L157-L195, so I could add wait and max_retries to the docstring 🙂 Edit: done in 2959e58

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the kwargs parameter still needed to be included? It looks like all available options have been wrapped.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems the bounds2img API documentation doesn't show all the available parameters. For example, the tutorial (https://contextily.readthedocs.io/en/latest/intro_guide.html#Fine-tune-zoom-levels) also mentions the zoom parameter, which is not listed in the bounds2img API documentatin.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm yeah, zoom isn't in the API docs, but I see it in the code at https://github.com/geopandas/contextily/blob/6d479a6d2b616c3a1c216befd2736dd30eb355b5/contextily/tile.py#L174-L175. Should I add the zoom parameter then, or stick with *kwargs?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems the bounds2img API documentation doesn't show all the available parameters. For example, the tutorial (contextily.readthedocs.io/en/latest/intro_guide.html#Fine-tune-zoom-levels) also mentions the zoom parameter, which is not listed in the bounds2img API documentatin.

Likely the bounds2img API documentation has changed since my last comment. The new documentation now shows the zoom parameter.

My concern with **kwargs is that users may pass bounds2img's parameters w, e, s, n or ll to the load_map_tiles function.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My concern with **kwargs is that users may pass bounds2img's parameters w, e, s, n or ll to the load_map_tiles function.

Fair point. I've added the zoom parameter and removed **kwargs in d06e2c0.


Returns
-------
raster : xarray.DataArray
Georefenced 3D data array of RGB value.

Raises
------
ModuleNotFoundError
If ``contextily`` is not installed. Follow
:doc:`install instructions for contextily <contextily:index>`, (e.g.
via ``pip install contextily``) before using this function.

Examples
--------
>>> import contextily
>>> from pygmt.datasets import load_map_tiles
>>> raster = load_map_tiles(
... region=[103.60, 104.06, 1.22, 1.49], # West, East, South, North
... source=contextily.providers.Stamen.TerrainBackground,
... lonlat=True, # bounding box coordinates are longitude/latitude
... )
>>> raster.sizes
Frozen({'band': 3, 'y': 1024, 'x': 1536})
>>> raster.coords
Coordinates:
* band (band) int64 0 1 2
* y (y) float64 1.663e+05 1.663e+05 1.663e+05 ... 1.272e+05 ...
* x (x) float64 1.153e+07 1.153e+07 1.153e+07 ... 1.158e+07 ...
"""
if contextily is None:
raise ModuleNotFoundError(
"Package `contextily` is required to be installed to use this function. "
"Please use `pip install contextily` or "
"`conda install -c conda-forge contextily` "
"to install the package"
weiji14 marked this conversation as resolved.
Show resolved Hide resolved
)

west, east, south, north = region
image, (left, right, bottom, top) = contextily.bounds2img(
w=west, s=south, e=east, n=north, source=source, ll=lonlat, **kwargs
)

# Turn RGBA image from channel-last (H, W, C) to channel-first (C, H, W)
# and get just RGB (3 band) by dropping RGBA's alpha channel
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why drop the alpha channel? GMT supports GeoTiff files with alpha channel.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the alpha channel is the same throughout (at least for the basemaps I tried before), so not very helpful.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the grdimage_img_c2s_with_intensity function, it can handle 4 bands with transparency.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but the function returns an xarray.DataArray, not a GeoTIFF. The current pygmt.grdimage doesn't actually support plotting 4-band xarray.DataArrays (or even 3-band ones, see #1555) yet. So I don't see how this alpha channel can realistically be used unless people know how to save the xarray.DataArray to a 4-band GeoTIFF with the bands as a proper color interpretation first (requires corteva/rioxarray#414 most likely).

That said, there are only a few XYZ tiles that I know of which have actual transparencies, e.g. the Stamen Toner Labels example at geopandas/contextily#114 (comment). Regular tiles like Stamen Terrain do have an alpha channel, but they don't really carry meaningful transparency information (i.e. they are all 0s or 1s).

rgb_image = image.transpose(2, 0, 1)[0:3, :, :]

# Georeference RGB image into an xarray.DataArray
dataarray = xr.DataArray(
data=rgb_image,
coords=dict(
band=[0, 1, 2], # Red, Green, Blue
y=np.linspace(start=top, stop=bottom, num=rgb_image.shape[1]),
x=np.linspace(start=left, stop=right, num=rgb_image.shape[2]),
),
dims=("band", "y", "x"),
)

# If rioxarray is installed, set the coordinate reference system
if hasattr(dataarray, "rio"):
dataarray = dataarray.rio.write_crs(input_crs="EPSG:3857")

return dataarray
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ dynamic = ["version"]

[project.optional-dependencies]
all = [
"contextily",
"geopandas",
"ipython"
]
Expand Down