Merge remote-tracking branch 'upstream/main' into flox-doc

* upstream/main: RTD maintenance (pydata#7477) fix the RTD build skipping feature (pydata#7476) Add benchmarks for to_dataframe and to_dask_dataframe (pydata#7474) allow skipping RTD builds (pydata#7470) create separate environment files for `python=3.11` (pydata#7469) Bump mamba-org/provision-with-micromamba from 14 to 15 (pydata#7466) install `numbagg` from `conda-forge` (pydata#7415) Fill missing data_vars during concat by reindexing (pydata#7400) [skip-cii] Add pyodide update instructions to HOW_TO_RELEASE (pydata#7449) [skip-ci] whats-new for next release (pydata#7455) v2023.01.0 whats-new (pydata#7440)
dcherian · Jan 25, 2023 · f4ff04c · f4ff04c
2 parents 230eac9 + f128f24
commit f4ff04c
Show file tree

Hide file tree

Showing 17 changed files with 662 additions and 55 deletions.
diff --git a/.github/workflows/benchmarks.yml b/.github/workflows/benchmarks.yml
@@ -22,7 +22,7 @@ jobs:
           fetch-depth: 0
 
       - name: Set up conda environment
-        uses: mamba-org/provision-with-micromamba@v14
+        uses: mamba-org/provision-with-micromamba@v15
         with:
           environment-file: ${{env.CONDA_ENV_FILE}}
           environment-name: xarray-tests

diff --git a/.github/workflows/ci-additional.yaml b/.github/workflows/ci-additional.yaml
@@ -53,7 +53,7 @@ jobs:
           echo "TODAY=$(date  +'%Y-%m-%d')" >> $GITHUB_ENV
 
       - name: Setup micromamba
-        uses: mamba-org/provision-with-micromamba@v14
+        uses: mamba-org/provision-with-micromamba@v15
         with:
           environment-file: ${{env.CONDA_ENV_FILE}}
           environment-name: xarray-tests
@@ -100,7 +100,7 @@ jobs:
         run: |
           echo "TODAY=$(date +'%Y-%m-%d')" >> $GITHUB_ENV
       - name: Setup micromamba
-        uses: mamba-org/provision-with-micromamba@v14
+        uses: mamba-org/provision-with-micromamba@v15
         with:
           environment-file: ${{env.CONDA_ENV_FILE}}
           environment-name: xarray-tests
@@ -156,7 +156,7 @@ jobs:
         run: |
           echo "TODAY=$(date +'%Y-%m-%d')" >> $GITHUB_ENV
       - name: Setup micromamba
-        uses: mamba-org/provision-with-micromamba@v14
+        uses: mamba-org/provision-with-micromamba@v15
         with:
           environment-file: ${{env.CONDA_ENV_FILE}}
           environment-name: xarray-tests
@@ -212,7 +212,7 @@ jobs:
           fetch-depth: 0 # Fetch all history for all branches and tags.
 
       - name: Setup micromamba
-        uses: mamba-org/provision-with-micromamba@v14
+        uses: mamba-org/provision-with-micromamba@v15
         with:
           environment-name: xarray-tests
           environment-file: false

diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
@@ -42,7 +42,7 @@ jobs:
       matrix:
         os: ["ubuntu-latest", "macos-latest", "windows-latest"]
         # Bookend python versions
-        python-version: ["3.8", "3.10"]
+        python-version: ["3.8", "3.10", "3.11"]
         env: [""]
         include:
           # Minimum python version:
@@ -67,7 +67,13 @@ jobs:
         run: |
           echo "TODAY=$(date +'%Y-%m-%d')" >> $GITHUB_ENV
 
-          if [[ ${{ matrix.os }} == windows* ]] ;
+          if [[ "${{matrix.python-version}}" == "3.11" ]]; then
+            if [[ ${{matrix.os}} == windows* ]]; then
+              echo "CONDA_ENV_FILE=ci/requirements/environment-windows-py311.yml" >> $GITHUB_ENV
+            else
+              echo "CONDA_ENV_FILE=ci/requirements/environment-py311.yml" >> $GITHUB_ENV
+            fi
+          elif [[ ${{ matrix.os }} == windows* ]] ;
           then
             echo "CONDA_ENV_FILE=ci/requirements/environment-windows.yml" >> $GITHUB_ENV
           elif [[ "${{ matrix.env }}" != "" ]] ;
@@ -86,7 +92,7 @@ jobs:
           echo "PYTHON_VERSION=${{ matrix.python-version }}" >> $GITHUB_ENV
 
       - name: Setup micromamba
-        uses: mamba-org/provision-with-micromamba@v14
+        uses: mamba-org/provision-with-micromamba@v15
         with:
           environment-file: ${{ env.CONDA_ENV_FILE }}
           environment-name: xarray-tests

diff --git a/.github/workflows/upstream-dev-ci.yaml b/.github/workflows/upstream-dev-ci.yaml
@@ -54,7 +54,7 @@ jobs:
         with:
           fetch-depth: 0 # Fetch all history for all branches and tags.
       - name: Set up conda environment
-        uses: mamba-org/provision-with-micromamba@v14
+        uses: mamba-org/provision-with-micromamba@v15
         with:
           environment-file: ci/requirements/environment.yml
           environment-name: xarray-tests

diff --git a/.readthedocs.yaml b/.readthedocs.yaml
@@ -1,9 +1,14 @@
 version: 2
 
 build:
-  os: ubuntu-20.04
+  os: ubuntu-22.04
   tools:
     python: mambaforge-4.10
+  jobs:
+    post_checkout:
+      - (git --no-pager log --pretty="tformat:%s" -1 | grep -vqF "[skip-rtd]") || exit 183
+    pre_install:
+      - git update-index --assume-unchanged doc/conf.py ci/requirements/doc.yml
 
 conda:
   environment: ci/requirements/doc.yml

diff --git a/HOW_TO_RELEASE.md b/HOW_TO_RELEASE.md
@@ -92,7 +92,15 @@ upstream        https://github.com/pydata/xarray (push)
       ```
     You're done pushing to main!
 
-13. Issue the release announcement to mailing lists & Twitter. For bug fix releases, I
+13. Update the version available on pyodide:
+    - Open the PyPI page for [Xarray downloads](https://pypi.org/project/xarray/#files)
+    - Clone the [pyodide repository](https://github.com/pyodide/pyodide).
+    - Edit `packages/xarray/meta.yaml` to update the
+        - link to the wheel (under "Built Distribution" on the PyPI page)
+        - SHA256 hash (Click "Show Hashes" next to the link to the wheel)
+    - Open a pull request to pyodide
+
+14. Issue the release announcement to mailing lists & Twitter. For bug fix releases, I
     usually only email [email protected]. For major/feature releases, I will email a broader
     list (no more than once every 3-6 months):
       - [email protected]

diff --git a/asv_bench/benchmarks/pandas.py b/asv_bench/benchmarks/pandas.py
@@ -3,7 +3,7 @@
 
 import xarray as xr
 
-from . import parameterized
+from . import parameterized, requires_dask
 
 
 class MultiIndexSeries:
@@ -24,3 +24,38 @@ def setup(self, dtype, subset):
     @parameterized(["dtype", "subset"], ([int, float], [True, False]))
     def time_from_series(self, dtype, subset):
         xr.DataArray.from_series(self.series)
+
+
+class ToDataFrame:
+    def setup(self, *args, **kwargs):
+        xp = kwargs.get("xp", np)
+        random_kws = kwargs.get("random_kws", {})
+        method = kwargs.get("method", "to_dataframe")
+
+        dim1 = 10_000
+        dim2 = 10_000
+        ds = xr.Dataset(
+            {
+                "x": xr.DataArray(
+                    data=xp.random.random((dim1, dim2), **random_kws),
+                    dims=["dim1", "dim2"],
+                    coords={"dim1": np.arange(0, dim1), "dim2": np.arange(0, dim2)},
+                )
+            }
+        )
+        self.to_frame = getattr(ds, method)
+
+    def time_to_dataframe(self):
+        self.to_frame()
+
+    def peakmem_to_dataframe(self):
+        self.to_frame()
+
+
+class ToDataFrameDask(ToDataFrame):
+    def setup(self, *args, **kwargs):
+        requires_dask()
+
+        import dask.array as da
+
+        super().setup(xp=da, random_kws=dict(chunks=5000), method="to_dask_dataframe")
diff --git a/ci/requirements/all-but-dask.yml b/ci/requirements/all-but-dask.yml
@@ -23,7 +23,8 @@ dependencies:
   - nc-time-axis
   - netcdf4
   - numba
-  - numpy
+  - numbagg
+  - numpy<1.24
   - packaging
   - pandas
   - pint
@@ -41,5 +42,3 @@ dependencies:
   - toolz
   - typing_extensions
   - zarr
-  - pip:
-      - numbagg
diff --git a/ci/requirements/doc.yml b/ci/requirements/doc.yml
@@ -18,7 +18,7 @@ dependencies:
   - nbsphinx
   - netcdf4>=1.5
   - numba
-  - numpy>=1.20
+  - numpy>=1.20,<1.24
   - packaging>=21.0
   - pandas>=1.3
   - pooch

diff --git a/ci/requirements/environment-py311.yml b/ci/requirements/environment-py311.yml
@@ -0,0 +1,48 @@
+name: xarray-tests
+channels:
+  - conda-forge
+  - nodefaults
+dependencies:
+  - aiobotocore
+  - boto3
+  - bottleneck
+  - cartopy
+  # - cdms2
+  - cfgrib
+  - cftime
+  - dask-core
+  - distributed
+  - flox
+  - fsspec!=2021.7.0
+  - h5netcdf
+  - h5py
+  - hdf5
+  - hypothesis
+  - iris
+  - lxml  # Optional dep of pydap
+  - matplotlib-base
+  - nc-time-axis
+  - netcdf4
+  # - numba
+  # - numbagg
+  - numexpr
+  - numpy
+  - packaging
+  - pandas
+  - pint
+  - pip
+  - pooch
+  - pre-commit
+  - pseudonetcdf
+  - pydap
+  - pytest
+  - pytest-cov
+  - pytest-env
+  - pytest-xdist
+  - rasterio
+  - scipy
+  - seaborn
+  # - sparse
+  - toolz
+  - typing_extensions
+  - zarr
diff --git a/ci/requirements/environment-windows-py311.yml b/ci/requirements/environment-windows-py311.yml
@@ -0,0 +1,44 @@
+name: xarray-tests
+channels:
+  - conda-forge
+dependencies:
+  - boto3
+  - bottleneck
+  - cartopy
+  # - cdms2  # Not available on Windows
+  # - cfgrib  # Causes Python interpreter crash on Windows: https://github.com/pydata/xarray/pull/3340
+  - cftime
+  - dask-core
+  - distributed
+  - flox
+  - fsspec!=2021.7.0
+  - h5netcdf
+  - h5py
+  - hdf5
+  - hypothesis
+  - iris
+  - lxml  # Optional dep of pydap
+  - matplotlib-base
+  - nc-time-axis
+  - netcdf4
+  # - numba
+  # - numbagg
+  - numpy
+  - packaging
+  - pandas
+  - pint
+  - pip
+  - pre-commit
+  - pseudonetcdf
+  - pydap
+  - pytest
+  - pytest-cov
+  - pytest-env
+  - pytest-xdist
+  - rasterio
+  - scipy
+  - seaborn
+  # - sparse
+  - toolz
+  - typing_extensions
+  - zarr
diff --git a/ci/requirements/environment-windows.yml b/ci/requirements/environment-windows.yml
@@ -22,7 +22,8 @@ dependencies:
   - nc-time-axis
   - netcdf4
   - numba
-  - numpy
+  - numbagg
+  - numpy<1.24
   - packaging
   - pandas
   - pint
@@ -41,5 +42,3 @@ dependencies:
   - toolz
   - typing_extensions
   - zarr
-  - pip:
-      - numbagg
diff --git a/ci/requirements/environment.yml b/ci/requirements/environment.yml
@@ -24,8 +24,9 @@ dependencies:
   - nc-time-axis
   - netcdf4
   - numba
+  - numbagg
   - numexpr
-  - numpy
+  - numpy<1.24
   - packaging
   - pandas
   - pint
@@ -45,5 +46,3 @@ dependencies:
   - toolz
   - typing_extensions
   - zarr
-  - pip:
-      - numbagg
diff --git a/doc/whats-new.rst b/doc/whats-new.rst
@@ -15,9 +15,9 @@ What's New
     np.random.seed(123456)
 
 
-.. _whats-new.2022.12.1:
+.. _whats-new.2023.01.1:
 
-v2022.12.1 (unreleased)
+v2023.01.1 (unreleased)
 -----------------------
 
 New Features
@@ -27,14 +27,42 @@ New Features
 Breaking changes
 ~~~~~~~~~~~~~~~~
 
-- :py:meth:`CFTimeIndex.get_loc` has removed the ``method`` and ``tolerance`` keyword arguments.
-  Use ``.get_indexer([key], method=..., tolerance=...)`` instead (:pull:`7361`).
-  By `Matthew Roeschke <https://github.com/mroeschke>`_.
 
 Deprecations
 ~~~~~~~~~~~~
 
 
+Bug fixes
+~~~~~~~~~
+
+- :py:func:`xarray.concat` can now concatenate variables present in some datasets but
+  not others (:issue:`508`, :pull:`7400`).
+  By `Kai Mühlbauer <https://github.com/kmuehlbauer>`_ and `Scott Chamberlin <https://github.com/scottcha>`_.
+
+Documentation
+~~~~~~~~~~~~~
+
+
+Internal Changes
+~~~~~~~~~~~~~~~~
+
+
+.. _whats-new.2023.01.0:
+
+v2023.01.0 (Jan 17, 2023)
+-------------------------
+
+This release includes a number of bug fixes. Thanks to the 14 contributors to this release:
+Aron Gergely, Benoit Bovy, Deepak Cherian, Ian Carroll, Illviljan, Joe Hamman, Justus Magin, Mark Harfouche,
+Matthew Roeschke, Paige Martin, Pierre, Sam Levang, Tom White,  stefank0.
+
+Breaking changes
+~~~~~~~~~~~~~~~~
+
+- :py:meth:`CFTimeIndex.get_loc` has removed the ``method`` and ``tolerance`` keyword arguments.
+  Use ``.get_indexer([key], method=..., tolerance=...)`` instead (:pull:`7361`).
+  By `Matthew Roeschke <https://github.com/mroeschke>`_.
+
 Bug fixes
 ~~~~~~~~~
 
@@ -51,10 +79,6 @@ Bug fixes
 - Preserve original dtype on accessing MultiIndex levels (:issue:`7250`,
   :pull:`7393`). By `Ian Carroll <https://github.com/itcarroll>`_.
 
-Documentation
-~~~~~~~~~~~~~
-
-
 Internal Changes
 ~~~~~~~~~~~~~~~~
 - Add the pre-commit hook `absolufy-imports` to convert relative xarray imports to

diff --git a/setup.cfg b/setup.cfg
@@ -67,6 +67,7 @@ classifiers =
     Programming Language :: Python :: 3.8
     Programming Language :: Python :: 3.9
     Programming Language :: Python :: 3.10
+    Programming Language :: Python :: 3.11
     Topic :: Scientific/Engineering
 
 [options]