Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use shape and dtype as typevars in NamedArray #8294

Merged
merged 173 commits into from
Oct 18, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
173 commits
Select commit Hold shift + click to select a range
5e706c4
Add from_array function
Illviljan Oct 7, 2023
adab7c9
Update core.py
Illviljan Oct 7, 2023
790336c
some fixes
Illviljan Oct 7, 2023
46482fa
Update test_namedarray.py
Illviljan Oct 7, 2023
8823dab
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 7, 2023
534a040
fixes
Illviljan Oct 7, 2023
f64c914
Merge branch 'namedarray_from_array' of https://github.com/Illviljan/…
Illviljan Oct 7, 2023
166b647
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 7, 2023
2f6af70
fixes
Illviljan Oct 7, 2023
9dc72a1
fixes
Illviljan Oct 7, 2023
db96f17
fixes
Illviljan Oct 7, 2023
446e2a5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 7, 2023
68a9d9c
Update utils.py
Illviljan Oct 7, 2023
40fac21
Merge branch 'namedarray_from_array' of https://github.com/Illviljan/…
Illviljan Oct 7, 2023
fd7bb9f
more
Illviljan Oct 7, 2023
4322eaf
Update core.py
Illviljan Oct 7, 2023
6d2f0ba
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 7, 2023
91f06c0
Update test_namedarray.py
Illviljan Oct 7, 2023
13d232c
Merge branch 'namedarray_from_array' of https://github.com/Illviljan/…
Illviljan Oct 7, 2023
60481e0
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 7, 2023
df13e1c
Update core.py
Illviljan Oct 7, 2023
5ce29c8
Merge branch 'namedarray_from_array' of https://github.com/Illviljan/…
Illviljan Oct 7, 2023
955f3c4
fixes
Illviljan Oct 7, 2023
ebdd69e
fkxes
Illviljan Oct 7, 2023
1232c6a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 7, 2023
1302979
more
Illviljan Oct 7, 2023
c6613a4
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 7, 2023
d199c21
Update test_namedarray.py
Illviljan Oct 7, 2023
be9cbf0
Update test_namedarray.py
Illviljan Oct 7, 2023
051c065
Update test_namedarray.py
Illviljan Oct 7, 2023
8efa3f3
Update test_namedarray.py
Illviljan Oct 7, 2023
f83a994
Update test_namedarray.py
Illviljan Oct 7, 2023
3063622
Update test_namedarray.py
Illviljan Oct 7, 2023
7f4c6f2
Update test_namedarray.py
Illviljan Oct 7, 2023
ede7e1b
move to NDArray instead
Illviljan Oct 8, 2023
3f5dcd1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 8, 2023
0c53676
more
Illviljan Oct 8, 2023
46de2d2
Merge branch 'namedarray_from_array' of https://github.com/Illviljan/…
Illviljan Oct 8, 2023
6fea3b4
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 8, 2023
80be18e
Rename and align more with numpy typing
Illviljan Oct 8, 2023
b8556e1
Add duck type testing
Illviljan Oct 8, 2023
bfb0f73
Merge branch 'namedarray_from_array' of https://github.com/Illviljan/…
Illviljan Oct 8, 2023
78da056
Merge branch 'main' into namedarray_from_array
Illviljan Oct 8, 2023
9b41e0e
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 8, 2023
ac11cf9
docstring
Illviljan Oct 8, 2023
257aca2
Update test_namedarray.py
Illviljan Oct 8, 2023
b2a8b45
fixes
Illviljan Oct 8, 2023
3126ef6
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 8, 2023
ecade8b
more
Illviljan Oct 8, 2023
8fbf0d6
Merge branch 'namedarray_from_array' of https://github.com/Illviljan/…
Illviljan Oct 8, 2023
3f5b440
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 8, 2023
01cf12f
Update utils.py
Illviljan Oct 8, 2023
0b2375e
fixes
Illviljan Oct 8, 2023
489d0a8
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 8, 2023
48d1b1d
more
Illviljan Oct 8, 2023
9974bc6
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 8, 2023
8e5d582
more
Illviljan Oct 8, 2023
c9ac96a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 8, 2023
dab40d2
Update core.py
Illviljan Oct 8, 2023
b6a9691
fixes
Illviljan Oct 8, 2023
08c8665
final
Illviljan Oct 10, 2023
704cc46
Follow numpy's example more with typing
Illviljan Oct 11, 2023
ab2a779
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 11, 2023
96ee335
Merge branch 'main' into namedarray_scalartype
Illviljan Oct 11, 2023
be193e6
Merge branch 'main' into namedarray_scalartype
Illviljan Oct 12, 2023
18b953d
Update utils.py
Illviljan Oct 13, 2023
f9d95a3
Update utils.py
Illviljan Oct 13, 2023
33a67c6
Merge branch 'main' into namedarray_scalartype
Illviljan Oct 13, 2023
a0bf4b1
Update utils.py
Illviljan Oct 13, 2023
16ce413
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 13, 2023
65c213e
Create _array_api.py
Illviljan Oct 13, 2023
0375cc9
Create _typing.py
Illviljan Oct 13, 2023
236b876
Update core.py
Illviljan Oct 13, 2023
2069e4f
Update utils.py
Illviljan Oct 13, 2023
0cad0be
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 13, 2023
d6eaa5d
Update _typing.py
Illviljan Oct 13, 2023
c7a7830
Update core.py
Illviljan Oct 13, 2023
df2c440
Update utils.py
Illviljan Oct 13, 2023
ddb035d
Merge branch 'namedarray_scalartype' of https://github.com/Illviljan/…
Illviljan Oct 13, 2023
5d9c2b0
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 13, 2023
53576d3
Will this make pre-commit happy?
Illviljan Oct 13, 2023
d443f26
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 13, 2023
1bbd8f1
Update _array_api.py
Illviljan Oct 13, 2023
bb0fecf
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 13, 2023
042a311
fixes
Illviljan Oct 13, 2023
014d69a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 13, 2023
842d828
more
Illviljan Oct 13, 2023
5b96617
Update core.py
Illviljan Oct 13, 2023
598a871
fixes
Illviljan Oct 13, 2023
ab26e87
Update test_namedarray.py
Illviljan Oct 13, 2023
79deaba
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 13, 2023
3d24448
fixes
Illviljan Oct 13, 2023
d126abc
Use Self becuase Variable subclasses
Illviljan Oct 14, 2023
8fe9828
fixes
Illviljan Oct 14, 2023
edc1dbb
Update test_namedarray.py
Illviljan Oct 14, 2023
aecdff4
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 14, 2023
1f52099
fixes
Illviljan Oct 14, 2023
117b850
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 14, 2023
f5bc328
Update core.py
Illviljan Oct 14, 2023
a938cb5
Update core.py
Illviljan Oct 14, 2023
b3d4d63
Update core.py
Illviljan Oct 14, 2023
bbedb3f
Merge branch 'main' into namedarray_scalartype
Illviljan Oct 14, 2023
d9543f4
Update variable.py
Illviljan Oct 14, 2023
587c9e7
Update variable.py
Illviljan Oct 14, 2023
28761c7
fix array api, add docstrings
Illviljan Oct 15, 2023
109c715
Fix typing so that a different array gets correct typing
Illviljan Oct 15, 2023
77b39f8
add _new with correct typing in variable
Illviljan Oct 15, 2023
1ffe6ef
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 15, 2023
6634265
Update core.py
Illviljan Oct 15, 2023
9e9590d
shape usually stays the same when copying
Illviljan Oct 15, 2023
8754d6b
Update variable.py
Illviljan Oct 15, 2023
6eb9a7a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 15, 2023
b15484d
Update test_namedarray.py
Illviljan Oct 15, 2023
21a30fb
Merge branch 'namedarray_scalartype' of https://github.com/Illviljan/…
Illviljan Oct 15, 2023
7dd2c12
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 15, 2023
b45a934
Update test_namedarray.py
Illviljan Oct 15, 2023
a258c3a
same shape when astyping
Illviljan Oct 15, 2023
526e10f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 15, 2023
fea1a64
Delete test_namedarray_sketching.py
Illviljan Oct 15, 2023
10da501
Merge branch 'namedarray_scalartype' of https://github.com/Illviljan/…
Illviljan Oct 15, 2023
154385b
typos
Illviljan Oct 15, 2023
cbcaddb
remove any typing for now
Illviljan Oct 15, 2023
88534fd
fixes
Illviljan Oct 15, 2023
c611e8e
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 15, 2023
fb1db9c
fixes
Illviljan Oct 15, 2023
011cdcd
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 15, 2023
54d33f9
Update indexing.py
Illviljan Oct 15, 2023
489ddd3
Merge branch 'namedarray_scalartype' of https://github.com/Illviljan/…
Illviljan Oct 15, 2023
715fc49
add namespace to some explicitindexing stuff
Illviljan Oct 15, 2023
97c72ff
Update variable.py
Illviljan Oct 15, 2023
9e349e1
Update duck_array_ops.py
Illviljan Oct 15, 2023
88fd3f9
Update duck_array_ops.py
Illviljan Oct 15, 2023
de969f0
fixes
Illviljan Oct 15, 2023
2629ce0
Update variable.py
Illviljan Oct 15, 2023
5b27949
Fixes
Illviljan Oct 15, 2023
41d0b0f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 15, 2023
6572abe
Update test_variable.py
Illviljan Oct 15, 2023
6cf0d4b
Revert "Update test_variable.py"
Illviljan Oct 15, 2023
201caf3
Update _array_api.py
Illviljan Oct 15, 2023
a88b5b7
Update _array_api.py
Illviljan Oct 16, 2023
c1b7bda
Merge branch 'main' into namedarray_scalartype
Illviljan Oct 16, 2023
4d3f5b4
Update _array_api.py
Illviljan Oct 16, 2023
61d3a02
as_compatible_data lose the typing
Illviljan Oct 16, 2023
e459d95
Update indexing.py
Illviljan Oct 16, 2023
4298c06
Update core.py
Illviljan Oct 16, 2023
b7af21b
Update core.py
Illviljan Oct 16, 2023
722711d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 16, 2023
25e4211
Update variable.py
Illviljan Oct 16, 2023
20195de
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 16, 2023
0081cb1
Update variable.py
Illviljan Oct 16, 2023
7d3a2c4
Merge branch 'namedarray_scalartype' of https://github.com/Illviljan/…
Illviljan Oct 16, 2023
559a45d
Update variable.py
Illviljan Oct 16, 2023
ef891b7
Update indexing.py
Illviljan Oct 16, 2023
97e4f00
Update xarray/core/variable.py
Illviljan Oct 16, 2023
a3268c1
cleanup
Illviljan Oct 16, 2023
65cc63c
Merge branch 'namedarray_scalartype' of https://github.com/Illviljan/…
Illviljan Oct 16, 2023
9b41320
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 16, 2023
b317540
Update core.py
Illviljan Oct 16, 2023
141dce5
Merge branch 'namedarray_scalartype' of https://github.com/Illviljan/…
Illviljan Oct 16, 2023
143b913
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 16, 2023
a36a21e
Update core.py
Illviljan Oct 16, 2023
66e3538
Update xarray/core/variable.py
Illviljan Oct 17, 2023
ab7b8ad
Apply suggestions from code review
Illviljan Oct 17, 2023
2f7be4d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 17, 2023
21c323f
Update core.py
Illviljan Oct 17, 2023
71e3d0c
Merge branch 'main' into namedarray_scalartype
Illviljan Oct 18, 2023
55034cc
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 18, 2023
e435e15
Update core.py
Illviljan Oct 18, 2023
c86d8b2
Merge branch 'namedarray_scalartype' of https://github.com/Illviljan/…
Illviljan Oct 18, 2023
1bce439
Update core.py
Illviljan Oct 18, 2023
eaf8ade
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 18, 2023
e807272
Update core.py
Illviljan Oct 18, 2023
9bf347f
Merge branch 'namedarray_scalartype' of https://github.com/Illviljan/…
Illviljan Oct 18, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 30 additions & 6 deletions xarray/core/variable.py
Original file line number Diff line number Diff line change
Expand Up @@ -369,6 +369,25 @@ def __init__(
if encoding is not None:
self.encoding = encoding

def _new(
self,
dims=_default,
data=_default,
attrs=_default,
):
dims_ = copy.copy(self._dims) if dims is _default else dims

if attrs is _default:
attrs_ = None if self._attrs is None else self._attrs.copy()
else:
attrs_ = attrs

if data is _default:
return type(self)(dims_, copy.copy(self._data), attrs_)
else:
cls_ = type(self)
return cls_(dims_, data, attrs_)

@property
def _in_memory(self):
return isinstance(
Expand Down Expand Up @@ -905,16 +924,17 @@ def _copy(
ndata = data_old
else:
# don't share caching between copies
ndata = indexing.MemoryCachedArray(data_old.array)
# TODO: MemoryCachedArray doesn't match the array api:
ndata = indexing.MemoryCachedArray(data_old.array) # type: ignore[assignment]

if deep:
ndata = copy.deepcopy(ndata, memo)

else:
ndata = as_compatible_data(data)
if self.shape != ndata.shape:
if self.shape != ndata.shape: # type: ignore[attr-defined]
raise ValueError(
f"Data shape {ndata.shape} must match shape of object {self.shape}"
f"Data shape {ndata.shape} must match shape of object {self.shape}" # type: ignore[attr-defined]
)

attrs = copy.deepcopy(self._attrs, memo) if deep else copy.copy(self._attrs)
Expand Down Expand Up @@ -1054,7 +1074,8 @@ def chunk(
# Using OuterIndexer is a pragmatic choice: dask does not yet handle
# different indexing types in an explicit way:
# https://github.com/dask/dask/issues/2883
ndata = indexing.ImplicitToExplicitIndexingAdapter(
# TODO: ImplicitToExplicitIndexingAdapter doesn't match the array api:
ndata = indexing.ImplicitToExplicitIndexingAdapter( # type: ignore[assignment]
data_old, indexing.OuterIndexer
)

Expand Down Expand Up @@ -2608,6 +2629,9 @@ class IndexVariable(Variable):

__slots__ = ()

# TODO: PandasIndexingAdapter doesn't match the array api:
_data: PandasIndexingAdapter # type: ignore[assignment]

def __init__(self, dims, data, attrs=None, encoding=None, fastpath=False):
super().__init__(dims, data, attrs, encoding, fastpath)
if self.ndim != 1:
Expand Down Expand Up @@ -2756,9 +2780,9 @@ def copy(self, deep: bool = True, data: T_DuckArray | ArrayLike | None = None):

else:
ndata = as_compatible_data(data)
if self.shape != ndata.shape:
if self.shape != ndata.shape: # type: ignore[attr-defined]
raise ValueError(
f"Data shape {ndata.shape} must match shape of object {self.shape}"
f"Data shape {ndata.shape} must match shape of object {self.shape}" # type: ignore[attr-defined]
)

attrs = copy.deepcopy(self._attrs) if deep else copy.copy(self._attrs)
Expand Down
122 changes: 122 additions & 0 deletions xarray/namedarray/_array_api.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
from types import ModuleType
from typing import Any

import numpy as np

from xarray.namedarray._typing import (
_arrayapi,
_DType,
_ScalarType,
_ShapeType,
_SupportsImag,
_SupportsReal,
)
from xarray.namedarray.core import NamedArray


def _get_data_namespace(x: NamedArray[Any, Any]) -> ModuleType:
if isinstance(x._data, _arrayapi):
return x._data.__array_namespace__()
else:
return np


def astype(
x: NamedArray[_ShapeType, Any], dtype: _DType, /, *, copy: bool = True
) -> NamedArray[_ShapeType, _DType]:
"""
Copies an array to a specified data type irrespective of Type Promotion Rules rules.

Parameters
----------
x : NamedArray
Array to cast.
dtype : _DType
Desired data type.
copy : bool, optional
Specifies whether to copy an array when the specified dtype matches the data
type of the input array x.
If True, a newly allocated array must always be returned.
If False and the specified dtype matches the data type of the input array,
the input array must be returned; otherwise, a newly allocated array must be
returned. Default: True.

Returns
-------
out : NamedArray
An array having the specified data type. The returned array must have the
same shape as x.

Examples
--------
>>> narr = NamedArray(("x",), np.array([1.5, 2.5]))
>>> astype(narr, np.dtype(int)).data
array([1, 2])
"""
if isinstance(x._data, _arrayapi):
xp = x._data.__array_namespace__()
return x._new(data=xp.astype(x, dtype, copy=copy))

# np.astype doesn't exist yet:
return x._new(data=x._data.astype(dtype, copy=copy)) # type: ignore[attr-defined]


def imag(
x: NamedArray[_ShapeType, np.dtype[_SupportsImag[_ScalarType]]], / # type: ignore[type-var]
) -> NamedArray[_ShapeType, np.dtype[_ScalarType]]:
"""
Returns the imaginary component of a complex number for each element x_i of the
input array x.

Parameters
----------
x : NamedArray
Input array. Should have a complex floating-point data type.

Returns
-------
out : NamedArray
An array containing the element-wise results. The returned array must have a
floating-point data type with the same floating-point precision as x
(e.g., if x is complex64, the returned array must have the floating-point
data type float32).

Examples
--------
>>> narr = NamedArray(("x",), np.array([1 + 2j, 2 + 4j]))
>>> imag(narr).data
array([2., 4.])
"""
xp = _get_data_namespace(x)
out = x._new(data=xp.imag(x._data))
return out


def real(
x: NamedArray[_ShapeType, np.dtype[_SupportsReal[_ScalarType]]], / # type: ignore[type-var]
) -> NamedArray[_ShapeType, np.dtype[_ScalarType]]:
"""
Returns the real component of a complex number for each element x_i of the
input array x.

Parameters
----------
x : NamedArray
Input array. Should have a complex floating-point data type.

Returns
-------
out : NamedArray
An array containing the element-wise results. The returned array must have a
floating-point data type with the same floating-point precision as x
(e.g., if x is complex64, the returned array must have the floating-point
data type float32).

Examples
--------
>>> narr = NamedArray(("x",), np.array([1 + 2j, 2 + 4j]))
>>> real(narr).data
array([1., 2.])
"""
xp = _get_data_namespace(x)
return x._new(data=xp.real(x._data))
Loading
Loading