Skip to content

Commit

Permalink
Merge branch 'main' into feat/write-empty-chunks
Browse files Browse the repository at this point in the history
  • Loading branch information
@d-v-b
d-v-b authored Dec 19, 2024
2 parents 3493c70 + 5bf7bcf commit d21d3ce
Show file tree
Hide file tree
Showing 41 changed files with 1,190 additions and 311 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/gpu_test.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ jobs:
cache: 'pip'
- name: Install Hatch and CuPy
run: |
python -m pip install --upgrade pip
python -m pip install --upgrade pip
pip install hatch
- name: Set Up Hatch Env
run: |
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/releases.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:

- name: Install PyBuild
run: |
python -m pip install --upgrade pip
python -m pip install --upgrade pip
pip install hatch
- name: Build wheel and sdist
run: hatch build
Expand Down Expand Up @@ -55,7 +55,7 @@ jobs:
with:
name: releases
path: dist
- uses: pypa/[email protected].2
- uses: pypa/[email protected].3
with:
user: __token__
password: ${{ secrets.pypi_password }}
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/test.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ jobs:
cache: 'pip'
- name: Install Hatch
run: |
python -m pip install --upgrade pip
python -m pip install --upgrade pip
pip install hatch
- name: Set Up Hatch Env
run: |
Expand Down Expand Up @@ -84,7 +84,7 @@ jobs:
cache: 'pip'
- name: Install Hatch
run: |
python -m pip install --upgrade pip
python -m pip install --upgrade pip
pip install hatch
- name: Set Up Hatch Env
run: |
Expand Down
4 changes: 3 additions & 1 deletion .pre-commit-config.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,16 +20,18 @@ repos:
rev: v5.0.0
hooks:
- id: check-yaml
- id: trailing-whitespace
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.13.0
hooks:
- id: mypy
files: src|tests
additional_dependencies:
# Package dependencies
- packaging
- donfig
- numcodecs[crc32c]
- numpy
- numpy==2.1 # until https://github.com/numpy/numpy/issues/28034 is resolved
- typing_extensions
- universal-pathlib
# Tests
Expand Down
2 changes: 1 addition & 1 deletion README-v3.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ hatch env create test
## Run the Tests

```
hatch run test:run
hatch run test:run
```

or
Expand Down
40 changes: 20 additions & 20 deletions bench/compress_normal.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Line # Hits Time Per Hit % Time Line Contents
==============================================================
137 def compress(source, char* cname, int clevel, int shuffle):
138 """Compress data in a numpy array.
139
139
140 Parameters
141 ----------
142 source : array-like
Expand All @@ -30,33 +30,33 @@ Line # Hits Time Per Hit % Time Line Contents
147 Compression level.
148 shuffle : int
149 Shuffle filter.
150
150
151 Returns
152 -------
153 dest : bytes-like
154 Compressed data.
155
155
156 """
157
157
158 cdef:
159 char *source_ptr
160 char *dest_ptr
161 Py_buffer source_buffer
162 size_t nbytes, cbytes, itemsize
163 200 506 2.5 0.2 array.array char_array_template = array.array('b', [])
164 array.array dest
165
165
166 # setup source buffer
167 200 458 2.3 0.2 PyObject_GetBuffer(source, &source_buffer, PyBUF_ANY_CONTIGUOUS)
168 200 119 0.6 0.0 source_ptr = <char *> source_buffer.buf
169
169
170 # setup destination
171 200 239 1.2 0.1 nbytes = source_buffer.len
172 200 103 0.5 0.0 itemsize = source_buffer.itemsize
173 200 2286 11.4 0.8 dest = array.clone(char_array_template, nbytes + BLOSC_MAX_OVERHEAD,
174 zero=False)
175 200 129 0.6 0.0 dest_ptr = <char *> dest.data.as_voidptr
176
176
177 # perform compression
178 200 1734 8.7 0.6 if _get_use_threads():
179 # allow blosc to use threads internally
Expand All @@ -67,24 +67,24 @@ Line # Hits Time Per Hit % Time Line Contents
184 cbytes = blosc_compress(clevel, shuffle, itemsize, nbytes,
185 source_ptr, dest_ptr,
186 nbytes + BLOSC_MAX_OVERHEAD)
187
187
188 else:
189 with nogil:
190 cbytes = blosc_compress_ctx(clevel, shuffle, itemsize, nbytes,
191 source_ptr, dest_ptr,
192 nbytes + BLOSC_MAX_OVERHEAD, cname,
193 0, 1)
194
194
195 # release source buffer
196 200 616 3.1 0.2 PyBuffer_Release(&source_buffer)
197
197
198 # check compression was successful
199 200 120 0.6 0.0 if cbytes <= 0:
200 raise RuntimeError('error during blosc compression: %d' % cbytes)
201
201
202 # resize after compression
203 200 1896 9.5 0.6 array.resize(dest, cbytes)
204
204
205 200 186 0.9 0.1 return dest

*******************************************************************************
Expand All @@ -100,19 +100,19 @@ Line # Hits Time Per Hit % Time Line Contents
==============================================================
75 def decompress(source, dest):
76 """Decompress data.
77
77
78 Parameters
79 ----------
80 source : bytes-like
81 Compressed data, including blosc header.
82 dest : array-like
83 Object to decompress into.
84
84
85 Notes
86 -----
87 Assumes that the size of the destination buffer is correct for the size of
88 the uncompressed data.
89
89
90 """
91 cdef:
92 int ret
Expand All @@ -122,7 +122,7 @@ Line # Hits Time Per Hit % Time Line Contents
96 array.array source_array
97 Py_buffer dest_buffer
98 size_t nbytes
99
99
100 # setup source buffer
101 200 573 2.9 0.2 if PY2 and isinstance(source, array.array):
102 # workaround fact that array.array does not support new-style buffer
Expand All @@ -134,13 +134,13 @@ Line # Hits Time Per Hit % Time Line Contents
108 200 112 0.6 0.0 release_source_buffer = True
109 200 144 0.7 0.1 PyObject_GetBuffer(source, &source_buffer, PyBUF_ANY_CONTIGUOUS)
110 200 98 0.5 0.0 source_ptr = <char *> source_buffer.buf
111
111
112 # setup destination buffer
113 200 552 2.8 0.2 PyObject_GetBuffer(dest, &dest_buffer,
114 PyBUF_ANY_CONTIGUOUS | PyBUF_WRITEABLE)
115 200 100 0.5 0.0 dest_ptr = <char *> dest_buffer.buf
116 200 84 0.4 0.0 nbytes = dest_buffer.len
117
117
118 # perform decompression
119 200 1856 9.3 0.8 if _get_use_threads():
120 # allow blosc to use threads internally
Expand All @@ -149,12 +149,12 @@ Line # Hits Time Per Hit % Time Line Contents
123 else:
124 with nogil:
125 ret = blosc_decompress_ctx(source_ptr, dest_ptr, nbytes, 1)
126
126
127 # release buffers
128 200 754 3.8 0.3 if release_source_buffer:
129 200 326 1.6 0.1 PyBuffer_Release(&source_buffer)
130 200 165 0.8 0.1 PyBuffer_Release(&dest_buffer)
131
131
132 # handle errors
133 200 128 0.6 0.1 if ret <= 0:
134 raise RuntimeError('error during blosc decompression: %d' % ret)
9 changes: 9 additions & 0 deletions docs/conf.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
"sphinx_issues",
"sphinx_copybutton",
"sphinx_design",
'sphinx_reredirects',
]

issues_github_path = "zarr-developers/zarr-python"
Expand Down Expand Up @@ -81,6 +82,14 @@
version = get_version("zarr")
release = get_version("zarr")

redirects = {
"spec": "https://zarr-specs.readthedocs.io",
"spec/v1": 'https://zarr-specs.readthedocs.io/en/latest/v1/v1.0.html',
"spec/v2": "https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html",
"spec/v3": "https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html",
"license": "https://github.com/zarr-developers/zarr-python/blob/main/LICENSE.txt"
}

# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
Expand Down
7 changes: 4 additions & 3 deletions docs/contributing.rst
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
Contributing to Zarr
====================
Contributing
============

Zarr is a community maintained project. We welcome contributions in the form of bug
reports, bug fixes, documentation, enhancement proposals and more. This page provides
Expand Down Expand Up @@ -307,7 +307,8 @@ Data format compatibility
The data format used by Zarr is defined by a specification document, which should be
platform-independent and contain sufficient detail to construct an interoperable
software library to read and/or write Zarr data using any programming language. The
latest version of the specification document is available from the :ref:`spec` page.
latest version of the specification document is available on the
`Zarr specifications website <https://zarr-specs.readthedocs.io>`_.

Here, **data format compatibility** means that all software libraries that implement a
particular version of the Zarr storage specification are interoperable, in the sense
Expand Down
1 change: 1 addition & 0 deletions docs/guide/index.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,6 @@ Guide
.. toctree::
:maxdepth: 1

whatsnew_v3
storage
consolidated_metadata
10 changes: 5 additions & 5 deletions docs/guide/storage.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Storage
Zarr-Python supports multiple storage backends, including: local file systems,
Zip files, remote stores via ``fsspec`` (S3, HTTP, etc.), and in-memory stores. In
Zarr-Python 3, stores must implement the abstract store API from
:class:`zarr.abc.store.Store`.
:class:`zarr.abc.store.Store`.

.. note::
Unlike Zarr-Python 2 where the store interface was built around a generic ``MutableMapping``
Expand Down Expand Up @@ -50,8 +50,8 @@ filesystem.
Zip Store
~~~~~~~~~

The :class:`zarr.storage.ZipStore` stores the contents of a Zarr hierarchy in a single
Zip file. The `Zip Store specification_` is currently in draft form.
The :class:`zarr.storage.ZipStore` stores the contents of a Zarr hierarchy in a single
Zip file. The `Zip Store specification_` is currently in draft form.

.. code-block:: python
Expand All @@ -65,7 +65,7 @@ Remote Store

The :class:`zarr.storage.RemoteStore` stores the contents of a Zarr hierarchy in following the same
logical layout as the ``LocalStore``, except the store is assumed to be on a remote storage system
such as cloud object storage (e.g. AWS S3, Google Cloud Storage, Azure Blob Store). The
such as cloud object storage (e.g. AWS S3, Google Cloud Storage, Azure Blob Store). The
:class:`zarr.storage.RemoteStore` is backed by `Fsspec_` and can support any Fsspec backend
that implements the `AbstractFileSystem` API,

Expand All @@ -80,7 +80,7 @@ Memory Store
~~~~~~~~~~~~

The :class:`zarr.storage.RemoteStore` a in-memory store that allows for serialization of
Zarr data (metadata and chunks) to a dictionary.
Zarr data (metadata and chunks) to a dictionary.

.. code-block:: python
Expand Down
14 changes: 14 additions & 0 deletions docs/guide/whatsnew_v3.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
What's new in v3
================

This page gives an overview of major changes and additions in version 3.


Dependencies
------------
- The new ``remote`` dependency group can be used to install a supported version of
``fsspec``, required for remote data access.
- The new ``gpu`` dependency group can be used to install a supported version of
``cuda``, required for GPU functionality.
- The ``jupyter`` optional dependency group has been removed, since v3 contains no
jupyter specific functionality.
5 changes: 2 additions & 3 deletions docs/index.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,7 @@ Zarr-Python
tutorial
guide/index
api/index
spec
release
license
contributing
roadmap

Expand All @@ -26,7 +24,8 @@ Zarr-Python
`Installation <installation.html>`_ |
`Source Repository <https://github.com/zarr-developers/zarr-python>`_ |
`Issue Tracker <https://github.com/zarr-developers/zarr-python/issues>`_ |
`Zulip Chat <https://ossci.zulipchat.com/>`_
`Zulip Chat <https://ossci.zulipchat.com/>`_ |
`Zarr specifications <https://zarr-specs.readthedocs.io>`_

Zarr is a file storage format for chunked, compressed, N-dimensional arrays based on an open-source specification.

Expand Down
4 changes: 0 additions & 4 deletions docs/license.rst
View file

This file was deleted.

Loading

0 comments on commit d21d3ce

Please sign in to comment.