[RELEASE] kvikio v23.10 #299
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Forward-merge branch-23.08 to branch-23.10
Switch to using rapids-cmake's rapids-cython modules for the Python build. It also uses rapids-cuda to support NATIVE or RAPIDS as the list of CUDA architectures. Resolves #186 Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) URL: #257
This PR enables a roundtrip between codecs other than LZ4. For LZ4 codec, there is already such support. The "roundtrip" here means ability to compress/store data using default numcodecs codec (e.g. `zstd`) and decompress using nvCOMP batch codecs. Reverse would also be true: ability to compress data using nvCOMP and decompress using CPU numcodecs codec. The original implementation of the nvCOMP codec was based on the assumption, taken from numcodecs LZ4 codec, that in order for the codecs to be compatible, each codec must write a small header (4 bytes) in the compressed chunk, which contains the original, uncompressed size of the data. However, after subsequent analysis of other numcodecs codecs, it turned out LZ4 was actually an exception! No other codec writes a header or any additional data to the compressed chunk. This PR: * moves the header creation down to a specific algorithm, LZ4 * switches to nvCOMP `nvcompBatched*GetDecompressSizeAsync` family of functions to get the original size of data during decompression. Authors: - Alexey Kamenev (https://github.com/Alexey-Kamenev) - Lawrence Mitchell (https://github.com/wence-) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - Lawrence Mitchell (https://github.com/wence-) URL: #253
There's no need for this upper bound pinning, the docs build with the latest Sphinx. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - AJ Schmidt (https://github.com/ajschmidt8) - Ajay Thorve (https://github.com/AjayThorve) URL: #260
Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Vukasin Milovanovic (https://github.com/vuule) URL: #262
Forward-merge branch-23.08 to branch-23.10
Fixes #143 cc. @ivirshup @jakirkham Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Alexey Kamenev (https://github.com/Alexey-Kamenev) - Benjamin Zaitlen (https://github.com/quasiben) URL: #261
This PR bumps the pinning and fixes the new warnings that appeared Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Ray Douglass (https://github.com/raydouglass) - Alexey Kamenev (https://github.com/Alexey-Kamenev) - Vukasin Milovanovic (https://github.com/vuule) - https://github.com/jakirkham URL: #258
Authors: - https://github.com/tell-rebanta - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) URL: #259
Introducing `open_cupy_array()`, which is a CUDA friendly version of `zarr.open_array` that reads and writes to CuPy arrays. Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Benjamin Zaitlen (https://github.com/quasiben) - Lawrence Mitchell (https://github.com/wence-) URL: #267
This PR replaces the `copy_prs` functionality from the `ops-bot` with the new dedicated `copy-pr-bot` GitHub application. Thorough documentation for the new `copy-pr-bot` application can be viewed below. - https://docs.gha-runners.nvidia.com/apps/copy-pr-bot/ **Important**: `copy-pr-bot` enforces signed commits. If an organization member opens a PR that contains unsigned commits, it will be deemed untrusted and therefore require an `/ok to test` comment. See the GitHub docs [here](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification) for information on how to set up commit signing. Any time a PR is deemed untrusted, it will receive a comment that looks like this: rapidsai/ci-imgs#63 (comment). Every subsequent commit on an untrusted PR will require an additional `/ok to test` comment. Any existing PRs that have unsigned commits after this change is merged will require an `/ok to test` comment for each subsequent commit _or_ the PR can be rebased to include signed commits as mentioned in the docs below: https://docs.gha-runners.nvidia.com/cpr/contributors. This information is all included on the documentation page linked above. _I've skipped CI on this PR since it's not a change that is tested._ [skip ci]
We now use the `cuFileReadAsync` symbol to check for availability of both the stream and batch API of cuFile. We used to look for the mangled symbol `_ZTS13CUfileOpError` to determine the availability of the batch API. However, since mangling is compiler specific, we prefer using `cuFileReadAsync` even though it means that the batch API isn't available until CUDA version 12.2. Additionally, this PR also implements `CUFILE_CHECK_STREAM_IO()`, which is used to check async IO errors. cc. @tell-rebanta Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Robert Maynard (https://github.com/robertmaynard) - Lawrence Mitchell (https://github.com/wence-) URL: #271
`read_async()` and `write_async()` now works in compat mode by falling back to the blocking implementation Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Lawrence Mitchell (https://github.com/wence-) URL: #272
I selected this version as it is what ships with Ubuntu 22.04. I also ran `doxygen -u` to update the Doxyfile. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Ray Douglass (https://github.com/raydouglass) - Mads R. B. Kristensen (https://github.com/madsbk) URL: #277
With the release of conda 23.7.3, `mamba mambabuild` stopped working. With boa installed, `conda mambabuild` uses the mamba solver, so just use that instead. See also rapidsai/cudf#14068. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Ray Douglass (https://github.com/raydouglass) URL: #278
Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - AJ Schmidt (https://github.com/ajschmidt8) - Benjamin Zaitlen (https://github.com/quasiben) - Vukasin Milovanovic (https://github.com/vuule) URL: #268
Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Lawrence Mitchell (https://github.com/wence-) URL: #279
This PR adds some [devcontainers](https://containers.dev/) to help simplify building the KvikIO C++ and Python libraries. It also adds an optional job to the `pr.yaml` to [build the KvikIO libs in each devcontainer](https://github.com/trxcllnt/kvikio/blob/fea/devcontainers/.github/workflows/pr.yaml#L62-L68), so the build caches are populated for devs by CI. A devcontainer can be launched by clicking the "Reopen in Container" button that VSCode shows when opening the repo (or by using the "Rebuild and Reopen in Container" command from the command palette):  Clicking this button will cause VSCode to prompt the user to select one of these devcontainer variants:  On startup, the devcontainer creates or updates the conda/pip environment using `kvikio/dependencies.yaml`. The envs/package caches are cached on the host via volume mounts, which are described in more detail in [`.devcontainer/README.md`](https://github.com/trxcllnt/kvikio/blob/fea/devcontainers/.devcontainer/README.md). The container includes convenience functions to clean, configure, and build the various KvikIO components: ```shell $ clean-kvikio-cpp # only cleans the C++ build dir $ clean-kvikio-python # only cleans the Python build dir $ clean-kvikio # cleans both C++ and Python build dirs $ configure-kvikio-cpp # only configures kvikio C++ lib $ build-kvikio-cpp # only builds kvikio C++ lib $ build-kvikio-python # only builds kvikio Python lib $ build-kvikio # builds both C++ and Python libs ``` * The C++ build script is a small wrapper around `cmake -S ~/kvikio/cpp -B ~/kvikio/cpp/build` and `cmake --build ~/kvikio/cpp/build` * The Python build script is a small wrapper around `pip install --editable ~/kvikio/cpp` Unlike `build.sh`, these convenience scripts *don't* install the libraries after building them. Instead, they automatically inject the correct arguments to build the C++ libraries from source and use their build dirs as package roots: ```shell $ cmake -S ~/kvikio/cpp -B ~/kvikio/cpp/build $ CMAKE_ARGS="-Dkvikio_ROOT=~/kvikio/cpp/build" \ # <-- this argument is automatic pip install -e ~/kvikio/cpp ``` Authors: - Paul Taylor (https://github.com/trxcllnt) - Vyas Ramasubramani (https://github.com/vyasr) - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - Jake Awe (https://github.com/AyodeAwe) URL: #273
This PR updates kvikio to use clang 16.0.6. The previous version has some minor formatting issues affecting several RAPIDS repos. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) URL: #280
Implements a version of `read_async` and `write_async` that takes by-value arguments and returns a `StreamFuture`. This is to make the common case easy where the user knows the file and buffer size. cc. @tell-rebanta Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - https://github.com/tell-rebanta - Lawrence Mitchell (https://github.com/wence-) URL: #275
PR updates `rapidsai/ci` references to `rapidsai/ci-conda` Authors: - Jake Awe (https://github.com/AyodeAwe) Approvers: - AJ Schmidt (https://github.com/ajschmidt8) URL: #284
Just adds the missing file parameter to a `sed_runner` command I've tested this locally and CI does not test this file. Authors: - Ray Douglass (https://github.com/raydouglass) Approvers: - AJ Schmidt (https://github.com/ajschmidt8)
Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Benjamin Zaitlen (https://github.com/quasiben) URL: #274
rapidsai/shared-workflows#122 was merged, so the devcontainers workflow should use `branch-23.10` for the release. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Ray Douglass (https://github.com/raydouglass) URL: #289
numcodecs 0.12 appears to have some breaking changes. I've submitted a fix upstream, but for this release of RAPIDS we'll need to upper bound the version to be safe. xref: zarr-developers/numcodecs#475 Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
❄️ Code freeze for
branch-23.10and v23.10 releaseWhat does this mean?
Only critical/hotfix level issues should be merged into
branch-23.10until release (merging of this PR).What is the purpose of this PR?
branch-23.10intomainfor the release