[RELEASE] kvikio v24.12 #572
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Forward-merge branch-24.10 into branch-24.12
Forward-merge branch-24.10 into branch-24.12
Forward-merge branch-24.10 into branch-24.12
Forward-merge branch-24.10 into branch-24.12
Forward-merge branch-24.10 into branch-24.12
* Add return type to `asarray` (makes easy to use with other typed values) * Include `asarray` in `pxd` for faster access within other Cython modules Authors: - https://github.com/jakirkham - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) URL: #475
Contributes to https://github.com/rapdisai/build-planning/issues/74 Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) URL: #478
Forward-merge branch-24.10 into branch-24.12
Forward-merge branch-24.10 into branch-24.12
Forward-merge branch-24.10 into branch-24.12
We need to export static class methods that returns a reference to a static variable like we do with inline functions: #442 Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Bradley Dice (https://github.com/bdice) URL: #492
Contributes to rapidsai/build-planning#94 Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - James Lamb (https://github.com/jameslamb) URL: #489
Support read directly from a http server like:
```python
import kvikio
import cupy
with kvikio.RemoteFile.from_http_url("http://127.0.0.1:9000/myfile") as f:
ary = cupy.empty(f.nbytes, dtype="uint8")
f.read(ary)
```
This PR is the first step to support S3 using libcurl instead of [aws-s3-sdk](#426), which has some pros and cons:
* Pros
* The [global conda pinning issue](#426 (comment)) is less of a problem.
* We can support other protocols such as http, ftp, and Azure’s storage, without much work.
* We avoid the [free-after-main issue in aws-s3-sdk](https://github.com/rapidsai/kvikio/blob/000126516db430988ab9af5ee1576ca3fe6afe27/cpp/include/kvikio/remote_handle.hpp#L87-L94). This is huge since we would otherwise have to pass around a `S3Context` in libcudf and cudf to handle shutdown correctly. This is not a problem in libcurl, see https://curl.se/libcurl/c/libcurl.html under `Global constants`.
* Cons
* Hard to support the AWS configuration file. We will require the user to either specify the options programmatically or through environment variables like `AWS_ACCESS_KEY_ID ` and `AWS_SECRET_ACCESS_KEY `.
Authors:
- Mads R. B. Kristensen (https://github.com/madsbk)
Approvers:
- Kyle Edwards (https://github.com/KyleFromNVIDIA)
- Lawrence Mitchell (https://github.com/wence-)
URL: #464
Forward-merge branch-24.10 into branch-24.12
Contributes to rapidsai/build-planning#106 Proposes specifying the RAPIDS version in `conda install` calls that install CI artifacts, to reduce the risk of CI jobs picking up artifacts from other releases. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Mike Sarahan (https://github.com/msarahan) URL: #495
This PR makes a couple of small, general improvements. - Improve the use of `std::tolower` to avoid UB. References [cppreference](https://en.cppreference.com/w/cpp/string/byte/tolower). - Improve the readability of POSIX read/write interface. - Improve the use of `std::map` to avoid a repeated search in the self-balancing BST, which takes `O(n)`. Authors: - Tianyu Liu (https://github.com/kingcrimsontianyu) - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - MithunR (https://github.com/mythrocks) URL: #493
Makes a few improvements to the Cython logic of `_to_string`: * Types the input as `str`: This allows Cython to use CPython API relevant for the specific type (it will check type on entry) * Do `is not None` check first: This way when Cython generates additional `None` checks inside the branch, the compiler can optimize them out * Use `str`'s `encode` method: Cython makes a direct CPython call to generate `bytes` and then extracts the pointer and size for C++ to copy Authors: - https://github.com/jakirkham Approvers: - Lawrence Mitchell (https://github.com/wence-) URL: #498
These are all pure C functions implemented in Cython. They do not raise. [Cython 3 adds checks for exceptions]( https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#exception-values-and-noexcept ) in these functions, which is unnecessary given none of these set exceptions. So add `noexcept` to turn off these Cython checks. Authors: - https://github.com/jakirkham Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) URL: #502
Works around curl/curl#15351 . We cache the `BUILD_TESTING` cache variable when it exists so that our tests are enabled when building curl from source. Authors: - Robert Maynard (https://github.com/robertmaynard) Approvers: - Tianyu Liu (https://github.com/kingcrimsontianyu) - Bradley Dice (https://github.com/bdice) URL: #503
In `Array`, `Py_ssize_t[::1]` objects are currently backed by [CPython `array`'s]( https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html#cpython-array-module ) with some internal bits expressed in Cython. However these are not compatible with [Python's Limited API and Stable ABI]( https://docs.python.org/3/c-api/stable.html#c-api-stability ). To address that, switch to [Cython's own `array` type]( https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html#cython-arrays ). As this is baked into Cython and doesn't use anything special, it is compatible with Python's Limited API and Stable ABI. xref: rapidsai/build-planning#42 Authors: - https://github.com/jakirkham Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) URL: #504
Implements AWS S3 read support using libcurl:
```python
import kvikio
import cupy
with kvikio.RemoteFile.from_s3_url("s://my-bucket/my-file") as f:
ary = cupy.empty(f.nbytes, dtype="uint8")
f.read(ary)
```
Supersedes #426
Authors:
- Mads R. B. Kristensen (https://github.com/madsbk)
Approvers:
- Bradley Dice (https://github.com/bdice)
- Lawrence Mitchell (https://github.com/wence-)
- Vyas Ramasubramani (https://github.com/vyasr)
URL: #479
Authors: - Robert Maynard (https://github.com/robertmaynard) - https://github.com/jakirkham Approvers: - https://github.com/jakirkham URL: #505
Since #479, the libkvikio wheel now includes platform-specific files. Stop tagging the wheel as "any". Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - Bradley Dice (https://github.com/bdice) URL: #507
closes #510 Authors: - Jacob Faibussowitsch (https://github.com/Jacobfaib) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) URL: #511
Don't use macros for CUDA driver functions, since that does not play nice with `decltype()`. Revert to using type-erased stubs. Authors: - Jacob Faibussowitsch (https://github.com/Jacobfaib) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) URL: #516
Minor cosmetic fixes for README :) Authors: - Jacob Faibussowitsch (https://github.com/Jacobfaib) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) URL: #517
When building with conda, the `__FILE__` macro will contain the (long) conda build path. At install time, this is replaced by the actual path, which may be shorter. Any extra characters are replaced by `\0`. The extra `\0` is problematic if `CurlHandle` later throws an exception to Cython since while converting the exception to Python, Cython might truncate the error message. Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) - Lawrence Mitchell (https://github.com/wence-) URL: #522
Contributes to rapidsai/build-planning#111 Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS. * printing `sccache` stats to CI logs * updating to the latest `rapids-dependency-file-generator` (v1.16.0) * always explicitly specifying `cpp` / `python` in calls to `rapids-upload-wheels-to-s3` Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #524
This updates cudf to use nvcomp 4.1.0.6. The version is updated in rapids-cmake in rapidsai/rapids-cmake#709. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - James Lamb (https://github.com/jameslamb) URL: #525
This PR makes some improvements on NVTX annotations in KvikIO, including:
- Improve NVTX range annotation by registering the range name string (which reduces profiling-time overhead) and making the registered string static (which further reduces the overhead).
- Separate the responsibility of "scoped range" from "function range" to improve clarity. Previously variadic macro `KVIKIO_NVTX_FUNC_RANGE(...)` is used for both the function range and arbitrary scoped range. However, the macro is somewhat a misnomer for the latter case, where `__func__` is not used in the message. This PR introduces `KVIKIO_NVTX_SCOPED_RANGE(msg, val)` for the latter case.
- Add NVTX marker macro `KVIKIO_NVTX_MARKER(message, payload)` to annotate an instantaneous event. One use case is to characterize asynchronous I/O operations (where the scoped range in a function does not apply): Once the profile is collected, the `nsys-rep` file can be exported, for instance, as a `sqlite`, and the marker data can be used to make a I/O-size histogram. As an example, for `PARQUET_READER_NVBENCH---parquet_read_io_compression`, the following I/O statistics can be obtained.
This histogram is generated using the Python script `python/kvikio/examples/kvikio_stat.py`. PR link: TBD...
```
FileHandle::pread()
Bins ...... Count
[ 0 B, 2 B) ...... 0
[ 2 B, 4 B) ...... 0
[ 4 B, 8 B) ...... 0
[ 8 B, 16 B) ...... 0
[ 16 B, 32 B) ...... 0
[ 32 B, 64 B) ...... 0
[ 64 B, 128 B) ...... 0
[ 128 B, 256 B) ...... 0
[ 256 B, 512 B) ...... 0
[ 512 B, 1024 B) ...... 0
[ 1024 B, 2 KB) ...... 0
[ 2 KB, 4 KB) ...... 0
[ 4 KB, 8 KB) ...... 0
[ 8 KB, 16 KB) ...... 0
[ 16 KB, 32 KB) ...... 0
[ 32 KB, 64 KB) ...... 0
[ 64 KB, 128 KB) ...... 0
[ 128 KB, 256 KB) ...... 45
[ 256 KB, 512 KB) ...... 30
[ 512 KB, 1024 KB) ...... 15
[ 1024 KB, 2 MB) ...... 120
[ 2 MB, 4 MB) ...... 150
[ 4 MB, 8 MB) ...... 60
[ 8 MB, 16 MB) ...... 0
[ 16 MB, 32 MB) ...... 0
[ 32 MB, 64 MB) ...... 0
[ 64 MB, 128 MB) ...... 0
[ 128 MB, 256 MB] ...... 30
FileHandle::pwrite()
Bins ...... Count
[ 0 B, 2 B) ...... 0
[ 2 B, 4 B) ...... 0
[ 4 B, 8 B) ...... 0
[ 8 B, 16 B) ...... 0
[ 16 B, 32 B) ...... 0
[ 32 B, 64 B) ...... 0
[ 64 B, 128 B) ...... 0
[ 128 B, 256 B) ...... 0
[ 256 B, 512 B) ...... 0
[ 512 B, 1024 B) ...... 0
[ 1024 B, 2 KB) ...... 0
[ 2 KB, 4 KB) ...... 0
[ 4 KB, 8 KB) ...... 0
[ 8 KB, 16 KB) ...... 0
[ 16 KB, 32 KB) ...... 0
[ 32 KB, 64 KB) ...... 0
[ 64 KB, 128 KB) ...... 0
[ 128 KB, 256 KB) ...... 8
[ 256 KB, 512 KB) ...... 8
[ 512 KB, 1024 KB) ...... 19
[ 1024 KB, 2 MB) ...... 24
[ 2 MB, 4 MB) ...... 4
[ 4 MB, 8 MB) ...... 0
[ 8 MB, 16 MB) ...... 0
[ 16 MB, 32 MB) ...... 0
[ 32 MB, 64 MB) ...... 0
[ 64 MB, 128 MB) ...... 0
[ 128 MB, 256 MB] ...... 2
posix_device_read()
Bins ...... Count
[ 0 B, 2 B) ...... 0
[ 2 B, 4 B) ...... 0
[ 4 B, 8 B) ...... 0
[ 8 B, 16 B) ...... 0
[ 16 B, 32 B) ...... 0
[ 32 B, 64 B) ...... 0
[ 64 B, 128 B) ...... 0
[ 128 B, 256 B) ...... 0
[ 256 B, 512 B) ...... 0
[ 512 B, 1024 B) ...... 0
[ 1024 B, 2 KB) ...... 0
[ 2 KB, 4 KB) ...... 0
[ 4 KB, 8 KB) ...... 0
[ 8 KB, 16 KB) ...... 0
[ 16 KB, 32 KB) ...... 0
[ 32 KB, 64 KB) ...... 0
[ 64 KB, 128 KB) ...... 0
[ 128 KB, 256 KB) ...... 75
[ 256 KB, 512 KB) ...... 60
[ 512 KB, 1024 KB) ...... 15
[ 1024 KB, 2 MB) ...... 150
[ 2 MB, 4 MB] ...... 1710
posix_device_write()
Bins ...... Count
[ 0 B, 2 B) ...... 0
[ 2 B, 4 B) ...... 0
[ 4 B, 8 B) ...... 0
[ 8 B, 16 B) ...... 0
[ 16 B, 32 B) ...... 0
[ 32 B, 64 B) ...... 0
[ 64 B, 128 B) ...... 0
[ 128 B, 256 B) ...... 0
[ 256 B, 512 B) ...... 0
[ 512 B, 1024 B) ...... 0
[ 1024 B, 2 KB) ...... 0
[ 2 KB, 4 KB) ...... 0
[ 4 KB, 8 KB) ...... 0
[ 8 KB, 16 KB) ...... 0
[ 16 KB, 32 KB) ...... 0
[ 32 KB, 64 KB) ...... 0
[ 64 KB, 128 KB) ...... 0
[ 128 KB, 256 KB) ...... 8
[ 256 KB, 512 KB) ...... 8
[ 512 KB, 1024 KB) ...... 19
[ 1024 KB, 2 MB) ...... 24
[ 2 MB, 4 MB] ...... 104
```
Authors:
- Tianyu Liu (https://github.com/kingcrimsontianyu)
- Mads R. B. Kristensen (https://github.com/madsbk)
Approvers:
- Mads R. B. Kristensen (https://github.com/madsbk)
URL: #518
Changes: - Adding Python bindings to `cuFileDriverOpen()` and `cuFileDriverClose()`. - We now [only open the cufile driver explicitly](#160) in CUDA versions older than v12.2. - Introducing `kvikio.cufile_driver.initialize()`, which open the cuFile driver and close it again at module exit. - Let CI fail if KvikIO wasn't built with cuFile support. * Except on cuda11.8+arm64; cuFile didn't support arm until cuda v12.4. - Some refactor and clean up! Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Lawrence Mitchell (https://github.com/wence-) - Vyas Ramasubramani (https://github.com/vyasr) URL: #514
Authors: - https://github.com/RichardScottOZ Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) URL: #532
See rapidsai/devcontainers#414 Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) URL: #534
Moving libcurl dependent functions to `remote_handle.cpp` (let's move the rest in a follow up PR) and setup building `libkvikio.so`.
### Background
Now that KvikIO has evolved into a standalone IO library that works without cuFile and CUDA, I think it makes sense to move to a shared library. Originally, KvikIO was a very thin wrapper around cuFile, which is why we decided to keep it header-only (not because of templates).
**Pros**:
- Enables us to statically compile libcurl into `libkvikio.so`.
* Required by our wheels. If we don’t do this, we will have to find another solution to #512
- Avoid having to compile libcurl in downstream projects. Currently, cudf must compile libcurl in every build.
- Reduce compile time in CI, both for KvikIO and downstream projects.
- Ease development by not having to rebuild downstream projects like cudf every time KvikIO is modified.
**Cons**:
- Projects cannot vendoring KvikIO as a header-only project. As far as we know, nobody does this.
Authors:
- Mads R. B. Kristensen (https://github.com/madsbk)
- Paul Taylor (https://github.com/trxcllnt)
Approvers:
- Robert Maynard (https://github.com/robertmaynard)
- Vyas Ramasubramani (https://github.com/vyasr)
- Bradley Dice (https://github.com/bdice)
URL: #527
... also don't build `remote_handle.cpp` if `KvikIO_REMOTE_SUPPORT=OFF`, which fixes #538 Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) - Bradley Dice (https://github.com/bdice) URL: #539
Due to a bug in cuda-python we must disallow cuda-python 12.6.1 and 11.8.4. This PR disallows those versions. See rapidsai/build-planning#116 for more information. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - https://github.com/jakirkham URL: #537
The CUDA and cufile include directories should not be exported as absolute paths when installing, only in the build tree. Add a `$<BUILD_INTERFACE>` generator expression. Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) - Bradley Dice (https://github.com/bdice) URL: #540
Follow-up to #537 Contributes to rapidsai/build-planning#116 That PR used `!=` requirements to skip a particular version of `cuda-python` that `kvikio` was incompatible with. A newer version of `cuda-python` (12.6.2 for CUDA 12, 11.8.5 for CUDA 11) was just released, and it also causes some build issues for RAPIDS libraries: rapidsai/cuvs#445 (comment) To unblock CI across RAPIDS, this proposes **temporarily** switching to ceilings on the `cuda-python` dependency here. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #543
Related to rapidsai/build-planning#33 and rapidsai/build-planning#74 The last use of CMake function `install_aliased_imported_targets()` here was removed in #478. This proposes removing the file holding its definition. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #545
CMake's `FindCUDAToolkit` has supported cuFile since 3.25. Use this support and remove the custom `FindcuFile` module. Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #542
Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Lawrence Mitchell (https://github.com/wence-) URL: #509
Contributes to rapidsai/build-planning#110 Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI. * checks on wheel size (compressed), - *to be sure they're under PyPI limits* - *and to prompt discussion on PRs that significantly increase wheel sizes* * checks on README formatting - *to ensure they'll render properly as the PyPI project homepages* - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/* Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #548
Contributes to rapidsai/build-planning#118 Modifies `libkvikio.load_library()` in the following ways: * prefer wheel-provided `libkvikio.so` to system installation * expose environment variable `RAPIDS_LIBKVIKIO_PREFER_SYSTEM_LIBRARY` for switching that preference * load `libkvikio.so` with `RTLD_LOCAL`, to prevent adding symbols to the global namespace ([dlopen docs](https://linux.die.net/man/3/dlopen)) ## Notes for Reviewers ### How I tested this Tested the general approach on rapidsai/cudf#17316 and rapidsai/cuspatial#1483. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #551
Backport to get https://gitlab.kitware.com/cmake/cmake/-/commit/b38a8e77cb3c8401b3022a68f07a4fd77b290524, so that `find_library()` finds `libcufile` correctly. Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) URL: #550
Contributes to rapidsai/build-planning#118 The pattern introduced in #551 breaks editable installs in devcontainers. In that type of build, `libkvikio.so` is built outside of the wheel but **not installed**, so it can't be found by `ld`. Extension modules in `kvikio` are able to find it via RPATHs instead. This proposes: * try-catching the entire library-loading attempt, to silently do nothing in cases like that * adding an import of the `kvikio` Python library in the `devcontainers` CI job, as a smoke test to catch issues like this in the future ## Notes for Reviewers ### How I tested this Reproduced that with the CUDA 12.5 pip devcontainers today: ```shell build-all python -c "import kvikio" # OSError: libkvikio.so: cannot open shared object file: No such file or directory ``` Confirmed that the changes in this PR fix that. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) - Lawrence Mitchell (https://github.com/wence-) URL: #553
This PR addresses issue #529. A new compatibility mode `AUTO` (previously proposed as `ALLOW`) is added that explicitly asks KvikIO to determine whether `ON` or `OFF` is eventually used for the I/O. Closes #529 Authors: - Tianyu Liu (https://github.com/kingcrimsontianyu) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - Vukasin Milovanovic (https://github.com/vuule) URL: #547
This PR fixes #558 The root cause is that POSIX and cuFile report different error messages for the same invalid operation which attempts to read from a write-only file. The unit test needs to take account of these different messages. Authors: - Tianyu Liu (https://github.com/kingcrimsontianyu) Approvers: - Vukasin Milovanovic (https://github.com/vuule) - Mads R. B. Kristensen (https://github.com/madsbk) URL: #559
Fixes #566 To clean up and make cufile capability detection more robust, we now use `cuFileGetVersion()`. Should fix the nightly failure: https://github.com/rapidsai/kvikio/actions/runs/12133477083/job/33877440755 NB: `cuFileGetVersion()` first became available in cufile v1.8 (CTK v12.3) thus the stream and batch API detection will return false for versions older than v1.8. I think this is acceptable for robustness. Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Lawrence Mitchell (https://github.com/wence-) - Vyas Ramasubramani (https://github.com/vyasr) - Bradley Dice (https://github.com/bdice) URL: #565
…apids-dask-dependency` (#513) Explicitly include dependencies on `libcufile*` and `*cudart*` in the Python Conda package. These seem to be pulled in implicitly by the C++ package. However as they are dependencies of the Python built shared objects, they should also be listed here. Also consolidate the Conda environment creation and installation of nightly dependencies into one step. By doing this in one step, we guarantee that all of our constraints are taken into account during environment creation. Lastly switch from using `dask` directly to `rapids-dask-dependency` to align Dask installation and pinning with the rest of RAPIDS. xref: rapidsai/build-planning#22 xef: #378 Authors: - https://github.com/jakirkham - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Bradley Dice (https://github.com/bdice) - Mads R. B. Kristensen (https://github.com/madsbk) URL: #513
This PR uses `libcurl` 8.5.0 at build time, and should permit `>=8.5.0,<9.0a0` at runtime. This is needed to align with conda-forge which uses `libcurl` 8, and also gets a new enough minor version to avoid some known CVEs. --------- Co-authored-by: jakirkham <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
❄️ Code freeze for
branch-24.12and v24.12 releaseWhat does this mean?
Only critical/hotfix level issues should be merged into
branch-24.12until release (merging of this PR).What is the purpose of this PR?
branch-24.12intomainfor the release