Initial round of updates #1
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…#11539) As suggested in rapidsai#11526 and captured in issue rapidsai#11536 the usage of both INT8 and UINT8 as supported types for byte_arrays is unnecessary and adds complexity to the code. This change removes INT8 as an option and only allows UINT8 columns to be written out as byte_arrays. ~~This matches with cudf string columns which contain an INT8 column for data.~~ closes rapidsai#11536 Authors: - Mike Wilson (https://github.com/hyperbolic2346) Approvers: - Tobias Ribizel (https://github.com/upsj) - Nghia Truong (https://github.com/ttnghia) - David Wendt (https://github.com/davidwendt) - MithunR (https://github.com/mythrocks) - Bradley Dice (https://github.com/bdice) URL: rapidsai#11539
…#11853) This PR adds a section to the developer documentation about various libcudf design decisions that affect users. These policies are important for us to document and communicate consistently. I am not sure what the best place for this information is, but I think the developer docs are a good place to start since until we address rapidsai#11481 we don't have a great way to publish any non-API user-facing libcudf documentation. I've created this draft PR to solicit feedback from other libcudf devs about other policies that we should be documenting in a similar manner. Once everyone is happy with the contents, I would suggest that we merge this into the dev docs for now and then revisit a better place once we've tackled rapidsai#11481. Partly addresses rapidsai#5505, rapidsai#1781. Resolves rapidsai#4511. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Jake Hemstad (https://github.com/jrhemstad) - Bradley Dice (https://github.com/bdice) - David Wendt (https://github.com/davidwendt) URL: rapidsai#11853
…i#11736) Resolves rapidsai#11684, required for eventually supporting Python 3.10 (which requires flake8 >= 4.0.0). flake8 >= 4.0.0, however, does not support parsing Cython code, even with rule exclusions. This necessitates the flake8-force plugin, which was designed (by a cupy developer) for forcing flake8 to check Cython code with a limited set of rules. Per this comment (rapidsai#11684 (comment)), this PR removes duplicate pinnings between pre-commit configuration and the developer conda environment. Developers should use pre-commit for style checks consistent with the CI environment. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Lawrence Mitchell (https://github.com/wence-) - AJ Schmidt (https://github.com/ajschmidt8) - GALI PREM SAGAR (https://github.com/galipremsagar) - Vyas Ramasubramani (https://github.com/vyasr) URL: rapidsai#11736
This adds the method `boolean onAllocFailure(long sizeRequested, int retryCount)` to `RmmEventHandler`, to help handling code keep track of the number of times an allocation failure has been retried. With this code callers can perform extra logic that depends on whether the callback was due to a brand new allocation failure, or one that has failed in the past and is being retried. This will be used here: NVIDIA/spark-rapids#6768 Authors: - Alessandro Bellina (https://github.com/abellina) Approvers: - Jason Lowe (https://github.com/jlowe) - Nghia Truong (https://github.com/ttnghia) URL: rapidsai#11940
Refactors the main device code used for `cudf::strings::pad` and `cudf::strings::zfill` for reuse in strings UDF pad and zfill functions. No new functions or features have been added, updated, or removed. The detail functions have been mainly just be moved to new file `cpp/include/cudf/strings/detail/pad_impl.cuh` Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Nghia Truong (https://github.com/ttnghia) - Tobias Ribizel (https://github.com/upsj) URL: rapidsai#11914
…apidsai#11917) Fixes a few simple gtests that may not get touched in the course of other PRs. This removes the `using namespace cudf::test` or similar declaration from gtests where it is improperly used. No code logic has changed just variable declarations and function calls. Reference rapidsai#11734 Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Nghia Truong (https://github.com/ttnghia) - Tobias Ribizel (https://github.com/upsj) - Vyas Ramasubramani (https://github.com/vyasr) URL: rapidsai#11917
This PR depends on dask/dask#9475 (**Now Merged**) After dask#9475, external libraries are now able to implement (and expose) their own `DataFrameBackendEntrypoint` definitions to specify custom creation functions for DataFrame collections. This PR introduces the `CudfBackendEntrypoint` class to create `dask_cudf.DataFrame` collections using the `dask.dataframe` API. By installing `dask_cudf` with this entrypoint definition in place, you get the following behavior in `dask.dataframe`: ```python import dask.dataframe as dd import dask # Tell Dask that you want to create DataFrame collections # with the "cudf" backend (for supported creation functions). # This can also be used in a context, or set in a yaml file dask.config.set({"dataframe.backend": "cudf"}) ddf = dd.from_dict({"a": range(10)}, npartitions=2) type(ddf) # dask_cudf.core.DataFrame ``` Note that the code snippet above does not require an explicit import of `cudf` or `dask_cudf`. The following creation functions will support backend dispatching after dask#9475: - `from_dict` - `read_paquet` - `read_json` - `read_orc` - `read_csv` - `read_hdf` See also: dask/design-docs#1 Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) URL: rapidsai#11920
This PR removes optional validation for some APIs. Performing these validations requires data introspection, which we do not want. This PR resolves rapidsai#5505. Authors: - Vyas Ramasubramani (https://github.com/vyasr) - David Wendt (https://github.com/davidwendt) Approvers: - Mark Harris (https://github.com/harrism) - GALI PREM SAGAR (https://github.com/galipremsagar) - Matthew Roeschke (https://github.com/mroeschke) - David Wendt (https://github.com/davidwendt) - Nghia Truong (https://github.com/ttnghia) - Jason Lowe (https://github.com/jlowe) URL: rapidsai#11938
On machines with multiple CUDA Toolkits installed it is possible to have a mismatch between the version of `nvcc` used to compile code and the version of `libnvrtc` used for the JIT code. This generally occurs when `LD_LIBRARY_PATH` points to a different version of the CUDA Toolkit. We now explicitly specify what toolkit library directory to search when JIT code during libcudf compilation. Authors: - Robert Maynard (https://github.com/robertmaynard) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: rapidsai#11787
This PR fixes an error that can occur when very small page sizes are used when writing Parquet files. rapidsai#11551 changed from fixed 5000 row page fragments to a scaled value based on the requested max page size. For small page sizes, the number of fragments to process can exceed 64k. The number of fragments is used as the `y` dimension when calling `gpuInitPageFragments`, and when it exceeds 64k the kernel fails to launch, ultimately leading to an invalid memory access. Authors: - Ed Seidl (https://github.com/etseidl) Approvers: - Vukasin Milovanovic (https://github.com/vuule) - Bradley Dice (https://github.com/bdice) - Karthikeyan (https://github.com/karthikeyann) URL: rapidsai#11869
Partially resolves rapidsai#11329. This helps to align our default behaviors for null and NaN equality across APIs, specifically for `make_collect_set_aggregation` in this PR. All functions should default to treating null values as equal to one another and NaN values as equal to one another. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - David Wendt (https://github.com/davidwendt) - Nghia Truong (https://github.com/ttnghia) URL: rapidsai#11621
For consistency across our documentation, this PR renames `libcudf++` to `libcudf`. Authors: - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - David Wendt (https://github.com/davidwendt) - GALI PREM SAGAR (https://github.com/galipremsagar) - Vyas Ramasubramani (https://github.com/vyasr) - Nghia Truong (https://github.com/ttnghia) URL: rapidsai#11953
…udf::test namespace (rapidsai#11959) Update text to include coding tests outside the `cudf` or the `cudf::test` namespace. Realized our test guidelines needed to be updated while working on rapidsai#11734. Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) - Nghia Truong (https://github.com/ttnghia) URL: rapidsai#11959
…i#11875) This PR ensures that cudf's default stream is properly passed to all kernel launches so that nothing implicitly runs on the CUDA default stream. It adds a small library that is built during the tests and overloads CUDA functions to throw an exception when usage of the default stream is detected. It also fixes all remaining usage of anything other than cudf's default stream (I fixed most of the issues in previous PRs, but I found a few others when finalizing this one). Resolves rapidsai#11929 Resolves rapidsai#11942 ### Important notes for reviewers: - **The changeset is deceptively large.** The vast majority of the changes are just a global find-and-replace of `cudf::get_default_stream()` for `cudf::default_stream_value`, as well as a few smaller fixes such as missing `CUDF_TEST_PROGRAM_MAIN` in a couple of tests and usage of `rmm::cuda_stream_default`. The meaningful changes are: - The new default stream getter/setter in `default_stream.[hpp|cpp]` - The addition of `cpp/tests/utilities/identify_stream_usage` - The changes to the base testing fixture in `cpp/include/cudf_test/base_fixture.hpp` to inject the custom stream. - The changes to CI in `ci/gpu/build.sh` to build and use the new library. - This PR is a breaking change because it moves the default stream into the detail namespace. Going forward the default stream may only be accessed using the public accessor `cudf::get_default_stream()`. I have added a corresponding setter, but it is also in the detail namespace since I do not want to publicly support changing the default stream yet, only for the purpose of testing. Reviewers, please leave comments if you disagree with those choices. - I have made getting and setting the default stream thread-safe, but there is still only a single stream. In multi-threaded applications we may want to support a stream per thread so that users could manually achieve PTDS with more fine-tuned control. Is this worthwhile? Even if it is, I'm inclined to wait for a subsequent PR to implement this unless someone feels strongly otherwise. - I'm currently only overloading `cudaLaunchKernel`. I can add overloads for other functions as well, but I didn't want to go through the effort of overloading every possible API. If reviewers have a minimal set that they'd like to see overloaded, let me know. [I've included links to all the relevant pages of the CUDA runtime API in the identify_stream_usage.cu file](https://github.com/rapidsai/cudf/pull/11875/files#diff-0b2762207c27c080acd2114475c7a1c06377a7c18c4e9c3de60ecbdc82a4dc61R99) if someone wants to look through them. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Jason Lowe (https://github.com/jlowe) - Bradley Dice (https://github.com/bdice) - Sevag H (https://github.com/sevagh) - https://github.com/brandon-b-miller - Jake Hemstad (https://github.com/jrhemstad) - David Wendt (https://github.com/davidwendt) URL: rapidsai#11875
… APIs. (rapidsai#11960) There is almost never a good reason to pass arguments as `unique_ptr<T> const&`. Since those arguments cannot be modified, the only use case is accessing the underlying pointer, at which point the function better communicates its intent by accepting the underlying pointer/reference as an argument instead and is also more flexible as a result. Resolves rapidsai#10393 Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) - Nghia Truong (https://github.com/ttnghia) URL: rapidsai#11960
Closes rapidsai#11916 cuda memcheck reports an OOB write in one of the tests. The root cause is an underallocated buffer for encoded pages. This PR fixes the computation of the maximum size of data pages (RLE encoded) when dictionary encoding is used. Other changes: Refactored max RLE page size computation to avoid code repetition. Use actual dictionary index width instead of (outdated) worst case. Authors: - Vukasin Milovanovic (https://github.com/vuule) Approvers: - David Wendt (https://github.com/davidwendt) - Bradley Dice (https://github.com/bdice) URL: rapidsai#11962
Adds support for reading parquet files with V2 page headers. Fixes rapidsai#11686 ~~Submitting as draft for now because I'm not sure how to do unit tests for this. libcudf cannot produce files with V2 headers, so I would need to either add files to a data directory somewhere, or add raw binary of some parquet files to parquet_test.cpp. Given the comment on the `DecimalRead` test, neither seems attractive. Suggestions are welcome. Perhaps use python to test?~~ Authors: - Ed Seidl (https://github.com/etseidl) Approvers: - Vukasin Milovanovic (https://github.com/vuule) - Mike Wilson (https://github.com/hyperbolic2346) - Matthew Roeschke (https://github.com/mroeschke) URL: rapidsai#11778
Partially resolves rapidsai#11329. This helps to align our default behaviors for null and NaN equality across APIs, specifically for `make_merge_sets_aggregation` in this PR. All functions should default to treating null values as equal to one another and NaN values as equal to one another. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Nghia Truong (https://github.com/ttnghia) - Vyas Ramasubramani (https://github.com/vyasr) - David Wendt (https://github.com/davidwendt) URL: rapidsai#11952
Now that rapids-cmake supports custom patches we can move cudf over to rapids-cmake for Thrust. This removes the need for custom install rules in cudf for Thrust, as rapids-cmake does that for us. This also separates out all Thrust patches so that we can better track upstream approval and remove as needed. Authors: - Robert Maynard (https://github.com/robertmaynard) Approvers: - Bradley Dice (https://github.com/bdice) - David Wendt (https://github.com/davidwendt) URL: rapidsai#11921
…11956) Fixes structs and lists gtests source files coded in namespace `cudf::test` These are the only 2 problem files for this in `cpp/tests/structs` and `cpp/tests/lists` and so will make those two directories complete. No function or test has changed just the source code reworked per namespaces. Reference rapidsai#11734 Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Vukasin Milovanovic (https://github.com/vuule) - Nghia Truong (https://github.com/ttnghia) URL: rapidsai#11956
) Simplifies the `cudf::strings::strip` function to use the `cudf::make_strings_column` that accepts an iterator of pairs. This factory has a highly tuned gather implementation for building a strings column from an vector (iterator) of strings in device memory. This was inspired by the review and work in rapidsai#11946. This also gives a small improvement in the performance of small columns of large strings and even more improvement in large columns of large-ish strings for strip. No function has changed just the internal implementation has been simplified. Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Bradley Dice (https://github.com/bdice) - Tobias Ribizel (https://github.com/upsj) URL: rapidsai#11954
This PR addresses rapidsai#11949. We are adding methods to get the current memory usage watermarks at the whole process level and adding a "scoped" maximum, where the user can reset the initial value, run cuDF functions, and then call the API to get what happened since the reset. For the scoped maximum, the `getScopedMaximumOutstanding` could have somewhat surprising results. If the scoped maximum is reset to 0 for example, and we only see frees for allocations done before the reset, we are going to see that the scoped maximum returned is 0. This is because our memory usage is literally negative in this scenario. The APIs here assume that the caller process is using a single thread to call into the GPU (for Spark it would be 1 concurrent task). Note I assume `Rmm.initialize` has been called, otherwise this doesn't track allocations done before that. Authors: - Alessandro Bellina (https://github.com/abellina) Approvers: - Jim Brennan (https://github.com/jbrennan333) - Jason Lowe (https://github.com/jlowe) URL: rapidsai#11950
…sai#11974) Resolves rapidsai#11605 This PR: - [x] Creates docs page entries for `cudf.CategoricalDtype`, `cudf.ListDtype`, `cudf.StructDtype`, `cudf.Decimal32Dtype`, `cudf.Decimal64Dtype`, `cudf.Decimal128Dtype`. - [x] Updates docstrings in all of the public APIs of the above dtypes. - [x] Links them in the `data-types.md` page where all supported dtypes are listed as a table. Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: rapidsai#11974
…rapidsai#11980) C++17's "constexpr if" provides the same functionality as `#if` directive, as used in the nvcomp adapter. This PR replaces macros with `constexpr` variables and uses them as conditions in "constexpr if" statements. Authors: - Vukasin Milovanovic (https://github.com/vuule) Approvers: - Nghia Truong (https://github.com/ttnghia) - Tobias Ribizel (https://github.com/upsj) URL: rapidsai#11980
Adds the pool memory resource to the libcudf basic example. Also adds README.md to the strings example and makes some minor fixes to the documentation. Closes rapidsai#11870 Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Jake Hemstad (https://github.com/jrhemstad) - Elias Stehle (https://github.com/elstehle) URL: rapidsai#11966
These functions cannot throw exceptions. Resolved rapidsai#11399 Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Nghia Truong (https://github.com/ttnghia) - Bradley Dice (https://github.com/bdice) URL: rapidsai#11973
…ai#11985) Brings the docs in line with the new way of getting the default stream in libcudf. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Yunsong Wang (https://github.com/PointKernel) - Tobias Ribizel (https://github.com/upsj) - Nghia Truong (https://github.com/ttnghia)
…apidsai#11835) By passing the CUDF_VERSION compile definition only to the single source that needs it, we can remove compiler cache misses when switching between branches with different cal-ver values. Authors: - Robert Maynard (https://github.com/robertmaynard) Approvers: - Bradley Dice (https://github.com/bdice) - Karthikeyan (https://github.com/karthikeyann) URL: rapidsai#11835
This PR removes the stale issue labeler workflow Authors: - Ray Douglass (https://github.com/raydouglass) Approvers: - AJ Schmidt (https://github.com/ajschmidt8) URL: rapidsai#11995
…i#11988) Cleanup some minor issues in the root cudf CMakeLists.txt. Make the seaching for `CUDA_SANITIZER` only occur when we are building tests as that doesn't need to be done for production builds. Move the gdb pretty print script logic to a separate region to better document what it is for. Authors: - Robert Maynard (https://github.com/robertmaynard) Approvers: - Bradley Dice (https://github.com/bdice) URL: rapidsai#11988
Update the introductory documentation to take account of current best practices. In particular, use `compute` judiciously on `dask-cuDF` objects, and explain why we are doing so. Closes rapidsai#4665 Closes rapidsai#4946 Closes rapidsai#10614 Closes rapidsai#11931 Authors: - Lawrence Mitchell (https://github.com/wence-) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) - Ashwin Srinath (https://github.com/shwina) URL: rapidsai#12114
…rt. (rapidsai#12221) The conda cudatoolkit packages do not include `<cuda_profiler_api.h>`. This PR makes CUDA profiling (introduced in rapidsai#9543) optional in the JNI layer. By default, profiling will be enabled. In GitHub Actions (rapidsai#12002), profiling will be disabled. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Jim Brennan (https://github.com/jbrennan333) - Robert Maynard (https://github.com/robertmaynard) URL: rapidsai#12221
Fix in CUB DeviceSegmentedSort allows for workaround to removed. The CUB fix is applied as a patch in the libcudf build process. Reference NVIDIA/cub#594 and rapidsai#12217 Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Elias Stehle (https://github.com/elstehle) - Robert Maynard (https://github.com/robertmaynard) URL: rapidsai#12234
Authors: - Yunsong Wang (https://github.com/PointKernel) Approvers: - David Wendt (https://github.com/davidwendt) - Bradley Dice (https://github.com/bdice) URL: rapidsai#12242
Fixes sort gtests source files coded in namespace cudf::test The segmented_sort_tests.cpp also included calls to detail functions which were unnecessary. These were changed to call the external API instead. This also allowed removing the default parameters from the segmented-sort detail functions. No function or test has changed just the source code reworked per namespaces. Reference rapidsai#11734 Authors: - David Wendt (https://github.com/davidwendt) Approvers: - https://github.com/nvdbaranec - Vukasin Milovanovic (https://github.com/vuule) - Nghia Truong (https://github.com/ttnghia) URL: rapidsai#12237
Fix include line for IO Cython modules
[gpuCI] Forward-merge branch-22.12 to branch-23.02 [skip gpuci]
Fixing bug in `SpillManager.spill_device_memory` where it would spill more than the limit. # Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Ashwin Srinath (https://github.com/shwina) - GALI PREM SAGAR (https://github.com/galipremsagar) - Vyas Ramasubramani (https://github.com/vyasr) URL: rapidsai#12252
When calculating page boundaries, the current Parquet writer does not take into account storage needed per page for repetition and definition level data. As a consequence pages may sometimes exceed the specified limit, which in turn impacts the ability to compress these pages with codecs that have a maximum buffer size. This PR fixes the page size calculation to take repetition and definition levels into account. ~~This also incorporates the fragment size reduction from 5000 to 1000 that was suggested in rapidsai#12130~~ Authors: - Ed Seidl (https://github.com/etseidl) Approvers: - Vukasin Milovanovic (https://github.com/vuule) - Nghia Truong (https://github.com/ttnghia) URL: rapidsai#12182
This PR adds support for the following function in `strings_udf`: - `str.replace` Part of rapidsai#9639 Authors: - https://github.com/brandon-b-miller Approvers: - Ashwin Srinath (https://github.com/shwina) URL: rapidsai#12207
) Previously, if any of the join keys were indices, we assumed that they all were, and provided an empty set of key columns with matching names in the left and right dataframe. This does the wrong thing for mixed join keys (on a combination of index and normal columns), producing more output columns than is correct. To avoid this, only skip matching key names if they name indices. Closes rapidsai#11550. Authors: - Lawrence Mitchell (https://github.com/wence-) Approvers: - Ashwin Srinath (https://github.com/shwina) URL: rapidsai#12271
Maybe closes rapidsai#9474. The `Series._label_encoding` function is used to integer encode columns. It makes much more sense for this to be a method of `Column` rather than `Series` (the index is irrelevant). Thus, I've moved the implementation to `Column`, Authors: - Ashwin Srinath (https://github.com/shwina) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: rapidsai#12040
…ns (rapidsai#12254) Removes the default parameter declarations from `cudf::detail::sort`, `cudf::detail::sorted_order`, `cudf::detail::sort_by_key`, `cudf::detail::stable_sorted_order`, and `cudf::detail::stable_sort_by_key` as well as `cudf::detail::quantile`, `cudf::detail::quantiles` and `cudf::detail::percentile_approx` Reference comment: rapidsai#12237 (comment) Also fixes incorrect stream parameter passed within a `quantiles.cu` detail function. No function has been added, modified, or changed. Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Mark Harris (https://github.com/harrism) - Nghia Truong (https://github.com/ttnghia) - Yunsong Wang (https://github.com/PointKernel) URL: rapidsai#12254
…ai#12230) There are several overloads of these functions which work differently. They are classified into 2 groups: * `superimpose_parent_nulls(null_mask, null_count, column)`: Performs superimposing nulls from somewhere else into the input column, * `superimpose_parent_nulls(column_view/table_view)`: Perform superimposing nulls of the input column(s) into their children columns-the input root column(s) are not affected. That is confusing. They should have different names to reflect their purposes. This PR renames these groups into more meaningful names: `superimpose_nulls` and `push_down_nulls`. No implementation has been changed. This also supports rapidsai#12027. Authors: - Nghia Truong (https://github.com/ttnghia) Approvers: - David Wendt (https://github.com/davidwendt) - MithunR (https://github.com/mythrocks) URL: rapidsai#12230
Closes rapidsai#9791 Adds `read_json` tests for decimal types. Tests are equivalent to existing `read_csv` tests for decimal types. Tests invoke both JSON parsers. Authors: - Vukasin Milovanovic (https://github.com/vuule) Approvers: - Nghia Truong (https://github.com/ttnghia) - MithunR (https://github.com/mythrocks) URL: rapidsai#12229
…idsai#12238) Rework `cudf::strings::pad` and `cudf::strings::zfill` to use the internal `make_strings_children`. This simplifies the code and allows the operation to throw an error if it the output would exceed the size limit of a column. No function has been added, removed, or changed. Reference rapidsai#12167 Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Nghia Truong (https://github.com/ttnghia) - MithunR (https://github.com/mythrocks) URL: rapidsai#12238
This issue was introduced in rapidsai#12017 merged, which triggers compiler error on some systems: ``` ../tests/io/json_chunked_reader.cpp: In function 'std::vector<cudf::io::table_with_metadata> skeleton_for_parellel_chunk_reader(cudf::host_span<std::unique_ptr<cudf::io::datasource> >, const cudf::io::json_reader_options&, int32_t, rmm::cuda_stream_view, rmm::mr::device_memory_resource*)': ../tests/io/json_chunked_reader.cpp:78:19: error: loop variable '<structured bindings>' creates a copy from type 'const std::pair<int, int>' [-Werror=range-loop-construct] 78 | for (auto const [chunk_start, chunk_end] : record_ranges) { | ^~~~~~~~~~~~~~~~~~~~~~~~ ../tests/io/json_chunked_reader.cpp:78:19: note: use reference type to prevent copying 78 | for (auto const [chunk_start, chunk_end] : record_ranges) { | ^~~~~~~~~~~~~~~~~~~~~~~~ | & ``` Fixing it is just by following the compiler recommendation: "use reference type to prevent copying". Authors: - Nghia Truong (https://github.com/ttnghia) Approvers: - David Wendt (https://github.com/davidwendt) - Yunsong Wang (https://github.com/PointKernel) URL: rapidsai#12280
Adds `cudf::strings::reverse` function. This is to support NVIDIA/spark-rapids#6885 Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Robert Maynard (https://github.com/robertmaynard) - Nghia Truong (https://github.com/ttnghia) - Christopher Harris (https://github.com/cwharris) - Jake Awe (https://github.com/AyodeAwe) - Bradley Dice (https://github.com/bdice) URL: rapidsai#12227
…vice_vector` (rapidsai#12251) Enables experimentation related to issue rapidsai#12186 Frequent allocation of pinned memory becomes costly in multi-GPU environment, where all devices synchronize and allocation creates huge bubbles in GPU utilization. This PR adds environment variable `LIBCUDF_PREFER_PAGEABLE_MEMORY` that controls whether `hostdevice_vector` uses pinned memory for the host side. `hostdevice_vector` makes up most of pinned memory allocations in cuIO, so this environment variable should greatly impact the execution when using multiple GPUs. Authors: - Vukasin Milovanovic (https://github.com/vuule) Approvers: - Nghia Truong (https://github.com/ttnghia) - Mike Wilson (https://github.com/hyperbolic2346) - Robert Maynard (https://github.com/robertmaynard) URL: rapidsai#12251
This PR removes various pieces of deprecated functionality. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) - GALI PREM SAGAR (https://github.com/galipremsagar) URL: rapidsai#12281
Motivated by rapidsai#10830 (comment), this PR attempts to consolidate some repetitive aspects of dask-cudf's groupby code with `_make_groupby_agg_call`, which replaces all `groupby_agg` calls made in groupby.py, which takes as input the few things that vary between calls. Note that while this doesn't depend on rapidsai#10830, it will be much easier to review once that is merged in, as I have based my work off the initial consolidation efforts that were made there. cc @bdice Authors: - Charles Blackmon-Luca (https://github.com/charlesbluca) Approvers: - Richard (Rick) Zamora (https://github.com/rjzamora) URL: rapidsai#10835
This PR adds GitHub Actions workflows to `cudf`. ### Task list Coverage required for this PR: - [x] C++ tests - [x] Python tests for cudf (first success: c3bdbce) - [x] Python tests for custreamz, cudf_kafka - [x] Java tests - [x] Check dask configuration - [x] `strings_udf` - [x] Benchmarks - [x] Notebooks - [x] Codecov - [x] Enable style checks script (`ci/check_style.sh`) - [x] Consolidate [existing checks job](https://github.com/rapidsai/cudf/blob/c37d33c86f86bac13cb78b1e4643b05be81115df/.github/workflows/dependency-files.yml#L7-L12) into the `pr.yaml` workflow in this PR. Future PRs: - [ ] Build Metrics Report - [ ] `libidentify_stream_usage` - [ ] JUnit output annotations? https://github.com/marketplace/actions/junit-report-action - [ ] Reduce exit code handling verbosity (use global variable with `trap` function) - [ ] Setup `rapids-dependency-file-generator` pre-commit hook (also setup for others repos: `rmm`, `rapids-cmake`, `cuspatial`, etc.) - [ ] Docs builds Authors: - AJ Schmidt (https://github.com/ajschmidt8) - Bradley Dice (https://github.com/bdice) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) - Nghia Truong (https://github.com/ttnghia) - Jordan Jacobelli (https://github.com/Ethyling) - Bradley Dice (https://github.com/bdice)
This change enables libcudf APIs throwing exceptions other than `cudf::logic_error`. This code was adapted from rmm. This contributes to rapidsai#10200. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Yunsong Wang (https://github.com/PointKernel) - Lawrence Mitchell (https://github.com/wence-) URL: rapidsai#12078
Fixes reduction gtests source files coded in namespace `cudf::test` No function or test has changed just the source code reworked per namespaces. Fixing this ahead of any changes for rapidsai#10432 Reference rapidsai#11734 Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Nghia Truong (https://github.com/ttnghia) - Bradley Dice (https://github.com/bdice) - MithunR (https://github.com/mythrocks) URL: rapidsai#12257
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR:
function.cuto make compilation succeedgroupby_function.pyto make importing possible