Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Add Nvcomp python bindings to kvikio. #24

Merged
merged 44 commits into from
Mar 29, 2022
Merged
Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
ff4d34d
Create first set of bindings and build command, need to get dependenc…
thomcom Mar 9, 2022
1e2759a
Include cuda headers.
thomcom Mar 9, 2022
6e771b0
Add nvcomp to python module. Start cleaning up the cython for delivery.
thomcom Mar 10, 2022
d55bdc2
Add Cascade benchmarks, batched snappy test, and update the library a…
thomcom Mar 10, 2022
c4697df
Got all the bindings working.
thomcom Mar 11, 2022
51198e4
What I meant was that Cascade Bindings are working now, which, they a…
thomcom Mar 11, 2022
bc82479
Cascaded decompress wrapped and working.
thomcom Mar 11, 2022
cf6e500
Add lz4 bindings and cascade test.
thomcom Mar 11, 2022
fc1ba4b
Working basic round trip tests for LZ4 and Cascaded.
thomcom Mar 11, 2022
63578c7
Parametrize and pass tests.
thomcom Mar 14, 2022
eaa2806
Fix styles.
thomcom Mar 15, 2022
31b6b34
Missed isort of nvcomp.pyx
thomcom Mar 15, 2022
b4112ce
Permuting a file for rerun tests?
thomcom Mar 15, 2022
38a49df
Try installing nvcomp.
thomcom Mar 16, 2022
b3174b7
Style
thomcom Mar 17, 2022
c248732
Try forcing installation of nvcomp for now.
thomcom Mar 17, 2022
17979da
Merge branch 'branch-22.04' into fea-add-nvcomp
thomcom Mar 18, 2022
3c0364c
Resolve conflict in setup.py.
thomcom Mar 21, 2022
147697c
Take upstreams branch-22.04, not mine.
thomcom Mar 21, 2022
5cf3746
Thanks Vyas.
thomcom Mar 21, 2022
3487026
NVcomp dependency linking.
thomcom Mar 21, 2022
474c95b
Final adjustments to cmake for now.
thomcom Mar 21, 2022
f8a7365
Fix isort
thomcom Mar 22, 2022
5b800e5
Remove non-test
thomcom Mar 22, 2022
8a1fc09
Add mypy types for nvcomp.py
thomcom Mar 22, 2022
776535d
Styling.
thomcom Mar 22, 2022
b3d1742
Fix a bad import.
thomcom Mar 22, 2022
d9e748b
Update python/kvikio/__init__.py
thomcom Mar 23, 2022
9b15f51
Update python/kvikio/nvcomp.py
thomcom Mar 23, 2022
863d408
Update python/tests/test_nvcomp.py
thomcom Mar 23, 2022
ef98409
Update python/tests/test_nvcomp.py
thomcom Mar 23, 2022
67fd158
Use Array and fix up some tests.
thomcom Mar 23, 2022
cb1b1c7
Fix conflict.
thomcom Mar 23, 2022
cd713b2
Added final-ish notebook version.
thomcom Mar 23, 2022
ffffa3a
This makes the nvcomp module visible within the kvikio module. I thin…
thomcom Mar 23, 2022
ca83525
Update python/kvikio/__init__.py
thomcom Mar 24, 2022
c76049d
Add docstrings and get rid of BatchedSnappy compressor that doesn't w…
thomcom Mar 24, 2022
aea11e9
nvcomp Cascade Benchmarks.ipynb -> nvcomp_cascade_benchmarks.ipynb
madsbk Mar 25, 2022
398f3b4
Take care of all of Vyas's questions except for aliases.
thomcom Mar 25, 2022
fe8ec24
Remove that notebook. The other is the same, but better.
thomcom Mar 25, 2022
f32b7c8
Change pynvcomp cython naming scheme to reflext current kvikio standa…
thomcom Mar 28, 2022
88e9f79
Merge branch 'branch-22.04' of https://github.com/rapidsai/kvikio int…
madsbk Mar 29, 2022
c5ac862
typos
madsbk Mar 29, 2022
4a1daf4
fixing format-all.py
madsbk Mar 29, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4,019 changes: 4,019 additions & 0 deletions notebooks/nvcomp_vs_zarr_lz4.ipynb

Large diffs are not rendered by default.

5 changes: 4 additions & 1 deletion python/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ project(
# various linking options for the python library is hardcoded to build with C, so until
# that is fixed we need to keep C.
C
CXX)
CXX
CUDA)

option(FIND_KVIKIO_CPP "Search for existing KVIKIO C++ installations before defaulting to local files"
OFF)
Expand All @@ -45,6 +46,8 @@ else()
set(KVIKIO_FOUND OFF)
endif()

find_package(CUDAToolkit REQUIRED)

if(NOT KVIKIO_FOUND)
add_subdirectory(../cpp kvikio-cpp)
endif()
Expand Down
1 change: 1 addition & 0 deletions python/cmake/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,5 @@
# the License.
# =============================================================================

include(thirdparty/get_nvcomp.cmake)
include(kvikio_python_helpers.cmake)
40 changes: 40 additions & 0 deletions python/cmake/thirdparty/get_nvcomp.cmake
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# =============================================================================
# Copyright (c) 2022, NVIDIA CORPORATION.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
# in compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under the License
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing permissions and limitations under
# the License.
# =============================================================================

# This function finds nvcomp and sets any additional necessary environment variables.
function(find_and_configure_nvcomp VERSION)

# Find or install nvcomp
rapids_cpm_find(
nvcomp ${VERSION}
GLOBAL_TARGETS nvcomp::nvcomp
CPM_ARGS GITHUB_REPOSITORY NVIDIA/nvcomp
GIT_TAG c435afaf4ba8a8d12f379d688effcb185886cec1
OPTIONS "BUILD_STATIC ON" "BUILD_TESTS OFF" "BUILD_BENCHMARKS OFF" "BUILD_EXAMPLES OFF"
)

if(NOT TARGET nvcomp::nvcomp)
add_library(nvcomp::nvcomp ALIAS nvcomp)
endif()

# Per-thread default stream
if(TARGET nvcomp AND PER_THREAD_DEFAULT_STREAM)
target_compile_definitions(nvcomp PRIVATE CUDA_API_PER_THREAD_DEFAULT_STREAM)
endif()

endfunction()

set(CUDF_MIN_VERSION_nvCOMP 2.1.0)

find_and_configure_nvcomp(${CUDF_MIN_VERSION_nvCOMP})
1 change: 0 additions & 1 deletion python/kvikio/__init__.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
# Copyright (c) 2021-2022, NVIDIA CORPORATION. All rights reserved.
# See file LICENSE for terms.


from . import _version
from ._lib import libkvikio # type: ignore

Expand Down
6 changes: 4 additions & 2 deletions python/kvikio/_lib/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@
# =============================================================================

# Set the list of Cython files to build
set(cython_modules arr libkvikio)

set(cython_modules arr libnvcomp libkvikio)
# Build all of the Cython targets
add_cython_modules("${cython_modules}")

target_link_libraries(libnvcomp nvcomp)
Loading