Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remote IO: http support #464

Merged
merged 73 commits into from
Oct 8, 2024
Merged
Changes from 1 commit
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
a01dc53
Impl. basic remote handle
madsbk Sep 17, 2024
a6467d5
RemoteHandle::read()
madsbk Sep 18, 2024
127b782
ci: rangehttpserver
madsbk Sep 18, 2024
089397f
libcurl>=7.75.0 # Need <https://curl.se/libcurl/c/CURLOPT_AWS_SIGV4.…
madsbk Sep 18, 2024
5f1fca0
cmake
madsbk Sep 18, 2024
eaacacf
fix ci rangehttpserver
madsbk Sep 18, 2024
59595a8
cleanup
madsbk Sep 18, 2024
f803220
get_file_size(): check -1
madsbk Sep 18, 2024
dd06044
Read(): check file size
madsbk Sep 18, 2024
06dcb35
LibCurl get/set: use lock
madsbk Sep 19, 2024
7501600
test_no_range_support
madsbk Sep 19, 2024
7e6315e
bump libcurl>=7.87.0
madsbk Sep 19, 2024
4f9ab63
ci: more libcurl>=7.87.0
madsbk Sep 19, 2024
a20774c
cmake: CURL 7.87.0
madsbk Sep 19, 2024
43421ed
ci_debug
madsbk Sep 19, 2024
7df0e9f
CURLOPT_FOLLOWLOCATION
madsbk Sep 19, 2024
40b4be3
overflow_error
madsbk Sep 19, 2024
943e30b
RemoteEndpoint
madsbk Sep 20, 2024
e8f6b12
cpp_RemoteEndpoint
madsbk Sep 20, 2024
5f73677
from_http_url
madsbk Sep 20, 2024
5dbe2e5
Update cpp/include/kvikio/remote_handle.hpp
madsbk Sep 23, 2024
f9873c0
setopt(): use CURLoption
madsbk Sep 23, 2024
4896452
comment: need CURL_WRITEFUNC_ERROR
madsbk Sep 23, 2024
e53b761
LibCurl: use vector
madsbk Sep 23, 2024
4ba4eb4
UniqueHandlePtr
madsbk Sep 23, 2024
bf716be
get_handle() / retain_handle()
madsbk Sep 23, 2024
83774e3
clean up
madsbk Sep 23, 2024
e884d3c
doc
madsbk Sep 23, 2024
6637089
utils: LocalHttpServer
madsbk Sep 24, 2024
a0098e9
http benchmark
madsbk Sep 24, 2024
0e6dc7a
test_http_io
madsbk Sep 24, 2024
f0b2f9f
example
madsbk Sep 25, 2024
8e7e2cf
cleanup
madsbk Sep 26, 2024
fa67219
cmake: rapids_cpm_find CURL
madsbk Sep 29, 2024
9906a14
cleanup
madsbk Sep 30, 2024
d38121b
doc
madsbk Sep 30, 2024
bdc1e22
cleanup
madsbk Sep 30, 2024
51057f6
Merge branch 'branch-24.10' of https://github.com/rapidsai/kvikio int…
madsbk Sep 30, 2024
383fa4c
clean up
madsbk Sep 30, 2024
0055373
docs
madsbk Sep 30, 2024
628e525
doc
madsbk Oct 1, 2024
8726f87
fix virtual dtor
madsbk Oct 1, 2024
3222c92
Merge branch 'branch-24.12' of https://github.com/rapidsai/kvikio int…
madsbk Oct 1, 2024
c94121d
fix benckmark
madsbk Oct 1, 2024
0c764ba
clean up
madsbk Oct 1, 2024
91d953b
libcurl.hpp
madsbk Oct 1, 2024
94e47ee
rename: RemoteFile.open_http
madsbk Oct 1, 2024
fcb9358
doc
madsbk Oct 1, 2024
e8018d1
cleanup
madsbk Oct 1, 2024
bb737b9
PushAndPopContext outside callback_device_memory()
madsbk Oct 1, 2024
f3f46f8
Merge branch 'branch-24.12' of https://github.com/rapidsai/kvikio int…
madsbk Oct 2, 2024
b8a9c0a
more nvtx
madsbk Oct 2, 2024
9cffe1c
Apply suggestions from code review
madsbk Oct 3, 2024
792f43f
Merge branch 'branch-24.12' of https://github.com/rapidsai/kvikio int…
madsbk Oct 3, 2024
1257a9f
std::function<decltype(curl_easy_cleanup)>
madsbk Oct 3, 2024
1006b24
getinfo: take output pointer
madsbk Oct 3, 2024
9ba9752
STATUS
madsbk Oct 3, 2024
128e3dd
close
madsbk Oct 3, 2024
0df1726
test_http_io: check is_remote_file_available
madsbk Oct 3, 2024
96af5ac
std::ptrdiff_t
madsbk Oct 3, 2024
feccdd9
Apply suggestions from code review
madsbk Oct 3, 2024
119c025
Merge branch 'remote-io' of github.com:madsbk/kvikio into remote-io
madsbk Oct 3, 2024
7dcc8f3
use conda's rangehttpserver
madsbk Oct 3, 2024
60f7331
doc
madsbk Oct 3, 2024
13e0b7a
CallbackContext ctor
madsbk Oct 3, 2024
474e694
host: libcurl==7.87.0
madsbk Oct 3, 2024
9022b18
run: remove libcurl>=7.87.0
madsbk Oct 3, 2024
8bbb33b
curl-7_87_0
madsbk Oct 3, 2024
46b9300
Merge branch 'branch-24.12' of https://github.com/rapidsai/kvikio int…
madsbk Oct 3, 2024
1176b67
kvikio/meta.yaml: libcurl==7.87.0
madsbk Oct 3, 2024
4ed88a2
Merge branch 'branch-24.12' of https://github.com/rapidsai/kvikio int…
madsbk Oct 3, 2024
cc7b291
Merge branch 'branch-24.12' of https://github.com/rapidsai/kvikio int…
madsbk Oct 7, 2024
58e3054
cmake clean up
madsbk Oct 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
clean up
madsbk committed Sep 30, 2024

Verified

This commit was signed with the committer’s verified signature.
madsbk Mads R. B. Kristensen
commit 383fa4c67cb21c62fd58c68fe5ca05769e95f683
5 changes: 4 additions & 1 deletion python/kvikio/kvikio/_lib/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -21,7 +21,10 @@ if(TARGET CURL::libcurl)
message(STATUS "Building remote_handle.pyx (libcurl found)")
list(APPEND cython_modules remote_handle.pyx)
else()
message(WARNING "Skipping remote_handle.pyx (libcurl not found or disabled)")
message(
WARNING
"Skipping remote_handle.pyx (please set KvikIO_REMOTE_SUPPORT=ON for remote file support)"
)
madsbk marked this conversation as resolved.
Show resolved Hide resolved
endif()

rapids_cython_create_modules(
3 changes: 1 addition & 2 deletions python/kvikio/kvikio/utils.py
Original file line number Diff line number Diff line change
@@ -6,6 +6,7 @@
import pathlib
import threading
import time
from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer


class LocalHttpServer:
@@ -18,8 +19,6 @@ def _server(
range_support: bool,
max_lifetime: int,
):
from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer

if range_support:
from RangeHTTPServer import RangeRequestHandler