Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate Parquet reader to pylibcudf #16078

Merged
merged 28 commits into from
Jul 20, 2024
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
c256f1e
Migrate expressions to pylibcudf
lithomas1 Jun 18, 2024
a57b132
fix typo in docs
lithomas1 Jun 18, 2024
5bc6917
Merge branch 'branch-24.08' of github.com:rapidsai/cudf into pylibcud…
lithomas1 Jun 24, 2024
7c0d72d
add to init file
lithomas1 Jun 24, 2024
fc2ce94
Add support to ArrowDataSource in SourceInfo
lithomas1 Jun 17, 2024
a26a0dd
more clean
lithomas1 Jun 17, 2024
89fb0dc
include cudf kafka change
lithomas1 Jun 17, 2024
2810558
remove orc changes
lithomas1 Jun 17, 2024
c805de6
Update kafka.pxd
lithomas1 Jun 17, 2024
57de928
Migrate expressions to pylibcudf
lithomas1 Jun 18, 2024
d0f090a
Migrate Parquet reader to pylibcudf
lithomas1 Jun 25, 2024
21b6b0a
Merge branch 'branch-24.08' of github.com:rapidsai/cudf into pylibcud…
lithomas1 Jul 12, 2024
378f288
chunked parquet reader fixes
lithomas1 Jul 12, 2024
a96d239
Merge branch 'branch-24.08' of github.com:rapidsai/cudf into pylibcud…
lithomas1 Jul 15, 2024
68b1385
finishup tests
lithomas1 Jul 15, 2024
731f3f3
fix json tests
lithomas1 Jul 16, 2024
5986d0b
Merge branch 'branch-24.08' of github.com:rapidsai/cudf into pylibcud…
lithomas1 Jul 16, 2024
d0f4ecf
bad merge
lithomas1 Jul 16, 2024
dca6d0d
extraneous change
lithomas1 Jul 16, 2024
0e60841
adjust to literal changes
lithomas1 Jul 17, 2024
43790a1
Merge branch 'branch-24.08' of github.com:rapidsai/cudf into pylibcud…
lithomas1 Jul 17, 2024
179df87
address some comments
lithomas1 Jul 18, 2024
45feb0f
fix docs
lithomas1 Jul 18, 2024
01a0fb7
Merge branch 'branch-24.08' of github.com:rapidsai/cudf into pylibcud…
lithomas1 Jul 18, 2024
fd5816e
updates
lithomas1 Jul 19, 2024
3987d04
Merge branch 'branch-24.08' of github.com:rapidsai/cudf into pylibcud…
lithomas1 Jul 19, 2024
8f9ff31
Merge branch 'branch-24.08' into pylibcudf-io-parquet
vyasr Jul 19, 2024
ed2ee68
Merge branch 'branch-24.08' of github.com:rapidsai/cudf into pylibcud…
lithomas1 Jul 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/cudf/source/user_guide/api_docs/pylibcudf/datetime.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
=======
copying
=======
========
datetime
========

.. automodule:: cudf._lib.pylibcudf.datetime
:members:
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
===========
expressions
===========

.. automodule:: cudf._lib.pylibcudf.expressions
:members:
1 change: 1 addition & 0 deletions docs/cudf/source/user_guide/api_docs/pylibcudf/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ This page provides API documentation for pylibcudf.
concatenate
copying
datetime
expressions
filling
gpumemoryview
groupby
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,4 @@ I/O Functions

avro
json
parquet
6 changes: 6 additions & 0 deletions docs/cudf/source/user_guide/api_docs/pylibcudf/io/parquet.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
=======
Parquet
=======

.. automodule:: cudf._lib.pylibcudf.io.parquet
:members:
1 change: 0 additions & 1 deletion python/cudf/cudf/_lib/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ set(cython_sources
copying.pyx
csv.pyx
datetime.pyx
expressions.pyx
filling.pyx
groupby.pyx
hash.pyx
Expand Down
3 changes: 1 addition & 2 deletions python/cudf/cudf/_lib/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (c) 2020-2023, NVIDIA CORPORATION.
# Copyright (c) 2020-2024, NVIDIA CORPORATION.
import numpy as np

from . import (
Expand All @@ -8,7 +8,6 @@
copying,
csv,
datetime,
expressions,
filling,
groupby,
hash,
Expand Down
156 changes: 0 additions & 156 deletions python/cudf/cudf/_lib/expressions.pyx

This file was deleted.

4 changes: 4 additions & 0 deletions python/cudf/cudf/_lib/io/utils.pxd
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@ cdef add_df_col_struct_names(
df,
child_names_dict
)
cdef update_col_struct_field_names(
Column col,
child_names
)
lithomas1 marked this conversation as resolved.
Show resolved Hide resolved
cdef update_struct_field_names(
table,
vector[column_name_info]& schema_info)
Expand Down
1 change: 0 additions & 1 deletion python/cudf/cudf/_lib/io/utils.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,6 @@ cdef cppclass iobase_data_sink(data_sink):
size_t bytes_written() with gil:
return buf.tell()


cdef add_df_col_struct_names(df, child_names_dict):
for name, child_names in child_names_dict.items():
col = df._data[name]
Expand Down
Loading
Loading