Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extract debug_utilities.hpp/cu from column_utilities.hpp/cu #13720

Merged
merged 32 commits into from
Oct 23, 2023
Merged
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
5e69cb2
Add new file in cmake
ttnghia Jul 18, 2023
c22317d
Extract print code
ttnghia Jul 18, 2023
edb5e4b
Replace stream
ttnghia Jul 18, 2023
c6bcdd5
WIP
ttnghia Jul 18, 2023
7f44d5e
Add comments
ttnghia Jul 18, 2023
5698705
Cleanup headers
ttnghia Jul 18, 2023
70aaeb9
Remove declaration
ttnghia Jul 18, 2023
1e8a158
Cleanup
ttnghia Jul 18, 2023
422b793
Cleanup
ttnghia Jul 18, 2023
45824ba
Rename namespace
ttnghia Jul 18, 2023
cc57d6d
Add comment
ttnghia Jul 18, 2023
01a5f0e
Remove parameter
ttnghia Jul 18, 2023
4cf2007
Add comment about usage
ttnghia Jul 19, 2023
d7bff67
Update meta.yaml
ttnghia Jul 19, 2023
851300c
Update meta.yaml
ttnghia Jul 19, 2023
ee09eb6
Change comment
ttnghia Jul 19, 2023
761a842
Fix struct test
ttnghia Jul 19, 2023
e7df8fb
Control adding source file by cmake
ttnghia Jul 19, 2023
58447f9
Revert "Control adding source file by cmake"
ttnghia Jul 19, 2023
6485ffc
Merge branch 'branch-23.08' into debug_utils
ttnghia Jul 19, 2023
1ed6980
Fix column_utility_tests.cpp
ttnghia Jul 19, 2023
25ccbac
Merge branch 'branch-23.10' into debug_utils
ttnghia Aug 11, 2023
ecebebd
Merge branch 'branch-23.10' into debug_utils
ttnghia Aug 26, 2023
b37b373
Merge branch 'branch-23.12' into debug_utils
ttnghia Oct 16, 2023
abbe5cf
Merge branch 'branch-23.12' into debug_utils
ttnghia Oct 16, 2023
df6e608
Merge branch 'branch-23.12' into debug_utils
ttnghia Oct 17, 2023
3ae27f3
Merge branch 'branch-23.12' into debug_utils
ttnghia Oct 18, 2023
ecbeef9
Use macro to detect gtest
ttnghia Oct 19, 2023
b6e2782
Merge branch 'branch-23.12' into debug_utils
ttnghia Oct 19, 2023
048d8cb
Remove header check
ttnghia Oct 20, 2023
cb6b874
Merge branch 'branch-23.12' into debug_utils
ttnghia Oct 20, 2023
ae2dd49
Extract debug tests
ttnghia Oct 20, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cpp/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -835,6 +835,7 @@ if(CUDF_BUILD_TESTUTIL)
tests/io/metadata_utilities.cpp
tests/utilities/base_fixture.cpp
tests/utilities/column_utilities.cu
tests/utilities/debug_utilities.cu
tests/utilities/table_utilities.cu
tests/utilities/tdigest_utilities.cu
)
Expand Down
38 changes: 5 additions & 33 deletions cpp/include/cudf_test/column_utilities.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,11 @@

#pragma once

//
// Do NOT include any header that directly or indirectly depends on gtest.
// It is to make sure this file can be used and compiled outside of the test framework.
//

#include <cudf/column/column.hpp>
#include <cudf/column/column_view.hpp>
#include <cudf/detail/utilities/vector_factories.hpp>
Expand Down Expand Up @@ -140,39 +145,6 @@ void expect_equal_buffers(void const* lhs, void const* rhs, std::size_t size_byt
*/
void expect_column_empty(cudf::column_view const& col);

/**
* @brief Formats a column view as a string
*
* @param col The column view
* @param delimiter The delimiter to put between strings
*/
std::string to_string(cudf::column_view const& col, std::string const& delimiter);

/**
* @brief Formats a null mask as a string
*
* @param null_mask The null mask buffer
* @param null_mask_size Size of the null mask (in rows)
*/
std::string to_string(std::vector<bitmask_type> const& null_mask, size_type null_mask_size);

/**
* @brief Convert column values to a host vector of strings
*
* @param col The column view
*/
std::vector<std::string> to_strings(cudf::column_view const& col);

/**
* @brief Print a column view to an ostream
*
* @param os The output stream
* @param col The column view
*/
void print(cudf::column_view const& col,
std::ostream& os = std::cout,
std::string const& delimiter = ",");

/**
* @brief Copy the null bitmask from a column view to a host vector
*
Expand Down
56 changes: 56 additions & 0 deletions cpp/include/cudf_test/debug_utilities.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
/*
* Copyright (c) 2020-2023, NVIDIA CORPORATION.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

#pragma once

//
// Do NOT include any header that directly or indirectly depends on gtest.
// It is to make sure this file can be used and compiled outside of the test framework.
//
// In order to use the APIs listed below outside of the test framework, simply include this header
// and add the corresponding source file `test/utilities/debug_utilities.cu` into the CMake
// target build.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to include the file as a ifdef in the CMakeLists.txt so it could be compiled in with a cmake parameter?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that is possible. I thought about that at the beginning but gave up as this is "just one file". Let me try again.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The source file tests/utilities/debug_utilities.cu is already part of the cudftestutil. We shouldn't be building it part of two different DSO as we could run into duplicate symbol issues from calling tests.

All the libcudf tests and benchmarks have access to this function since they link to to cudftestutil. If you are debugging something external, you can gain access to this feature by adding cudftestutil to your link line and not be required to recompile libcudf.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I only see your comment after pushed my cmake changes 😄 . Anyway, I tried to avoid duplicate symbols using if/else. Please take a look.

Copy link
Contributor Author

@ttnghia ttnghia Jul 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition, I don't know if it is possible to add cudftestutil into libcudf.so, not the exec C++ application, since I need to output cudf::test::print into libcudf.so to be loaded in Spark.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spark builds libcudf statically right and embeds some extra code into a new libcudf.so no reason that can't also embed cudftestuil.

This approach is the wrong way to accomplish our goals.

Copy link
Contributor Author

@ttnghia ttnghia Jul 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, so I reverted the approach with cmake changes.
Personally I had (and still have, while compiling spark-rapids-jni) so many troubles with linking cudftestutil (due to my limited cmake experience) so I still prefer manually add the source file. In addition, extracting the print module should also make the code base cleaner.

Copy link
Contributor Author

@ttnghia ttnghia Oct 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fact, adding cudftestutil is not very straightforward, and I don't know how to fix this:

[INFO]      [exec] /opt/rh/devtoolset-11/root/usr/libexec/gcc/x86_64-redhat-linux/11/ld: 
/home/nghiat/Devel/jni/target/libcudf-install/lib64/libgtest.a(gtest-all.cc.o): relocation 
R_X86_64_32S against symbol `_ZTVN7testing8internal17TestEventRepeaterE' can not be 
used when making a shared object; recompile with -fPIC

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to enable position independent code ( the -fPIC flag )for the libgtest to resolve those link error. Can you please open this as an issue on rapids-cmake and I can fix the issue.

//

#include <cudf/column/column_view.hpp>
#include <cudf/null_mask.hpp>

namespace cudf::test {

/**
* @brief Formats a column view as a string
*
* @param col The input column view
* @param delimiter The delimiter to put between strings
*/
std::string to_string(cudf::column_view const& col, std::string const& delimiter);

/**
* @brief Convert column values to a host vector of strings
*
* @param col The input column view
*/
std::vector<std::string> to_strings(cudf::column_view const& col);

/**
* @brief Print a column view to an ostream
*
* @param col The input column view
* @param os The output stream
*/
void print(cudf::column_view const& col, std::ostream& os = std::cout);

} // namespace cudf::test
85 changes: 0 additions & 85 deletions cpp/include/cudf_test/detail/column_utilities.hpp

This file was deleted.

1 change: 1 addition & 0 deletions cpp/tests/groupby/structs_tests.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@

#include <cudf_test/base_fixture.hpp>
#include <cudf_test/column_wrapper.hpp>
#include <cudf_test/debug_utilities.hpp>
davidwendt marked this conversation as resolved.
Show resolved Hide resolved
#include <cudf_test/iterator_utilities.hpp>
#include <cudf_test/type_lists.hpp>

Expand Down
Loading
Loading