Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use cudf_test temp_directory class for nvtext::subword_tokenize gbenchmark #14558

Merged
merged 8 commits into from
Dec 6, 2023
Merged

Use cudf_test temp_directory class for nvtext::subword_tokenize gbenchmark #14558

Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
d0d234c
Use cudf_test temp_directory class for nvtext::subword_tokenize gbenc…
davidwendt Dec 4, 2023
f8fb58d
pin triage labeler to tag v4
davidwendt Dec 4, 2023
1103711
Merge branch 'branch-24.02' into fix-subword-gbench-temp-dir
davidwendt Dec 4, 2023
4308d09
fix labeler.yml
davidwendt Dec 4, 2023
634374a
undo labeler changes
davidwendt Dec 4, 2023
9bd5cd6
Merge branch 'branch-24.02' into fix-subword-gbench-temp-dir
davidwendt Dec 4, 2023
1358ed7
Merge branch 'branch-24.02' into fix-subword-gbench-temp-dir
davidwendt Dec 6, 2023
958fa00
move global var to function static init var
davidwendt Dec 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions cpp/benchmarks/text/subword.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,19 +18,22 @@
#include <benchmarks/synchronization/synchronization.hpp>

#include <cudf_test/column_wrapper.hpp>
#include <cudf_test/file_utilities.hpp>

#include <cudf/strings/strings_column_view.hpp>

#include <nvtext/subword_tokenize.hpp>

#include <filesystem>
#include <fstream>
#include <iostream>
#include <vector>

temp_directory const subword_tmpdir{"cudf_gbench"};
davidwendt marked this conversation as resolved.
Show resolved Hide resolved

static std::string create_hash_vocab_file()
{
std::string dir_template{std::filesystem::temp_directory_path().string()};
if (char const* env_p = std::getenv("WORKSPACE")) dir_template = env_p;
auto dir_template = subword_tmpdir.path();
std::string hash_file = dir_template + "/hash_vocab.txt";
// create a fake hashed vocab text file for this test
// this only works with words in the strings in the benchmark code below
Expand All @@ -57,7 +60,7 @@ static void BM_subword_tokenizer(benchmark::State& state)
auto const nrows = static_cast<cudf::size_type>(state.range(0));
std::vector<char const*> h_strings(nrows, "This is a test ");
cudf::test::strings_column_wrapper strings(h_strings.begin(), h_strings.end());
std::string hash_file = create_hash_vocab_file();
static std::string hash_file = create_hash_vocab_file();
std::vector<uint32_t> offsets{14};
uint32_t max_sequence_length = 64;
uint32_t stride = 48;
Expand Down