Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[post build lint] Check for absolute paths #172

Merged
merged 68 commits into from
Jan 20, 2023
Merged
Changes from 55 commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
863231d
[post build lint] Check for absolute paths
autoantwort Aug 25, 2021
15778b6
Apply suggestions from code review
autoantwort Oct 7, 2021
9212d1b
Fix build and logic
autoantwort Oct 7, 2021
326672e
Merge branch 'main' into check-pkg-config-files
autoantwort Oct 12, 2021
cef8cb0
nicole's CRs
strega-nil-ms Oct 18, 2021
761dc17
fix build
autoantwort Oct 19, 2021
7fd595a
Merge branch 'main' into check-pkg-config-files
autoantwort Oct 29, 2021
7e9fdd0
Ignore dot in extension
autoantwort Oct 29, 2021
f58d9ae
Merge branch 'main' into check-pkg-config-files
autoantwort Nov 9, 2021
6840f39
fix build
autoantwort Nov 9, 2021
d7f0f47
Merge branch 'main' into check-pkg-config-files
autoantwort Nov 20, 2021
42c62d7
fix build
autoantwort Nov 20, 2021
80704ee
don't check .la files
autoantwort Nov 21, 2021
2c1fb86
also check for the buildtree dir
autoantwort Nov 23, 2021
cd19401
Merge branch 'main' into check-pkg-config-files
autoantwort Nov 27, 2021
1fd5a30
don't check .yaml files
autoantwort Nov 28, 2021
6154d36
ignore comments in files
autoantwort Dec 1, 2021
47787c5
Merge branch 'main' into check-pkg-config-files
autoantwort Dec 21, 2021
86072b7
adapt code
autoantwort Dec 21, 2021
ea1e189
fix comment detection
autoantwort Dec 24, 2021
10049d7
Update include/vcpkg/base/util.h
autoantwort Dec 30, 2021
03d24cd
Change order of conditions
autoantwort Dec 30, 2021
b51e0b1
Update src/vcpkg/postbuildlint.cpp
autoantwort Dec 30, 2021
5d19692
Use localized message
autoantwort Dec 31, 2021
b48fd0f
Merge branch 'main' into check-pkg-config-files
autoantwort Dec 31, 2021
81c991d
Update message map
autoantwort Dec 31, 2021
aa5264b
Merge branch 'main' into check-pkg-config-files
autoantwort Feb 21, 2022
8b2cc5c
Merge branch 'main' into check-pkg-config-files
autoantwort Mar 6, 2022
d58b64e
Update messages.en.json
autoantwort Mar 8, 2022
6db65d5
Merge branch 'main' into check-pkg-config-files
autoantwort Mar 24, 2022
04e77f5
Merge branch 'main' into check-pkg-config-files
autoantwort Apr 8, 2022
dce3e55
Merge branch 'main' into check-pkg-config-files
autoantwort Apr 18, 2022
9df3a83
Merge branch 'main' into check-pkg-config-files
autoantwort May 12, 2022
dec596d
Merge branch 'main' into check-pkg-config-files
autoantwort May 19, 2022
20a0747
Merge remote-tracking branch 'origin/main' into check-pkg-config-files
BillyONeal May 20, 2022
cc4cbc8
Merge branch 'main' into check-pkg-config-files
autoantwort Jun 25, 2022
7b535b4
No line endings in message
autoantwort Jun 25, 2022
dfa58d3
Merge remote-tracking branch 'origin/main' into check-pkg-config-files
BillyONeal Aug 11, 2022
3fd3ea2
Merge remote-tracking branch 'origin/main' into check-pkg-config-files
BillyONeal Aug 18, 2022
024b814
Apply the if/return transform requested by https://github.com/microso…
BillyONeal Aug 18, 2022
a1b3aa1
Merge remote-tracking branch 'origin/main' into check-pkg-config-files
BillyONeal Aug 22, 2022
6421db4
Merge branch 'main' into check-pkg-config-files
autoantwort Sep 2, 2022
bdcb84c
Merge branch 'main' into check-pkg-config-files
autoantwort Sep 26, 2022
2da94b4
Merge remote-tracking branch 'origin/main' into check-pkg-config-files
BillyONeal Sep 30, 2022
535e47a
Merge branch 'main' into check-pkg-config-files
autoantwort Oct 7, 2022
72476ba
Add policy to disable absolute paths check
autoantwort Oct 7, 2022
646525c
Make code more testable
autoantwort Oct 16, 2022
9b0c5f8
Fix wrong if
autoantwort Oct 18, 2022
2329d8d
Fix performace
autoantwort Oct 19, 2022
771a5a1
Use std::string::find != std::string::npos instead of Strings::contai…
autoantwort Oct 19, 2022
1994010
Merge branch 'main' into check-pkg-config-files
autoantwort Oct 19, 2022
2df175b
Fix warnings
autoantwort Oct 19, 2022
23c9e7d
Merge branch 'main' into check-pkg-config-files
autoantwort Oct 28, 2022
0a5c013
Merge branch 'main' into check-pkg-config-files
autoantwort Nov 3, 2022
028dd92
Merge remote-tracking branch 'origin/main' into check-pkg-config-files
BillyONeal Nov 17, 2022
a2a42d5
Fix detection of the end of raw strings literals and copy tests from …
autoantwort Nov 17, 2022
4092884
No optimization
autoantwort Nov 17, 2022
22c3d0e
Fix paths on windows
autoantwort Nov 17, 2022
c1188c6
No need for .native()
autoantwort Nov 17, 2022
f8b77a2
More tests and less memory usage
autoantwort Nov 19, 2022
0937195
Don't search for downloads folder
autoantwort Nov 19, 2022
bc23f1f
Ignore hash comments in .conf files
autoantwort Nov 28, 2022
298a2f4
Fix formatting
autoantwort Nov 28, 2022
4fabb06
Fix buffer overflow
autoantwort Nov 28, 2022
ed7e9c8
Merge branch 'main' into check-pkg-config-files
autoantwort Dec 5, 2022
cc3f741
Add e2e tests
autoantwort Dec 8, 2022
ccf6b23
Merge branch 'main' into check-pkg-config-files
autoantwort Jan 18, 2023
5222349
Adapt ee7bb9819253c8c440105b055e2b7f06e2a204a1
autoantwort Jan 18, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions include/vcpkg/base/messages.h
Original file line number Diff line number Diff line change
@@ -309,6 +309,8 @@ namespace vcpkg::msg
DECLARE_MSG_ARG(option, "editable");
DECLARE_MSG_ARG(package_name, "zlib");
DECLARE_MSG_ARG(path, "/foo/bar");
DECLARE_MSG_ARG(paths, "/foo/bar, /foo/bar2");
DECLARE_MSG_ARG(absolute_paths, "/foo/bar, /foo/bar2");
DECLARE_MSG_ARG(row, "42");
DECLARE_MSG_ARG(spec, "zlib:x64-windows");
DECLARE_MSG_ARG(system_api, "CreateProcessW");
@@ -1088,6 +1090,11 @@ namespace vcpkg
"One or more {vendor} credential providers failed to authenticate. See '{url}' for more details "
"on how to provide credentials.");
DECLARE_MESSAGE(FeedbackAppreciated, (), "", "Thank you for your feedback!");
DECLARE_MESSAGE(FilesContainAbsolutePath,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to tell the user about the policy now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought the policy should only be used internally to that merging this PR does the break the world. Imho we should not tell the users how to ignore warnings that warns before broken packages.

(msg::absolute_paths, msg::paths),
"The ('{absolute_paths}') part should remain unchanged, as a list of paths",
"The following files contain an absolute path ('{absolute_paths}'): {paths}\n"
"There should be no absolute paths in the installed package, only relative ones.");
DECLARE_MESSAGE(FetchingBaselineInfo,
(msg::package_name),
"",
4 changes: 4 additions & 0 deletions include/vcpkg/base/strings.h
Original file line number Diff line number Diff line change
@@ -12,6 +12,8 @@
#include <algorithm>
#include <vector>

#include "vcpkg/base/fwd/span.h"

namespace vcpkg::Strings::details
{
// first looks up to_string on `T` using ADL; then, if that isn't found,
@@ -218,6 +220,8 @@ namespace vcpkg::Strings

Optional<StringView> find_at_most_one_enclosed(StringView input, StringView left_tag, StringView right_tag);

bool contains_any_ignoring_c_comments(const std::string& source, View<StringView> to_find);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

source should be a StringView


bool equals(StringView a, StringView b);

template<class T>
7 changes: 7 additions & 0 deletions include/vcpkg/base/util.h
Original file line number Diff line number Diff line change
@@ -266,6 +266,13 @@ namespace vcpkg::Util
return std::find(begin(cont), end(cont), v);
}

template<class Range, class T>
bool contains(const Range& r, const T& el)
{
using std::end;
return Util::find(r, el) != end(r);
}

template<class Container, class Pred>
auto find_if(Container&& cont, Pred pred)
{
1 change: 1 addition & 0 deletions include/vcpkg/build.h
Original file line number Diff line number Diff line change
@@ -311,6 +311,7 @@ namespace vcpkg
SKIP_DUMPBIN_CHECKS,
SKIP_ARCHITECTURE_CHECK,
CMAKE_HELPER_PORT,
SKIP_ABSOLUTE_PATHS_CHECK,
// Must be last
COUNT,
};
2 changes: 2 additions & 0 deletions locales/messages.json
Original file line number Diff line number Diff line change
@@ -403,6 +403,8 @@
"FileNotFound": "{path}: file not found",
"_FileNotFound.comment": "An example of {path} is /foo/bar.",
"FileSystemOperationFailed": "Filesystem operation failed:",
"FilesContainAbsolutePath": "The following files contain an absolute path ('{absolute_paths}'): {paths}\nThere should be no absolute paths in the installed package, only relative ones.",
"_FilesContainAbsolutePath.comment": "The ('{absolute_paths}') part should remain unchanged, as a list of paths An example of {absolute_paths} is /foo/bar, /foo/bar2. An example of {paths} is /foo/bar, /foo/bar2.",
"FilesExported": "Files exported at: {path}",
"_FilesExported.comment": "An example of {path} is /foo/bar.",
"FishCompletion": "vcpkg fish completion is already added at \"{path}\".",
18 changes: 18 additions & 0 deletions src/vcpkg-test/strings.cpp
Original file line number Diff line number Diff line change
@@ -3,6 +3,7 @@
#include <vcpkg/base/api_stable_format.h>
#include <vcpkg/base/expected.h>
#include <vcpkg/base/strings.h>
#include <vcpkg/base/view.h>

#include <stdint.h>

@@ -55,6 +56,23 @@ TEST_CASE ("find_first_of", "[strings]")
REQUIRE(find_first_of("abcdefg", "gb") == std::string("bcdefg"));
}

TEST_CASE ("contains_any_ignoring_c_comments", "[strings]")
{
using vcpkg::Strings::contains_any_ignoring_c_comments;
vcpkg::StringView to_find[] = {"abc", "wer"};
REQUIRE(contains_any_ignoring_c_comments(R"(abc)", to_find));
REQUIRE(contains_any_ignoring_c_comments(R"("abc")", to_find));
REQUIRE_FALSE(contains_any_ignoring_c_comments(R"("" //abc)", to_find));
REQUIRE_FALSE(contains_any_ignoring_c_comments(R"(/*abc*/ "")", to_find));
REQUIRE_FALSE(contains_any_ignoring_c_comments("// test \\\nabc", to_find));
REQUIRE(contains_any_ignoring_c_comments("\"//\" test \\\nabc", to_find));
REQUIRE(contains_any_ignoring_c_comments(R"(R"-( // abc )-")", to_find));
REQUIRE(contains_any_ignoring_c_comments(R"(R"-( /* abc */ )-")", to_find));
REQUIRE(contains_any_ignoring_c_comments(R"(R"-()- /* abc */ )-")", to_find));
REQUIRE(contains_any_ignoring_c_comments(R"(qwer )", to_find));
REQUIRE(contains_any_ignoring_c_comments("\"a\" \"g\" // er \n abc)", to_find));
}

TEST_CASE ("edit distance", "[strings]")
{
using vcpkg::Strings::byte_edit_distance;
3 changes: 2 additions & 1 deletion src/vcpkg/base/messages.cpp
Original file line number Diff line number Diff line change
@@ -635,11 +635,12 @@ namespace vcpkg
REGISTER_MESSAGE(FeedbackAppreciated);
REGISTER_MESSAGE(FetchingBaselineInfo);
REGISTER_MESSAGE(FetchingRegistryInfo);
REGISTER_MESSAGE(FishCompletion);
REGISTER_MESSAGE(FloatingPointConstTooBig);
REGISTER_MESSAGE(FileNotFound);
REGISTER_MESSAGE(FilesExported);
REGISTER_MESSAGE(FileSystemOperationFailed);
REGISTER_MESSAGE(FishCompletion);
REGISTER_MESSAGE(FilesContainAbsolutePath);
REGISTER_MESSAGE(FollowingPackagesMissingControl);
REGISTER_MESSAGE(FollowingPackagesNotInstalled);
REGISTER_MESSAGE(FollowingPackagesUpgraded);
67 changes: 67 additions & 0 deletions src/vcpkg/base/strings.cpp
Original file line number Diff line number Diff line change
@@ -382,6 +382,73 @@ Optional<StringView> Strings::find_at_most_one_enclosed(StringView input, String
return result.front();
}

bool vcpkg::Strings::contains_any_ignoring_c_comments(const std::string& source, View<StringView> to_find)
{
std::string::size_type offset = 0;
std::string::size_type no_comment_offset = 0;
while (offset != std::string::npos)
{
no_comment_offset = std::max(offset, no_comment_offset);
auto start = source.find_first_of("/\"", no_comment_offset);
if (start == std::string::npos || start + 1 == source.size() || no_comment_offset == std::string::npos)
{
return Util::any_of(to_find,
[&](StringView s) { return Strings::contains(StringView(source).substr(offset), s); });
}

if (source[start] == '/')
{
if (source[start + 1] == '/' || source[start + 1] == '*')
{
if (Util::any_of(to_find, [&](StringView s) {
return Strings::contains(StringView(source).substr(offset, start - offset), s);
}))
{
return true;
}
if (source[start + 1] == '/')
{
offset = source.find_first_of('\n', start);
while (offset != std::string::npos && source[offset - 1] == '\\')
offset = source.find_first_of('\n', offset + 1);
if (offset != std::string::npos) ++offset;
continue;
}
offset = source.find_first_of('/', start + 1);
while (offset != std::string::npos && source[offset - 1] != '*')
offset = source.find_first_of('/', offset + 1);
if (offset != std::string::npos) ++offset;
continue;
}
}
else if (source[start] == '\"')
{
if (start > 0 && source[start - 1] == 'R') // raw string literals
{
auto end = source.find_first_of('(', start);
if (end == std::string::npos)
{
// invalid c++, but allowed: auto test = 'R"'
no_comment_offset = start + 1;
continue;
}
auto d_char_sequence = source.substr(start, end - start);
d_char_sequence.push_back('\"');
no_comment_offset = source.find(d_char_sequence, end);
if (no_comment_offset != std::string::npos) no_comment_offset += d_char_sequence.size();
continue;
}
no_comment_offset = source.find_first_of('"', start + 1);
while (no_comment_offset != std::string::npos && source[no_comment_offset - 1] == '\\')
no_comment_offset = source.find_first_of('"', no_comment_offset + 1);
if (no_comment_offset != std::string::npos) ++no_comment_offset;
continue;
}
no_comment_offset = start + 1;
}
return false;
}

bool Strings::equals(StringView a, StringView b)
{
if (a.size() != b.size()) return false;
3 changes: 3 additions & 0 deletions src/vcpkg/build.cpp
Original file line number Diff line number Diff line change
@@ -216,6 +216,7 @@ namespace vcpkg
static const std::string NAME_SKIP_DUMPBIN_CHECKS = "PolicySkipDumpbinChecks";
static const std::string NAME_SKIP_ARCHITECTURE_CHECK = "PolicySkipArchitectureCheck";
static const std::string NAME_CMAKE_HELPER_PORT = "PolicyCmakeHelperPort";
static const std::string NAME_SKIP_ABSOLUTE_PATHS_CHECK = "PolicySkipAbsolutePathsCheck";

static std::remove_const_t<decltype(ALL_POLICIES)> generate_all_policies()
{
@@ -246,6 +247,7 @@ namespace vcpkg
case BuildPolicy::SKIP_DUMPBIN_CHECKS: return NAME_SKIP_DUMPBIN_CHECKS;
case BuildPolicy::SKIP_ARCHITECTURE_CHECK: return NAME_SKIP_ARCHITECTURE_CHECK;
case BuildPolicy::CMAKE_HELPER_PORT: return NAME_CMAKE_HELPER_PORT;
case BuildPolicy::SKIP_ABSOLUTE_PATHS_CHECK: return NAME_SKIP_ABSOLUTE_PATHS_CHECK;
default: Checks::unreachable(VCPKG_LINE_INFO);
}
}
@@ -266,6 +268,7 @@ namespace vcpkg
case BuildPolicy::SKIP_DUMPBIN_CHECKS: return "VCPKG_POLICY_SKIP_DUMPBIN_CHECKS";
case BuildPolicy::SKIP_ARCHITECTURE_CHECK: return "VCPKG_POLICY_SKIP_ARCHITECTURE_CHECK";
case BuildPolicy::CMAKE_HELPER_PORT: return "VCPKG_POLICY_CMAKE_HELPER_PORT";
case BuildPolicy::SKIP_ABSOLUTE_PATHS_CHECK: return "VCPKG_POLICY_SKIP_ABSOLUTE_PATHS_CHECK";
default: Checks::unreachable(VCPKG_LINE_INFO);
}
}
98 changes: 98 additions & 0 deletions src/vcpkg/postbuildlint.cpp
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
#include <vcpkg/base/cofffilereader.h>
#include <vcpkg/base/files.h>
#include <vcpkg/base/messages.h>
#include <vcpkg/base/system.print.h>
#include <vcpkg/base/system.process.h>
#include <vcpkg/base/util.h>

#include <vcpkg/build.h>
#include <vcpkg/installedpaths.h>
#include <vcpkg/packagespec.h>
#include <vcpkg/postbuildlint.buildtype.h>
#include <vcpkg/postbuildlint.h>
@@ -1025,6 +1027,95 @@ namespace vcpkg::PostBuildLint
return LintStatus::SUCCESS;
}

static LintStatus check_no_absolute_paths_in(const Filesystem& fs, const Path& dir, Span<Path> absolute_paths)
{
static constexpr StringLiteral extensions[] = {"h", "hpp", "hxx", "py", "sh", "cmake", "pc", "cfg", "conf"};
std::vector<std::pair<Path, std::string>> files_and_contents;
for (auto& path : fs.get_regular_files_recursive(dir, IgnoreErrors{}))
{
if (path.extension().empty())
{
auto contents = fs.read_contents(path, VCPKG_LINE_INFO);
if (Strings::starts_with(contents, "#!"))
{
files_and_contents.emplace_back(std::move(path), std::move(contents));
}
}
else if (Util::contains(extensions, path.extension().substr(1 /* ignore dot */)))
{
auto contents = fs.read_contents(path, VCPKG_LINE_INFO);
files_and_contents.emplace_back(std::move(path), std::move(contents));
}
}
std::vector<std::string> string_paths;
for (const auto& path : absolute_paths)
{
#if defined(_WIN32)
auto path_preferred = path;
path_preferred.make_preferred();
string_paths.push_back(path_preferred.generic_u8string());
string_paths.push_back(std::move(path).native());
#else
string_paths.push_back(path.native());
#endif
}
const auto stringview_paths = Util::fmap(string_paths, [](std::string& s) { return StringView(s); });

std::string result;
for (const auto& path_and_contents : files_and_contents)
{
const auto extension = path_and_contents.first.extension().substr(1 /* ignore dot */);
const bool is_header = extension == "h" || extension == "hpp" || extension == "hxx";
bool found_absolute = false;
if (is_header)
{
found_absolute = Util::any_of(string_paths, [&path_and_contents](const std::string& path) {
StringView sv(path);
bool contains = path_and_contents.second.find(path) != std::string::npos;
// First do a cheap search and then a complicated one
return contains && Strings::contains_any_ignoring_c_comments(path_and_contents.second, {&sv, 1});
});
}
else
{
found_absolute = Util::any_of(string_paths, [&path_and_contents, extension](const std::string& path) {
if (extension == "cfg" || extension == "conf")
{
return Strings::contains(path_and_contents.second, path);
}
for (size_t offset = 0;;)
{
const auto index = path_and_contents.second.find(path, offset);
if (index == std::string::npos) return false;
{ // .py, .sh, .cmake or .pc file
const auto before = path_and_contents.second.find_last_of("\n#", index);
if (before == std::string::npos) return true;
if (path_and_contents.second[before] == '\n') return true; // not a comment
}
offset = index + path.size();
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part looks extractable like contains_any_ignoring_c_comments. And it should handle #s not at the start of a line.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does that

});
}
if (found_absolute)
{
result += "\n ";
result += path_and_contents.first.native();
}
}

if (result.empty())
{
return LintStatus::SUCCESS;
}

msg::print(Color::warning,
msg::format(msgFilesContainAbsolutePath,
msg::absolute_paths = Strings::join("', '", absolute_paths),
msg::paths = result)
.append_raw("\n\n"));
return LintStatus::PROBLEM_DETECTED;
}

static void operator+=(size_t& left, const LintStatus& right) { left += static_cast<size_t>(right); }

static size_t perform_all_checks_and_return_error_count(const PackageSpec& spec,
@@ -1153,6 +1244,13 @@ namespace vcpkg::PostBuildLint
error_count += check_no_files_in_dir(fs, package_dir);
error_count += check_no_files_in_dir(fs, package_dir / "debug");
error_count += check_pkgconfig_dir_only_in_lib_dir(fs, package_dir);
if (!build_info.policies.is_enabled(BuildPolicy::SKIP_ABSOLUTE_PATHS_CHECK))
{
error_count += check_no_absolute_paths_in(
fs,
package_dir,
std::vector<Path>{package_dir.native(), paths.installed().root().native(), paths.build_dir(spec)});
}

return error_count;
}