Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[post build lint] Check for absolute paths #172

Merged
merged 68 commits into from
Jan 20, 2023
Merged
Show file tree
Hide file tree
Changes from 38 commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
863231d
[post build lint] Check for absolute paths
autoantwort Aug 25, 2021
15778b6
Apply suggestions from code review
autoantwort Oct 7, 2021
9212d1b
Fix build and logic
autoantwort Oct 7, 2021
326672e
Merge branch 'main' into check-pkg-config-files
autoantwort Oct 12, 2021
cef8cb0
nicole's CRs
strega-nil-ms Oct 18, 2021
761dc17
fix build
autoantwort Oct 19, 2021
7fd595a
Merge branch 'main' into check-pkg-config-files
autoantwort Oct 29, 2021
7e9fdd0
Ignore dot in extension
autoantwort Oct 29, 2021
f58d9ae
Merge branch 'main' into check-pkg-config-files
autoantwort Nov 9, 2021
6840f39
fix build
autoantwort Nov 9, 2021
d7f0f47
Merge branch 'main' into check-pkg-config-files
autoantwort Nov 20, 2021
42c62d7
fix build
autoantwort Nov 20, 2021
80704ee
don't check .la files
autoantwort Nov 21, 2021
2c1fb86
also check for the buildtree dir
autoantwort Nov 23, 2021
cd19401
Merge branch 'main' into check-pkg-config-files
autoantwort Nov 27, 2021
1fd5a30
don't check .yaml files
autoantwort Nov 28, 2021
6154d36
ignore comments in files
autoantwort Dec 1, 2021
47787c5
Merge branch 'main' into check-pkg-config-files
autoantwort Dec 21, 2021
86072b7
adapt code
autoantwort Dec 21, 2021
ea1e189
fix comment detection
autoantwort Dec 24, 2021
10049d7
Update include/vcpkg/base/util.h
autoantwort Dec 30, 2021
03d24cd
Change order of conditions
autoantwort Dec 30, 2021
b51e0b1
Update src/vcpkg/postbuildlint.cpp
autoantwort Dec 30, 2021
5d19692
Use localized message
autoantwort Dec 31, 2021
b48fd0f
Merge branch 'main' into check-pkg-config-files
autoantwort Dec 31, 2021
81c991d
Update message map
autoantwort Dec 31, 2021
aa5264b
Merge branch 'main' into check-pkg-config-files
autoantwort Feb 21, 2022
8b2cc5c
Merge branch 'main' into check-pkg-config-files
autoantwort Mar 6, 2022
d58b64e
Update messages.en.json
autoantwort Mar 8, 2022
6db65d5
Merge branch 'main' into check-pkg-config-files
autoantwort Mar 24, 2022
04e77f5
Merge branch 'main' into check-pkg-config-files
autoantwort Apr 8, 2022
dce3e55
Merge branch 'main' into check-pkg-config-files
autoantwort Apr 18, 2022
9df3a83
Merge branch 'main' into check-pkg-config-files
autoantwort May 12, 2022
dec596d
Merge branch 'main' into check-pkg-config-files
autoantwort May 19, 2022
20a0747
Merge remote-tracking branch 'origin/main' into check-pkg-config-files
BillyONeal May 20, 2022
cc4cbc8
Merge branch 'main' into check-pkg-config-files
autoantwort Jun 25, 2022
7b535b4
No line endings in message
autoantwort Jun 25, 2022
dfa58d3
Merge remote-tracking branch 'origin/main' into check-pkg-config-files
BillyONeal Aug 11, 2022
3fd3ea2
Merge remote-tracking branch 'origin/main' into check-pkg-config-files
BillyONeal Aug 18, 2022
024b814
Apply the if/return transform requested by https://github.com/microso…
BillyONeal Aug 18, 2022
a1b3aa1
Merge remote-tracking branch 'origin/main' into check-pkg-config-files
BillyONeal Aug 22, 2022
6421db4
Merge branch 'main' into check-pkg-config-files
autoantwort Sep 2, 2022
bdcb84c
Merge branch 'main' into check-pkg-config-files
autoantwort Sep 26, 2022
2da94b4
Merge remote-tracking branch 'origin/main' into check-pkg-config-files
BillyONeal Sep 30, 2022
535e47a
Merge branch 'main' into check-pkg-config-files
autoantwort Oct 7, 2022
72476ba
Add policy to disable absolute paths check
autoantwort Oct 7, 2022
646525c
Make code more testable
autoantwort Oct 16, 2022
9b0c5f8
Fix wrong if
autoantwort Oct 18, 2022
2329d8d
Fix performace
autoantwort Oct 19, 2022
771a5a1
Use std::string::find != std::string::npos instead of Strings::contai…
autoantwort Oct 19, 2022
1994010
Merge branch 'main' into check-pkg-config-files
autoantwort Oct 19, 2022
2df175b
Fix warnings
autoantwort Oct 19, 2022
23c9e7d
Merge branch 'main' into check-pkg-config-files
autoantwort Oct 28, 2022
0a5c013
Merge branch 'main' into check-pkg-config-files
autoantwort Nov 3, 2022
028dd92
Merge remote-tracking branch 'origin/main' into check-pkg-config-files
BillyONeal Nov 17, 2022
a2a42d5
Fix detection of the end of raw strings literals and copy tests from …
autoantwort Nov 17, 2022
4092884
No optimization
autoantwort Nov 17, 2022
22c3d0e
Fix paths on windows
autoantwort Nov 17, 2022
c1188c6
No need for .native()
autoantwort Nov 17, 2022
f8b77a2
More tests and less memory usage
autoantwort Nov 19, 2022
0937195
Don't search for downloads folder
autoantwort Nov 19, 2022
bc23f1f
Ignore hash comments in .conf files
autoantwort Nov 28, 2022
298a2f4
Fix formatting
autoantwort Nov 28, 2022
4fabb06
Fix buffer overflow
autoantwort Nov 28, 2022
ed7e9c8
Merge branch 'main' into check-pkg-config-files
autoantwort Dec 5, 2022
cc3f741
Add e2e tests
autoantwort Dec 8, 2022
ccf6b23
Merge branch 'main' into check-pkg-config-files
autoantwort Jan 18, 2023
5222349
Adapt ee7bb9819253c8c440105b055e2b7f06e2a204a1
autoantwort Jan 18, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions include/vcpkg/base/messages.h
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,8 @@ namespace vcpkg::msg
DECLARE_MSG_ARG(option, "editable");
DECLARE_MSG_ARG(package_name, "zlib");
DECLARE_MSG_ARG(path, "/foo/bar");
DECLARE_MSG_ARG(paths, "/foo/bar, /foo/bar2");
DECLARE_MSG_ARG(absolute_paths, "/foo/bar, /foo/bar2");
DECLARE_MSG_ARG(row, "42");
DECLARE_MSG_ARG(spec, "zlib:x64-windows");
DECLARE_MSG_ARG(system_api, "CreateProcessW");
Expand Down Expand Up @@ -820,6 +822,11 @@ namespace vcpkg
"One or more {vendor} credential providers failed to authenticate. See '{url}' for more details "
"on how to provide credentials.");
DECLARE_MESSAGE(FeedbackAppreciated, (), "", "Thank you for your feedback!");
DECLARE_MESSAGE(FilesContainAbsolutePath,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to tell the user about the policy now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought the policy should only be used internally to that merging this PR does the break the world. Imho we should not tell the users how to ignore warnings that warns before broken packages.

(msg::absolute_paths, msg::paths),
"The ('{absolute_paths}') part should remain unchanged, as a list of paths",
"The following files contain an absolute path ('{absolute_paths}'): {paths}\n"
"There should be no absolute paths in the installed package, only relative ones.");
DECLARE_MESSAGE(FishCompletion, (msg::path), "", "vcpkg fish completion is already added at \"{path}\".");
DECLARE_MESSAGE(
ForceSystemBinariesOnWeirdPlatforms,
Expand Down
7 changes: 7 additions & 0 deletions include/vcpkg/base/util.h
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,13 @@ namespace vcpkg::Util
return std::find(begin(cont), end(cont), v);
}

template<class Range, class T>
bool contains(const Range& r, const T& el)
{
using std::end;
return Util::find(r, el) != end(r);
}

template<class Container, class Pred>
auto find_if(Container&& cont, Pred pred)
{
Expand Down
1 change: 1 addition & 0 deletions locales/messages.en.json
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,7 @@
"FailedToStoreBinaryCache": "Failed to store binary cache {path}",
"FailedVendorAuthentication": "One or more {vendor} credential providers failed to authenticate. See '{url}' for more details on how to provide credentials.",
"FeedbackAppreciated": "Thank you for your feedback!",
"FilesContainAbsolutePath": "The following files contain an absolute path ('{absolute_paths}'): {paths}\nThere should be no absolute paths in the installed package, only relative ones.",
"FishCompletion": "vcpkg fish completion is already added at \"{path}\".",
"ForceSystemBinariesOnWeirdPlatforms": "Environment variable VCPKG_FORCE_SYSTEM_BINARIES must be set on arm, s390x, and ppc64le platforms.",
"FormattedParseMessageExpression": "on expression: {value}",
Expand Down
2 changes: 2 additions & 0 deletions locales/messages.json
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,8 @@
"FailedVendorAuthentication": "One or more {vendor} credential providers failed to authenticate. See '{url}' for more details on how to provide credentials.",
"_FailedVendorAuthentication.comment": "An example of {vendor} is Azure. An example of {url} is https://github.com/microsoft/vcpkg.",
"FeedbackAppreciated": "Thank you for your feedback!",
"FilesContainAbsolutePath": "The following files contain an absolute path ('{absolute_paths}'): {paths}\nThere should be no absolute paths in the installed package, only relative ones.",
"_FilesContainAbsolutePath.comment": "The ('{absolute_paths}') part should remain unchanged, as a list of paths An example of {absolute_paths} is /foo/bar, /foo/bar2. An example of {paths} is /foo/bar, /foo/bar2.",
"FishCompletion": "vcpkg fish completion is already added at \"{path}\".",
"_FishCompletion.comment": "An example of {path} is /foo/bar.",
"ForceSystemBinariesOnWeirdPlatforms": "Environment variable VCPKG_FORCE_SYSTEM_BINARIES must be set on arm, s390x, and ppc64le platforms.",
Expand Down
1 change: 1 addition & 0 deletions src/vcpkg/base/messages.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -542,6 +542,7 @@ namespace vcpkg
REGISTER_MESSAGE(FailedVendorAuthentication);
REGISTER_MESSAGE(FeedbackAppreciated);
REGISTER_MESSAGE(FishCompletion);
REGISTER_MESSAGE(FilesContainAbsolutePath);
REGISTER_MESSAGE(ForceSystemBinariesOnWeirdPlatforms);
REGISTER_MESSAGE(FormattedParseMessageExpression);
REGISTER_MESSAGE(GenerateMsgErrorParsingFormatArgs);
Expand Down
101 changes: 101 additions & 0 deletions src/vcpkg/postbuildlint.cpp
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
#include <vcpkg/base/cofffilereader.h>
#include <vcpkg/base/files.h>
#include <vcpkg/base/messages.h>
#include <vcpkg/base/system.print.h>
#include <vcpkg/base/system.process.h>
#include <vcpkg/base/util.h>

#include <vcpkg/build.h>
#include <vcpkg/installedpaths.h>
#include <vcpkg/packagespec.h>
#include <vcpkg/postbuildlint.buildtype.h>
#include <vcpkg/postbuildlint.h>
Expand Down Expand Up @@ -1025,6 +1027,101 @@ namespace vcpkg::PostBuildLint
return LintStatus::SUCCESS;
}

static LintStatus check_no_absolute_paths_in(const Filesystem& fs, const Path& dir, Span<Path> absolute_paths)
{
static constexpr StringLiteral extensions[] = {"h", "hpp", "hxx", "py", "sh", "cmake", "pc", "cfg", "conf"};
std::vector<std::pair<Path, std::string>> files_and_contents;
autoantwort marked this conversation as resolved.
Show resolved Hide resolved
for (auto& path : fs.get_regular_files_recursive(dir, IgnoreErrors{}))
{
if (path.extension().empty())
{
auto contents = fs.read_contents(path, VCPKG_LINE_INFO);
if (Strings::starts_with(contents, "#!"))
{
files_and_contents.emplace_back(std::move(path), std::move(contents));
}
}
else if (Util::contains(extensions, path.extension().substr(1 /* ignore dot */)))
{
auto contents = fs.read_contents(path, VCPKG_LINE_INFO);
files_and_contents.emplace_back(std::move(path), std::move(contents));
}
}
std::vector<std::string> string_paths;
for (const auto& path : absolute_paths)
{
#if defined(_WIN32)
auto path_preferred = path;
path_preferred.make_preferred();
string_paths.push_back(path_preferred.generic_u8string());
string_paths.push_back(std::move(path).native());
autoantwort marked this conversation as resolved.
Show resolved Hide resolved
#else
string_paths.push_back(path.native());
#endif
}

std::string result;
for (const auto& path_and_contents : files_and_contents)
{
const auto extension = path_and_contents.first.extension().substr(1 /* ignore dot */);
const bool is_header = extension == "h" || extension == "hpp" || extension == "hxx";
vicroms marked this conversation as resolved.
Show resolved Hide resolved
if (Util::any_of(string_paths, [&path_and_contents, extension, is_header](const std::string& path) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (Util::any_of(string_paths, [&path_and_contents, extension, is_header](const std::string& path) {
if (Util::any_of(string_paths, [&path_and_contents, extension, is_header](StringView path) {

Would be nice if we could use StringView here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That won't do anything, the input is already a vector<std::string>.

if (extension == "cfg" || extension == "conf")
{
return Strings::contains(path_and_contents.second, path);
}
for (size_t offset = 0;;)
{
const auto index = path_and_contents.second.find(path, offset);
if (index == std::string::npos) return false;
if (is_header)
{
bool new_line = false;
for (auto start = index;;)
{
const auto before = path_and_contents.second.find_last_of("\n/", start);
if (before == std::string::npos) return true;
if (path_and_contents.second[before] == '\n')
{
if (before == 0) return true;
new_line = true;
start = before - 1;
continue;
}
if (path_and_contents.second[before + 1] == '*') break; // is in a comment
if (before == 0) return true; // not in a comment
if (!new_line && path_and_contents.second[before - 1] == '/') break; // is in a comment
if (path_and_contents.second[before - 1] == '*') return true; // is not in a comment
start = before - 1;
}
}
else
{ // .py, .sh, .cmake or .pc file
const auto before = path_and_contents.second.find_last_of("\n#", index);
if (before == std::string::npos) return true;
if (path_and_contents.second[before] == '\n') return true; // not a comment
}
offset = index + path.size();
}
}))
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

given that this is a warning, not a hard error, I'd prefer we just checked for the absolute paths without trying to look for whether we're in a comment. This is just a lot of complexity for little gain, imo.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a hard error in the ci...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right but we can fix those issues.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could, but this is a lot of work for nothing

Copy link
Member

@vicroms vicroms Oct 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add tests at least for the part that looks for absolute paths in header files.

result += "\n ";
result += path_and_contents.first.native();
}
}

if (!result.empty())
{
vicroms marked this conversation as resolved.
Show resolved Hide resolved
msg::print(Color::warning,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
msg::print(Color::warning,
msg::println_warning(

Copy link
Member

@BillyONeal BillyONeal Aug 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would change the effects; I would prefer to merge this as is but I think we would welcome a following PR that changes all the post build checks to conform with our warnings guidelines. (Although @JavierMatosD may already have such a PR outstanding...)

msg::format(msgFilesContainAbsolutePath,
msg::absolute_paths = Strings::join("', '", absolute_paths),
msg::paths = result)
.append_raw("\n\n"));
return LintStatus::PROBLEM_DETECTED;
}
return LintStatus::SUCCESS;
}

static void operator+=(size_t& left, const LintStatus& right) { left += static_cast<size_t>(right); }

static size_t perform_all_checks_and_return_error_count(const PackageSpec& spec,
Expand Down Expand Up @@ -1153,6 +1250,10 @@ namespace vcpkg::PostBuildLint
error_count += check_no_files_in_dir(fs, package_dir);
error_count += check_no_files_in_dir(fs, package_dir / "debug");
error_count += check_pkgconfig_dir_only_in_lib_dir(fs, package_dir);
error_count += check_no_absolute_paths_in(
fs,
package_dir,
std::vector<Path>{package_dir.native(), paths.installed().root().native(), paths.build_dir(spec)});

return error_count;
}
Expand Down