Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vcpkg fails to install anything - File does not have the expected hash #4663

Closed
parsa opened this issue Nov 6, 2018 · 21 comments
Closed

Vcpkg fails to install anything - File does not have the expected hash #4663

parsa opened this issue Nov 6, 2018 · 21 comments
Labels
category:vcpkg-bug The issue is with the vcpkg system (including helper scripts in `scripts/cmake/`)

Comments

@parsa
Copy link

parsa commented Nov 6, 2018

Steps to reproduce:

  • vcpkg install --triplet x64-windows boost (or any other package)
    Expected outcome: Boost (or any other package) would install
    Actual outcome: An error message complaining about hash mismatch for the installed 7-zip is displayed

Output:

A suitable version of 7zip was not found (required v18.1.0). Downloading portable 7zip v18.1.0...
Downloading 7zip...
  https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 -> C:\Repos\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg
File does not have the expected hash:
             url : [ https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 ]
       File path : [ C:\Repos\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg.part ]
   Expected hash : [ 8c75314102e68d2b2347d592f8e3eb05812e1ebb525decbac472231633753f1d4ca31c8e6881a36144a8da26b2571305b3ae3f4e2b85fc4a290aeda63d1a13b8 ]
     Actual hash : [ a9dfaaafd15d98a2ac83682867ec5766720acf6e99d40d1a00d480692752603bf3f3742623f0ea85647a92374df405f331afd6021c5cf36af43ee8db198129c0 ]

Detail:

  • OS: Windows 10
  • Vcpkg commit: 1c01a56

Full Output:

PS C:\Repos\vcpkg> vcpkg install --triplet x64-windows boost
The following packages will be built and installed:
    boost[core]:x64-windows
  * boost-accumulators[core]:x64-windows
  * boost-algorithm[core]:x64-windows
  * boost-align[core]:x64-windows
  * boost-any[core]:x64-windows
  * boost-array[core]:x64-windows
  * boost-asio[core]:x64-windows
  * boost-assert[core]:x64-windows
  * boost-assign[core]:x64-windows
  * boost-atomic[core]:x64-windows
  * boost-beast[core]:x64-windows
  * boost-bimap[core]:x64-windows
  * boost-bind[core]:x64-windows
  * boost-build[core]:x64-windows
  * boost-callable-traits[core]:x64-windows
  * boost-chrono[core]:x64-windows
  * boost-circular-buffer[core]:x64-windows
  * boost-compatibility[core]:x64-windows
  * boost-compute[core]:x64-windows
  * boost-concept-check[core]:x64-windows
  * boost-config[core]:x64-windows
  * boost-container[core]:x64-windows
  * boost-container-hash[core]:x64-windows
  * boost-context[core]:x64-windows
  * boost-contract[core]:x64-windows
  * boost-conversion[core]:x64-windows
  * boost-convert[core]:x64-windows
  * boost-core[core]:x64-windows
  * boost-coroutine[core]:x64-windows
  * boost-coroutine2[core]:x64-windows
  * boost-crc[core]:x64-windows
  * boost-date-time[core]:x64-windows
  * boost-detail[core]:x64-windows
  * boost-disjoint-sets[core]:x64-windows
  * boost-dll[core]:x64-windows
  * boost-dynamic-bitset[core]:x64-windows
  * boost-endian[core]:x64-windows
  * boost-exception[core]:x64-windows
  * boost-fiber[core]:x64-windows
  * boost-filesystem[core]:x64-windows
  * boost-flyweight[core]:x64-windows
  * boost-foreach[core]:x64-windows
  * boost-format[core]:x64-windows
  * boost-function[core]:x64-windows
  * boost-function-types[core]:x64-windows
  * boost-functional[core]:x64-windows
  * boost-fusion[core]:x64-windows
  * boost-geometry[core]:x64-windows
  * boost-gil[core]:x64-windows
  * boost-graph[core]:x64-windows
  * boost-graph-parallel[core]:x64-windows
  * boost-hana[core]:x64-windows
  * boost-hana-msvc[core]:x64-windows
  * boost-heap[core]:x64-windows
  * boost-hof[core]:x64-windows
  * boost-icl[core]:x64-windows
  * boost-integer[core]:x64-windows
  * boost-interprocess[core]:x64-windows
  * boost-interval[core]:x64-windows
  * boost-intrusive[core]:x64-windows
  * boost-io[core]:x64-windows
  * boost-iostreams[core]:x64-windows
  * boost-iterator[core]:x64-windows
  * boost-lambda[core]:x64-windows
  * boost-lexical-cast[core]:x64-windows
  * boost-local-function[core]:x64-windows
  * boost-locale[core]:x64-windows
  * boost-lockfree[core]:x64-windows
  * boost-log[core]:x64-windows
  * boost-logic[core]:x64-windows
  * boost-math[core]:x64-windows
  * boost-metaparse[core]:x64-windows
  * boost-modular-build-helper[core]:x64-windows
  * boost-move[core]:x64-windows
  * boost-mp11[core]:x64-windows
  * boost-mpl[core]:x64-windows
  * boost-msm[core]:x64-windows
  * boost-multi-array[core]:x64-windows
  * boost-multi-index[core]:x64-windows
  * boost-multiprecision[core]:x64-windows
  * boost-numeric-conversion[core]:x64-windows
  * boost-odeint[core]:x64-windows
  * boost-optional[core]:x64-windows
  * boost-parameter[core]:x64-windows
  * boost-phoenix[core]:x64-windows
  * boost-poly-collection[core]:x64-windows
  * boost-polygon[core]:x64-windows
  * boost-pool[core]:x64-windows
  * boost-predef[core]:x64-windows
  * boost-preprocessor[core]:x64-windows
  * boost-process[core]:x64-windows
  * boost-program-options[core]:x64-windows
  * boost-property-map[core]:x64-windows
  * boost-property-tree[core]:x64-windows
  * boost-proto[core]:x64-windows
  * boost-ptr-container[core]:x64-windows
  * boost-python[core]:x64-windows
  * boost-qvm[core]:x64-windows
  * boost-random[core]:x64-windows
  * boost-range[core]:x64-windows
  * boost-ratio[core]:x64-windows
  * boost-rational[core]:x64-windows
  * boost-regex[core]:x64-windows
  * boost-scope-exit[core]:x64-windows
  * boost-serialization[core]:x64-windows
  * boost-signals[core]:x64-windows
  * boost-signals2[core]:x64-windows
  * boost-smart-ptr[core]:x64-windows
  * boost-sort[core]:x64-windows
  * boost-spirit[core]:x64-windows
  * boost-stacktrace[core]:x64-windows
  * boost-statechart[core]:x64-windows
  * boost-static-assert[core]:x64-windows
  * boost-system[core]:x64-windows
  * boost-test[core]:x64-windows
  * boost-thread[core]:x64-windows
  * boost-throw-exception[core]:x64-windows
  * boost-timer[core]:x64-windows
  * boost-tokenizer[core]:x64-windows
  * boost-tti[core]:x64-windows
  * boost-tuple[core]:x64-windows
  * boost-type-erasure[core]:x64-windows
  * boost-type-index[core]:x64-windows
  * boost-type-traits[core]:x64-windows
  * boost-typeof[core]:x64-windows
  * boost-ublas[core]:x64-windows
  * boost-units[core]:x64-windows
  * boost-unordered[core]:x64-windows
  * boost-utility[core]:x64-windows
  * boost-uuid[core]:x64-windows
  * boost-variant[core]:x64-windows
  * boost-vcpkg-helpers[core]:x64-windows
  * boost-vmd[core]:x64-windows
  * boost-wave[core]:x64-windows
  * boost-winapi[core]:x64-windows
  * boost-xpressive[core]:x64-windows
  * boost-yap[core]:x64-windows
  * bzip2[core]:x64-windows
  * liblzma[core]:x64-windows
  * openssl[core]:x64-windows
  * openssl-windows[core]:x64-windows
  * python3[core]:x64-windows
  * zlib[core]:x64-windows
Additional packages (*) will be modified to complete this operation.
Starting package 1/143: boost-vcpkg-helpers:x64-windows
Building package boost-vcpkg-helpers[core]:x64-windows...
A suitable version of cmake was not found (required v3.11.4). Downloading portable cmake v3.11.4...
Extracting cmake...
A suitable version of 7zip was not found (required v18.1.0). Downloading portable 7zip v18.1.0...
Downloading 7zip...
  https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 -> C:\Repos\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg
File does not have the expected hash:
             url : [ https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 ]
       File path : [ C:\Repos\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg.part ]
   Expected hash : [ 8c75314102e68d2b2347d592f8e3eb05812e1ebb525decbac472231633753f1d4ca31c8e6881a36144a8da26b2571305b3ae3f4e2b85fc4a290aeda63d1a13b8 ]
     Actual hash : [ a9dfaaafd15d98a2ac83682867ec5766720acf6e99d40d1a00d480692752603bf3f3742623f0ea85647a92374df405f331afd6021c5cf36af43ee8db198129c0 ]
@parsa parsa changed the title vcpkg fails to install anything- File does not have the expected hash Vcpkg fails to install anything - File does not have the expected hash Nov 6, 2018
@Rastaban
Copy link
Contributor

Rastaban commented Nov 7, 2018

That's annoyhing, I just tried the above command on a clean machine and was unable to reproduce the issue.

 A suitable version of 7zip was not found (required v18.1.0). Downloading portable 7zip v18.1.0...
Downloading 7zip...
  https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 -> E:\vcpkg\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg
Downloading 7zip... done.
Extracting 7zip...

Does it still repro? I ask only because it could have been an intermittent issue in the nuget service that is now resolved so it worked for me but not for you earlier.

@szx917
Copy link

szx917 commented Nov 7, 2018

same here
OS: win10 x64

>vcpkg install curl
The following packages will be built and installed:
    curl[core,ssl]:x86-windows
  * openssl[core]:x86-windows
  * openssl-windows[core]:x86-windows
  * zlib[core]:x86-windows
Additional packages (*) will be modified to complete this operation.
Starting package 1/4: zlib:x86-windows
Building package zlib[core]:x86-windows...
A suitable version of cmake was not found (required v3.11.4). Downloading portable cmake v3.11.4...
Extracting cmake...
A suitable version of 7zip was not found (required v18.1.0). Downloading portable 7zip v18.1.0...
Downloading 7zip...
  https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 -> C:\Users\Jane Wang\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg
File does not have the expected hash:
             url : [ https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 ]
       File path : [ C:\Users\****\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg.part ]
   Expected hash : [ 8c75314102e68d2b2347d592f8e3eb05812e1ebb525decbac472231633753f1d4ca31c8e6881a36144a8da26b2571305b3ae3f4e2b85fc4a290aeda63d1a13b8 ]
     Actual hash : [ a9dfaaafd15d98a2ac83682867ec5766720acf6e99d40d1a00d480692752603bf3f3742623f0ea85647a92374df405f331afd6021c5cf36af43ee8db198129c0 ]

@anders-wind
Copy link
Contributor

I'm experiencing the same issue. Tried
vcpkg commit: 9ad1662
Fails at 7-Zip.Commandline/18.1.0 extraction.

@hjabird
Copy link

hjabird commented Nov 7, 2018

This can be temporarily and foolishly 'fixed' by replacing the hash value in vcpkg/scripts/vcpkgTools (tool name="7zip" os="windows") with the actual hash. I'm pretty sure that it such an act ranks highly in the list of security sins.

@ijean
Copy link

ijean commented Nov 7, 2018

I can confirm, same problem, on a clean Windows 10 x64 machine.

@hjabird Instead of changing the hash, probably a better idea is to bump the version to 18.05 which is the latest stable version and solves some Windows 10 bugs.

@parsa
Copy link
Author

parsa commented Nov 7, 2018

@Rastaban tried again on a clean VM and it does still reproduce. Same exact hashes. Also I my machine I just changed the hash in scripts/vcpkgTools.xml and the issue went away and Boost installed just fine.

@derekseiple
Copy link

derekseiple commented Nov 7, 2018

I can confirm the same failure as well.

To work around this I copied the original 7-zip.commandline.18.1.0.nupkg (the one with the correct hash) that I had on hand from another install location into the vcpkg/downloads directory. It was then able to install packages as normal.

This doesn't help though if you can't get your hands on the right file...

@ras0219-msft
Copy link
Contributor

Thank you everyone for quickly jumping in and reporting this!

I've just pushed 068032b to master, which downgrades the version of 7-zip to a previous version while we work on some internal changes to fix this more permanently for 18.1.0.

The root cause of the issue is that NuGet.org decided to modify the existing published nupkg files and inject signature files into them, which changes their hash. Our system to protect users from malicious files detected this change and correctly refused to use the un-validated executable.

Unfortunately, it isn't as simple as changing the expected hash, because existing users will still have the older file downloaded. We also can't just rename the file, because nuget.exe demands that the file be named in a very specific way.

As mentioned above, we're working on fixing this properly as a high priority, but 068032b will fix the issue for now.

Thanks again!

@jherico
Copy link
Contributor

jherico commented Nov 7, 2018

Since applying the hotfix I'm seeing a new problem

Expected C:\Users\bdavi\Git\vcpkg\downloads\tools\cmake-3.11.4-windows\cmake-3.11.4-win32-x86\bin\cmake.exe to exist after fetching

This didn't make sense to me because the file it's looking for exists. I dug into the code and discovered that fs::stdfs::exists was returning false for the executable.

I created a brand new local C++ project and verified the same result, not just for the executable, but for all the files in the archive. However, if I wiped the folder and then re-extracted the files using WinRAR, the files suddenly visible as far as fs::stdfs::exists was concerned.

Something about the 16 version of 7zip is setting file attributes so that even though they're visible in the command line, can be executed, and visible in explorer, the C++ library's filesystem exists function thinks they're not present. This actually feels like a bug in the C++ library, but it's blocking my use of vcpkg.

@ras0219-msft
Copy link
Contributor

Thanks for the further investigation.

I was able to confirm some strange behavior (though not this particular issue) when using the old 7zip as well. I've pushed a different workaround (273b8ce) that's a bit more permanent by mapping the new hash in the tool back to the old hash.

The reason to map the new hash to the old hash is to improve compatibility for existing users which might git pull but forget to ./bootstrap-vcpkg.bat -- they'll see everything keep working as before, because they already have the old version downloaded. For new users, they will definitely be bootstrapping, so the new file's hash will get (hackily) mapped to the old hash and the check will pass.

A proper fix is still desirable, but this particular duct tape should hold for longer since it's using the same (latest available) version of 7zip as before.

@Farwaykorse
Copy link
Contributor

Farwaykorse commented Nov 19, 2018

This issue still appears for anyone who has not ran .\bootstrap-vcpkg.bat since the patch.
It is not something I'd do when there is no update warning.

Why not increase the vcpkg version?

@degski
Copy link

degski commented Nov 19, 2018

@ras0219-msft @Farwaykorse Yes, the version should be increased so at least a message to run .\bootstrap-vcpkg.bat pops up.

@ras0219-msft The nuget versioning is wacky, 7zip versioning is format YY,MM, i.e. 18.01, and not number.number.

@Adraesh
Copy link

Adraesh commented Nov 27, 2018

Hello, I have the same issue, I am trying to run .\bootstrap-vcpkg.bat but it failed due to toolsrc/ folder not existing.

I got vcpkg 1.50 through VS 2017 Nuget Package Manager and by this way you don't get the toolsrc/ folder...

Thank you in advance.

@aybassiouny
Copy link
Contributor

This seems to be an issue especially when going back in time to get earlier versions of libraries.

@rhvarrier
Copy link

This issue is still present in 2020.01 version

@ghost
Copy link

ghost commented Oct 6, 2020

For me it was solved by running the command with admin privileges...

@malidiab
Copy link

malidiab commented Oct 7, 2020

updating vcpkg to 2020.06.15- fixes the problem to me

@Nikmeh922
Copy link

Nikmeh922 commented Dec 29, 2020

I can confirm the same failure as well.

To work around this I copied the original 7-zip.commandline.18.1.0.nupkg (the one with the correct hash) that I had on hand from another install location into the vcpkg/downloads directory. It was then able to install packages as normal.

This doesn't help though if you can't get your hands on the right file...

Thank you. This helped a lot!!! @derekseiple

@parsa
Copy link
Author

parsa commented Dec 29, 2020

Thank you everyone for quickly jumping in and reporting this!

I've just pushed 068032b to master, which downgrades the version of 7-zip to a previous version while we work on some internal changes to fix this more permanently for 18.1.0.

The root cause of the issue is that NuGet.org decided to modify the existing published nupkg files and inject signature files into them, which changes their hash. Our system to protect users from malicious files detected this change and correctly refused to use the un-validated executable.

Unfortunately, it isn't as simple as changing the expected hash, because existing users will still have the older file downloaded. We also can't just rename the file, because nuget.exe demands that the file be named in a very specific way.

As mentioned above, we're working on fixing this properly as a high priority, but 068032b will fix the issue for now.

Thanks again!

This issue has been long solved for me, and I don't think the follow-ups describe the same issue. If my opinion as the person who opened this issue counts, this issue can be closed.

@PhoebeHui
Copy link
Contributor

I agree, thank you parsa for your response! closing this issue now.

@fly2sky2018
Copy link

I solve the preblem
1)download this files
https://raw.githubusercontent.com/boostorg/boost/boost-1.75.0/LICENSE_1_0.txt
https://raw.githubusercontent.com/boostorg/boost/boost-1.75.0/boostcpp.jam
2)skip the hash test
add skip code at the vcpkg_download_distfile.camke

@microsoft microsoft locked as resolved and limited conversation to collaborators Nov 21, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
category:vcpkg-bug The issue is with the vcpkg system (including helper scripts in `scripts/cmake/`)
Projects
None yet
Development

No branches or pull requests