All packages with sources hosted by GitHub currently do not download due to changes in GitHub's archives #29288

bernhardmgruber · 2023-01-30T19:45:39Z

Host Environment

OS: Ubuntu 22.04 as used by GitHub actions
Compiler: GCC, clang, icpx

To Reproduce

I run this inside my GitHub action:
vcpkg install alpaka fmt tinyobjloader boost-mp11 boost-atomic boost-smart-ptr boost-functional boost-container boost-iostreams[core] catch2 xsimd

Failure logs

Excerpted from: https://github.com/alpaka-group/llama/actions/runs/4046805344/jobs/6961596764

Computing installation plan...
The following packages will be built and installed:
    alpaka[core]:x64-linux -> 0.9.0#1
  * boost-align[core]:x64-linux -> 1.81.0
  * boost-array[core]:x64-linux -> 1.81.0
  * boost-assert[core]:x64-linux -> 1.81.0
    boost-atomic[core]:x64-linux -> 1.81.0
  * boost-bind[core]:x64-linux -> 1.81.0
  * boost-build[core]:x64-linux -> 1.81.0
  * boost-concept-check[core]:x64-linux -> 1.81.0
  * boost-config[core]:x64-linux -> 1.81.0
    boost-container[core]:x64-linux -> 1.81.0
  * boost-container-hash[core]:x64-linux -> 1.81.0
  * boost-conversion[core]:x64-linux -> 1.81.0
  * boost-core[core]:x64-linux -> 1.81.0
  * boost-describe[core]:x64-linux -> 1.81.0
  * boost-detail[core]:x64-linux -> 1.81.0
  * boost-dynamic-bitset[core]:x64-linux -> 1.81.0
  * boost-function[core]:x64-linux -> 1.81.0
  * boost-function-types[core]:x64-linux -> 1.81.0
    boost-functional[core]:x64-linux -> 1.81.0
  * boost-fusion[core]:x64-linux -> 1.81.0
  * boost-integer[core]:x64-linux -> 1.81.0
  * boost-intrusive[core]:x64-linux -> 1.81.0
  * boost-io[core]:x64-linux -> 1.81.0
    boost-iostreams[core]:x64-linux -> 1.81.0
  * boost-iterator[core]:x64-linux -> 1.81.0
  * boost-modular-build-helper[core]:x64-linux -> 1.81.0
  * boost-move[core]:x64-linux -> 1.81.0
    boost-mp11[core]:x64-linux -> 1.81.0
  * boost-mpl[core]:x64-linux -> 1.81.0
  * boost-numeric-conversion[core]:x64-linux -> 1.81.0
  * boost-optional[core]:x64-linux -> 1.81.0
  * boost-predef[core]:x64-linux -> 1.81.0
  * boost-preprocessor[core]:x64-linux -> 1.81.0
  * boost-random[core]:x64-linux -> 1.81.0
  * boost-range[core]:x64-linux -> 1.81.0
  * boost-regex[core]:x64-linux -> 1.81.0
    boost-smart-ptr[core]:x64-linux -> 1.81.0
-- Performing post-build validation
Stored binary cache: "/home/runner/.cache/vcpkg/archives/14/14a7e106a0e9aee1d43b39d00c62a19db02aa62430357d6a214483e0610b9e70.zip"
Elapsed time to handle boost-config:x64-linux: 417.8 ms
Installing 4/56 boost-assert:x64-linux...
Building boost-assert[core]:x64-linux...
-- Downloading https://github.com/boostorg/assert/archive/boost-1.81.0.tar.gz -> boostorg-assert-boost-1.81.0.tar.gz...
[DEBUG] To include the environment variables in debug output, pass --debug-env
[DEBUG] Feature flag 'binarycaching' unset
[DEBUG] Feature flag 'compilertracking' unset
[DEBUG] Feature flag 'registries' unset
[DEBUG] Feature flag 'versions' unset
[DEBUG] Detected CI environment: GitHub_Actions
[DEBUG] 1000: popen( curl --fail -L https://github.com/boostorg/assert/archive/boost-1.81.0.tar.gz --create-dirs --output /usr/local/share/vcpkg/downloads/boostorg-assert-boost-1.81.0.tar.gz.1750.part 2>&1)
[DEBUG] 1000: cmd_execute_and_stream_data() returned 0 after   206285 us
error: Failed to download from mirror set
error: File does not have the expected hash:
url: https://github.com/boostorg/assert/archive/boost-1.81.0.tar.gz
File: /usr/local/share/vcpkg/downloads/boostorg-assert-boost-1.81.0.tar.gz.1750.part
Expected hash: c443b61a9ad29d4c9707aaaa997300a959fdef2d360aec9ee221abc[12](https://github.com/alpaka-group/llama/actions/runs/4046805344/jobs/6961596764#step:8:13)cd8446b3bc969e6c282ba778bee7d6fb54cc02655f15e01eacf1ad87262534dc4cc32bf
Actual hash: cb031c35d3f4342e862b46f309874bc28773cc95657841b772ff7b52dd8495c92eb496273e80ee50155b418e0[13](https://github.com/alpaka-group/llama/actions/runs/4046805344/jobs/6961596764#step:8:14)a3cc62b352bdd6d90274f49206c89089c1aa6
[DEBUG] /mnt/vss/_work/1/s/src/vcpkg/base/downloads.cpp(956): 
[DEBUG] Time in subprocesses: 206285 us
[DEBUG] Time in parsing JSON: 3 us
[DEBUG] Time in JSON reader: 0 us
[DEBUG] Time in filesystem: 100 us
[DEBUG] Time in loading ports: 0 us
[DEBUG] Exiting after 207.4 ms (207047 us)

CMake Error at scripts/cmake/vcpkg_download_distfile.cmake:32 (message):
      
      Failed to download file with error: 1
      If you use a proxy, please check your proxy setting. Possible causes are:
      
      1. You are actually using an HTTP proxy, but setting HTTPS_PROXY variable
         to `https://address:port`. This is not correct, because `https://` prefix
         claims the proxy is an HTTPS proxy, while your proxy (v2ray, shadowsocksr
         , etc..) is an HTTP proxy. Try setting `http://address:port` to both
         HTTP_PROXY and HTTPS_PROXY instead.
      
      2. You are using Fiddler. Currently a bug (https://github.com/microsoft/vcpkg/issues/17752)
         will set HTTPS_PROXY to `https://fiddler_address:port` which lead to problem 1 above.
         Workaround is open Windows 10 Settings App, and search for Proxy Configuration page,
         Change `http=address:port;https=address:port` to `address`, and fill the port number.
      
      3. Your proxy's remote server is out of service.
      
      In future vcpkg releases, if you are using Windows, you no longer need to set
      HTTP(S)_PROXY environment variables. Vcpkg will simply apply Windows IE Proxy
      Settings set by your proxy software. See (https://github.com/microsoft/vcpkg-tool/pull/49)
      and (https://github.com/microsoft/vcpkg-tool/pull/77)
      
      Otherwise, please submit an issue at https://github.com/Microsoft/vcpkg/issues

Call Stack (most recent call first):
  scripts/cmake/vcpkg_download_distfile.cmake:273 (z_vcpkg_download_distfile_show_proxy_and_fail)
  scripts/cmake/vcpkg_from_github.cmake:106 (vcpkg_download_distfile)
  ports/boost-assert/portfile.cmake:3 (vcpkg_from_github)
  scripts/ports.cmake:[14](https://github.com/alpaka-group/llama/actions/runs/4046805344/jobs/6961596764#step:8:15)7 (include)


error: building boost-assert:x64-linux failed with: BUILD_FAILED
error: Please ensure you're using the latest port files with `git pull` and `vcpkg update`.
Then check for known issues at:
    https://github.com/microsoft/vcpkg/issues?q=is%3Aissue+is%3Aopen+in%3Atitle+boost-assert
You can submit a new issue at:
    https://github.com/microsoft/vcpkg/issues/new?template=report-package-build-failure.md&title=[boost-assert]+Build+error
Include '[boost-assert] Build error' in your bug report title, the following version information in your bug description, and attach any relevant failure logs from above.
    vcpkg-tool version: [20](https://github.com/alpaka-group/llama/actions/runs/4046805344/jobs/6961596764#step:8:21)[22](https://github.com/alpaka-group/llama/actions/runs/4046805344/jobs/6961596764#step:8:23)-12-14-7ae0d8527fb488fde10a89c2813802dc9b03b6f9
    vcpkg-scripts version: e69dd972b 20[23](https://github.com/alpaka-group/llama/actions/runs/4046805344/jobs/6961596764#step:8:24)-01-20 (10 days ago)


Please use the prefilled template from /usr/local/share/vcpkg/installed/vcpkg/issue_body.md when reporting your issue.
Error: Process completed with exit code 1.

Unfortunately, my GitHub action does not expose the /usr/local/share/vcpkg/installed/vcpkg/issue_body.md, so I cannot attach it :/

Additional context

This just started failing about an hour ago. Earlier GitHub actions were completed successfully.

The text was updated successfully, but these errors were encountered:

Jik · 2023-01-30T19:53:09Z

I have a similar problem in my local computer with a different package, also just started today while everything was working 24 hours ago:

OS: Windows 10
vcpkg-tool version: 2023-01-24-8a88d63f241d391772fbde69af9cab96c3c64c75
vcpkg-scripts version: 91dd61b 2023-01-27 (3 days ago)

Just ran git pull and vcpkg update. Running vcpkg install glfw3 reports:

Downloading https://github.com/glfw/glfw/archive/7482de6071d21db77a7236155da44c172a7f6c9e.tar.gz
error: Failed to download from mirror set
error: File does not have the expected hash:
url: https://github.com/glfw/glfw/archive/7482de6071d21db77a7236155da44c172a7f6c9e.tar.gz
File: C:\dev\vcpkg\downloads\glfw-glfw-7482de6071d21db77a7236155da44c172a7f6c9e.tar.gz.20140.part
Expected hash: ec45b620338cf36a8dbdf7aaf54d7c3a49a1be4ae1a1ef95f1531094fec670870713969bbc23476769d374c7a71d93f6540ab64c46fb5f66f4402bb2d15c7d87
Actual hash: f2541cc801adb9a4d6ea6932c234953240d902212bd0a2d3a6f8d95c1dbff3adbd8e8e757ae991a544d979000c351c209d5e4054f71427d1a0490dbbfe0763fb

If I download the URL https://github.com/glfw/glfw/archive/7482de6071d21db77a7236155da44c172a7f6c9e.tar.gz with Chrome independently of vcpkg, the sha512sum of that file matches the "Actual hash" reported above.

Same thing also with vcpkg install openvr, but vcpkg install nlohmann-json seems to have worked.

dkonik · 2023-01-30T20:03:44Z

Having the same issue with multiple different packages.

willeccles · 2023-01-30T20:20:03Z

I am not using vcpkg, but I cannot find anywhere else on the internet that anyone else appears to be having this issue. CMake just spit out an error for a mismatched hash on nlohmann-json v3.10.4's archive using the same archive URL as the boost one in the OP. I have been using this URL with no issues until now (with this same hash). Perhaps GitHub has done something funny?

firthm01 · 2023-01-30T20:22:21Z

Same issue with several boost packages - windows and ubuntu tested so far
Same commit was compiling fine on Friday (using Github Actions)

willeccles · 2023-01-30T20:28:42Z

I've just opened a support ticket with GitHub. I suspect this is an issue on their end.

BillyONeal · 2023-01-30T20:54:52Z

I've just opened a support ticket with GitHub. I suspect this is an issue on their end.

They changed something about their gzipper

BillyONeal · 2023-01-30T20:58:39Z

I've just opened a support ticket with GitHub. I suspect this is an issue on their end.

Do you have a link for this? They have been a bit vocal in the past that "if you expect the hash to not change you need to get the bit from a download/artifact rather than from a source archive" so they may say "too bad" to us.

willeccles · 2023-01-30T21:00:49Z

I can't link to it as it's some sort of "personal ticket" or something associated with my account. Will let you know what they say, though.

Surprising that they are okay with that happening. I have been using the same hash for the same URL for a year and a half or so and not had any issues. Also, since the archive is listed as a release artifact, you'd think the same rules would apply there...

SuperElectron · 2023-01-30T21:17:18Z

our production builds are broken due to this issue

akaddoura · 2023-01-30T21:17:51Z

Same issue is happening here. Multiple packages are failing. Is it possible to forcibly change the hash keys as a temp fix?

cmazakas · 2023-01-30T21:29:51Z

Same issue is happening here. Multiple packages are failing. Is it possible to forcibly change the hash keys as a temp fix?

Technically, you can always just hard-edit the portfiles themselves to have the proper matching hashes.

mathstuf · 2023-01-30T21:32:23Z

Note that the /archive/ API endpoint has always been git archive under the hood. I don't believe that git archive has ever had reproducible guarantees about its behavior. Proper release entities with source archives should always be preferred where possible (please prod projects to provide them as well). Though given how hard it is to even get tags sometimes…

BillyONeal · 2023-01-30T21:36:12Z

bazel-contrib/SIG-rules-authors#11 (comment) GitHub folks commenting here

prince-chrismc · 2023-01-30T21:48:33Z

https://github.blog/changelog/2023-01-30-git-archive-checksums-may-change/ GitHub confirmed the changes

yanggaome · 2023-01-30T21:49:53Z

Our pipeline (Microsoft internal repo) failed...

BillyONeal · 2023-01-31T01:02:01Z

So when shall this be in effect?

We are waiting to see just as much as everyone else.

mathstuf · 2023-01-31T01:07:57Z

It is impractical for vcpkg to host such an archive 'automatically' because that would implicate us in "distribution" rules of whatever license applies to that archive.

I wasn't saying for vcpkg to rehost. I was asking if vcpkg (ideally in coordination with other distros) could push for releases to contain actual archives (even if just re-uploads of the /archive/ API at the time) instead of relying on the automatic /archive/ links.

BillyONeal · 2023-01-31T01:17:12Z

I wasn't saying for vcpkg to rehost. I was asking if vcpkg (ideally in coordination with other distros) could push for releases to contain actual archives (even if just re-uploads of the /archive/ API at the time) instead of relying on the automatic /archive/ links.

That only works if we are pointing to an actual release. In this case GitHub appears to be trying to revert the change so it should be automatically fixed.

bernhardmgruber · 2023-01-31T09:22:46Z

My CI seems to work again and vcpkg seems to download all dependencies correctly. My original issue is thus resolved. However, since the title has changed to raise attention to a more general problem, I leave it up to one of the maintainers here to close the issue. Thanks for keeping everyone up-to-date!

Neumann-A · 2023-01-31T10:59:07Z

So that people don't have to search:

Git community discussion: https://github.com/orgs/community/discussions/45830#discussioncomment-4828473

Bazel discussion: bazel-contrib/SIG-rules-authors#11

Git mailing list to make git archive reproducible: https://public-inbox.org/git/[email protected]/T/#r656f88c00f1250b5b5bb7fa1ad7b3fecd3592d79

BillyONeal · 2023-01-31T19:20:44Z

GitHub appears to have completed the revert. Thanks for the bug report and sorry for the disruption everyone!

Added the ability for the 'default' port version to get a value in case they all change at once (as almost happened due to microsoft#29288) microsoft#28356 did not apply post-source stubs for url and locale. I suspect they were just forgotten to be checked in? microsoft#28617 did not record the port version change in generate-ports.ps1. Also fixed all files to have LF line endings. (since changing boost-vcpkg-helper version also changed all the other ports anyway) Also changed "When changing this file also update and run scripts/boost/generate-ports.ps1" to "Automatically generated by scripts/boost/generate-ports.ps1" because I think that better conveys the consequences (your changes will get nuked).

Added the ability for the 'default' port version to get a value in case they all change at once (as almost happened due to #29288) #28356 did not apply post-source stubs for url and locale. I suspect they were just forgotten to be checked in? #28617 did not record the port version change in generate-ports.ps1. Also fixed all files to have LF line endings. (since changing boost-vcpkg-helper version also changed all the other ports anyway) Also changed "When changing this file also update and run scripts/boost/generate-ports.ps1" to "Automatically generated by scripts/boost/generate-ports.ps1" because I think that better conveys the consequences (your changes will get nuked).

bernhardmgruber added the category:port-bug The issue is with a library, which is something the port should already support label Jan 30, 2023

firthm01 mentioned this issue Jan 30, 2023

[boost-preprocessor] Build error: File does not have the expected hash #29293

Closed

mattdurak mentioned this issue Jan 30, 2023

[brotli] build failure #29290

Closed

This was referenced Jan 30, 2023

[glm] build failure #29289

Closed

[glfw3] Build error #29292

Closed

[glm] Updated the hash of the glm artifact #29291

Closed

BillyONeal changed the title ~~[boost-assert] Build error due to hash missmatch (Ubuntu 22.04 GHA)~~ All packages with sources hosted by GitHub currently do not download due to changes in GitHub's archives Jan 30, 2023

BillyONeal pinned this issue Jan 30, 2023

This was referenced Jan 30, 2023

error: Failed to download from mirror set, does not have the right hash #29296

Closed

[libjpeg-turbo] Build error #29297

Closed

BillyONeal mentioned this issue Jan 30, 2023

[draco] Build error #29298

Closed

AnticliMaxtic mentioned this issue Jan 30, 2023

[pugixml] fix SHA512 #29294

Closed

7 tasks

This was referenced Jan 30, 2023

[boost-config] error building x64-osx #29299

Closed

[json-c] Build error #29300

Closed

mathstuf mentioned this issue Jan 30, 2023

GitHub /archive/ tarballs hash changes spack/spack#35250

Closed

3 tasks

jjgalvez mentioned this issue Jan 31, 2023

Build issues due to vcpkg rustdesk/rustdesk#3013

Closed

LilyWangLL assigned Cheney-W Jan 31, 2023

MonicaLiu0311 mentioned this issue Jan 31, 2023

[gumbo] Build error #29307

Closed

BillyONeal mentioned this issue Jan 31, 2023

[zlib] Build error #29308

Closed

dg0yt mentioned this issue Jan 31, 2023

[clipper2] Add new port #29303

Merged

11 tasks

BillyONeal closed this as completed Jan 31, 2023

BillyONeal added category:infrastructure Pertaining to the CI/Testing infrastrucutre and removed category:port-bug The issue is with a library, which is something the port should already support labels Jan 31, 2023

BillyONeal unpinned this issue Jan 31, 2023

SuperElectron mentioned this issue Jan 31, 2023

vcpkg error with build Azure/azure-sdk-for-cpp#4310

Closed

BillyONeal mentioned this issue Jan 31, 2023

[boost] Fix generate-ports.ps1 to match the tree. #29338

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

All packages with sources hosted by GitHub currently do not download due to changes in GitHub's archives #29288

All packages with sources hosted by GitHub currently do not download due to changes in GitHub's archives #29288

bernhardmgruber commented Jan 30, 2023 •

edited

Loading

Jik commented Jan 30, 2023

dkonik commented Jan 30, 2023

willeccles commented Jan 30, 2023

firthm01 commented Jan 30, 2023

willeccles commented Jan 30, 2023

BillyONeal commented Jan 30, 2023

BillyONeal commented Jan 30, 2023

willeccles commented Jan 30, 2023

SuperElectron commented Jan 30, 2023

akaddoura commented Jan 30, 2023

cmazakas commented Jan 30, 2023

mathstuf commented Jan 30, 2023

BillyONeal commented Jan 30, 2023

prince-chrismc commented Jan 30, 2023

yanggaome commented Jan 30, 2023

BillyONeal commented Jan 31, 2023

mathstuf commented Jan 31, 2023 •

edited

Loading

BillyONeal commented Jan 31, 2023

bernhardmgruber commented Jan 31, 2023

Neumann-A commented Jan 31, 2023

BillyONeal commented Jan 31, 2023

All packages with sources hosted by GitHub currently do not download due to changes in GitHub's archives #29288

All packages with sources hosted by GitHub currently do not download due to changes in GitHub's archives #29288

Comments

bernhardmgruber commented Jan 30, 2023 • edited Loading

Jik commented Jan 30, 2023

dkonik commented Jan 30, 2023

willeccles commented Jan 30, 2023

firthm01 commented Jan 30, 2023

willeccles commented Jan 30, 2023

BillyONeal commented Jan 30, 2023

BillyONeal commented Jan 30, 2023

willeccles commented Jan 30, 2023

SuperElectron commented Jan 30, 2023

akaddoura commented Jan 30, 2023

cmazakas commented Jan 30, 2023

mathstuf commented Jan 30, 2023

BillyONeal commented Jan 30, 2023

prince-chrismc commented Jan 30, 2023

yanggaome commented Jan 30, 2023

BillyONeal commented Jan 31, 2023

mathstuf commented Jan 31, 2023 • edited Loading

BillyONeal commented Jan 31, 2023

bernhardmgruber commented Jan 31, 2023

Neumann-A commented Jan 31, 2023

BillyONeal commented Jan 31, 2023

bernhardmgruber commented Jan 30, 2023 •

edited

Loading

mathstuf commented Jan 31, 2023 •

edited

Loading