Unexpected error, unable to continue. Please report (gpg: no valid OpenPGP data found.)

[dabrahams]

Describe the bug
Please see this report

Workflow configuration (please complete the following information):

• Action version (uses): swift-actions/setup-swift@v1
• Platform (runs-on): ubuntu-latest
• Swift version (swift-version): 5.8

[nickpdemarco]

Gentle bump: this is breaking CI for Hylo.

[dabrahams]

I am no expert on the tech underlying this plugin, but to my layman's eye it looks like we are failing here trying to import non-existent GPG data. Which we are trying to download here. It strikes me that the download could fail and I don't see any code to detect that case. I note that this plugin never fails on MacOS, but then it also never has to download these GPG keys.

[dabrahams]

At least one alternative plugin fails in exactly the same way. Not sure what it means yet. I've never failed to get the keys file myself (so far)…

[dabrahams]

So far, this alternative swift-installation action doesn't seem to be causing these spurious failures. Another workaround I know of is to use a dev container in which you've pre-installed swift, making this step unnecessary.

[stevapple]

This is also breaking CI for Teco Core, which uses a merge queue that doesn’t allow manually retrying failed tests.

I decided to work around by disabling merge queue temporarily, but I hope this can be resolved ASAP🙏

[sanderploegsma]

Another workaround I know of is to use a dev container in which you've pre-installed swift, making this step unnecessary.

Oh wow, thanks for the tip! I am already using a devcontainer on one of my repositories that is running into this issue, never considered to reuse it in CI as well.

[sanderploegsma]

@dabrahams I've seen at least one workflow with slashmo/install-swift where it also failed to download the keys with the same error. However, because the install script in that action doesn't contain set -e it ignores the error and happily continues with verifying the signature (which also fails and is ignored) and finally it installs the toolchain.

So the underlying problem here seems to be that the signing key download occasionally fails for an unknown reason. Whatever that reason may be, handling it is an interesting discussion: should the action just skip the signature verification if this occurs?

[dabrahams]

Heh, not being aware of the nutty details of how shell scripts deal with errors without -e (beyond that they are easily ignored in that case), I had wondered whether @slashmo 's script was extremely carefully written to work within those rules or just carelessly overlooking error handling. I guess it's the latter. No, the action shouldn't skip signature verification; that's a security feature!

I think what this means is that there's currently no known reliable procedure for automating the installation of Swift on Linux and the only workable approach for github actions is to use a devcontainer.

[sanderploegsma]

Yes that was my conclusion as well, until I tried it and my CI build times increased more than tenfold (~1min with this action vs >12min with a devcontainer)…

I am interested to see why downloading the signing keys and signatures from swift.org is so inconsistent, since that seems to be the culprit.

[Nef10]

A run with debug log enabled looks like this for me:

##[debug]Evaluating condition for step: 'Install Swift'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Install Swift
##[debug]Loading inputs
##[debug]Loading env
Run swift-actions/[email protected]
  with:
    swift-version: 5.8.1
##[debug]Resolved range 5.8.1
##[debug]Found matching version 5.8.1
##[debug]isExplicit: 5.8.1
##[debug]explicit? true
##[debug]checking cache: /opt/hostedtoolcache/swift-Ubuntu/5.8.1/x64
##[debug]not found
##[debug]No matching installation found
##[debug]Fetching verification keys
##[debug]Downloading https://swift.org/keys/all-keys.asc
##[debug]Destination /home/runner/work/_temp/3249f1b8-c55a-42cf-9aec-94684a857815
##[debug]download complete
##[debug]Importing verification keys
/usr/bin/gpg --import /home/runner/work/_temp/3249f1b8-c55a-42cf-9aec-94684a857815
gpg: directory '/home/runner/.gnupg' created
gpg: keybox '/home/runner/.gnupg/pubring.kbx' created
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
Error: Unexpected error, unable to continue. Please report at https://github.com/swift-actions/setup-swift/issues
The process '/usr/bin/gpg' failed with exit code 2
Stacktrace:
Error: The process '/usr/bin/gpg' failed with exit code 2
    at ExecState._setResult (/home/runner/work/_actions/swift-actions/setup-swift/v1.24.0/dist/index.js:3295:25)
    at ExecState.CheckComplete (/home/runner/work/_actions/swift-actions/setup-swift/v1.24.0/dist/index.js:3278:18)
    at ChildProcess.<anonymous> (/home/runner/work/_actions/swift-actions/setup-swift/v1.24.0/dist/index.js:3172:27)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1100:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)
##[debug]Node Action run completed with exit code 1
##[debug]Finishing: Install Swift

So it looks like it downloaded something but no valid data?

Also, it's flaky for me and seems to work like 50% of the time - so maybe this download and key import can just be re-tried like 3 times?

[sanderploegsma]

I did some debugging of a failing workflow run using action-tmate and it turns out the swift.org server compresses the response, so the file being downloaded is actually gzipped, which is why it cannot be imported in gpg. The same happens in my terminal:

❯ curl -v --output all-keys.asc https://www.swift.org/keys/all-keys.asc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 17.253.53.204:443...
* Connected to www.swift.org (17.253.53.204) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
} [318 bytes data]
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* (304) (IN), TLS handshake, Unknown (8):
{ [21 bytes data]
* (304) (IN), TLS handshake, Certificate (11):
{ [3610 bytes data]
* (304) (IN), TLS handshake, CERT verify (15):
{ [80 bytes data]
* (304) (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* (304) (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=swift.org; O=Apple Inc.; ST=California; C=US
*  start date: Oct 25 18:09:36 2022 GMT
*  expire date: Nov 24 18:09:35 2023 GMT
*  subjectAltName: host "www.swift.org" matched cert's "www.swift.org"
*  issuer: CN=Apple Public Server ECC CA 12 - G1; O=Apple Inc.; ST=California; C=US
*  SSL certificate verify ok.
* using HTTP/1.1
> GET /keys/all-keys.asc HTTP/1.1
> Host: www.swift.org
> User-Agent: curl/7.88.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: Apple
< Date: Sun, 20 Aug 2023 18:33:51 GMT
< Content-Type: text/plain; charset=UTF-8
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Last-Modified: Fri, 18 Aug 2023 19:09:05 GMT
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=31536000; includeSubdomains
< Content-Encoding: gzip
< Cache-Control: max-age=180, public
< Accept-Ranges: bytes
< Content-Length: 10412
< Via: http/1.1 nlams2-edge-lx-007.ts.apple.com (acdn/84.14362), http/1.1 nlams2-edge-bx-014.ts.apple.com (acdn/84.14362)
< X-Cache: hit-fresh, hit-fresh
< CDNUUID: 9d0b2fa7-c23b-451d-9b8c-e55736742dc1-427031446
< Etag: "41c9-6033743275640"
< Age: 10
< Connection: keep-alive
< 
{ [1300 bytes data]
100 10412  100 10412    0     0   165k      0 --:--:-- --:--:-- --:--:--  221k
* Connection #0 to host www.swift.org left intact

❯ file all-keys.asc 
all-keys.asc: gzip compressed data, max speed, from Unix, original size modulo 2^32 16841

I tried forcing the server to send an uncompressed response by setting accept-encoding: identity and by setting accept-encoding: gzip;q=0 but both attempts still yielded a gzipped response.

What I don't get is why this happens intermittently. It's as if some of the swift.org servers are misconfigured and therefore some send a compressed response while others don't, which is why the action is able to succeed every now and then.

This also makes fixing the bug a little bit annoying: since there is no way of knowing whether the response is compressed or not, the action should first check the file information and then decompress if needed.

I suspect the same issue occurs when the action is able to download the signing keys but then fails to verify the signature file; probably the signature file response is gzipped as well.

[dabrahams]

Wow great sleuthing! Definitely worthy of a report to bugs.swift.org! Et voilà: swiftlang/swift#68047

[stevapple]

This also makes fixing the bug a little bit annoying: since there is no way of knowing whether the response is compressed or not, the action should first check the file information and then decompress if needed.

Nice catch! I wonder if it’s possible to send Accept-Encoding: gzip (though cURL flag --compressed) to force the server respond with a gzipped file?

Edit: It seemed to work! cURL is clever enough to identify if the response is gzipped, and to decompress the response body accordingly.

[sanderploegsma]

Nice catch! I wonder if it’s possible to send Accept-Encoding: gzip (though cURL flag --compressed) to force the server respond with a gzipped file?

Maybe, although you can never be sure whether the server respects the Accept-Encoding header (as it already ignores explicitly asking to not compress it).

Ideally the action should check the Content-Encoding response header, but since the current implementation downloads the file through @actions/tool-cache the response headers are not accessible. So it would either have to be refactored to use @actions/http-client directly or check whether the downloaded file is gzipped by checking the magic number (1f 8b) or something.

[sanderploegsma]

@dabrahams in the mean time if you're looking for another workaround that doesn't involve skipping the signature verification, you could consider using the swift Docker image in your action workflow:

name: Swift

on:
  pull_request:
  push:
    branches: [ "main" ]

jobs:
  test:
    runs-on: ubuntu-latest
    container: swift:5.8
    steps:
      - uses: actions/checkout@v3
      - name: Run tests
        run: swift test

This is as fast as using swift-actions/setup-swift, taking around 30s to initialize the container

[dabrahams]

@sanderploegsma Thanks that's an interesting possibility. We currently use a devcontainer because we need an LLVM-15+ installation and a custom .pc file, but the trade-off might be worth it.

[dabrahams]

I've seen at least one workflow with slashmo/install-swift where it also failed to download the keys with the same error. However, because the install script in that action doesn't contain set -e it ignores the error and happily continues with verifying the signature (which also fails and is ignored) and finally it installs the toolchain.

When I set up swift with that workflow, I am seeing other problems (failure to find libswiftGlibc.so), FWIW.

[dabrahams]

@dabrahams in the mean time if you're looking for another workaround that doesn't involve skipping the signature verification, you could consider using the swift Docker image in your action workflow:

@sanderploegsma fun fact: that image doesn't have node preinstalled, which means that most other actions fail. And to install nodejs requires

        apt update
        apt install -y curl
        curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
        apt install -y nodejs

which is slow… so I can't win(?)

[redsun82]

We were affected by this as well, and very frequently since yesterday. Following the excellent @sanderploegsma's investigation I tried this, and I can confirm it worked in our case.

I don't think it's really PR worthy (it probably does not work on Windows, it doesn't have any tests, and requires gunzip being installed instead of relying on some npm library), but at least it confirms what was found out previously on this issue. Maybe someone else can pick this up to do a proper PR? Typescript's not my strong suit, and I cannot devote too much time to it at the moment.

You can try it out using redsun82/setup-swift@b2b6f77ab14f6a9b136b520dc53ec8eca27d2b99, if it helps 🙂

[pfusik]

Since swift.org redirects to www.swift.org, why not fetch the latter in the first place? This might be pure luck, but it helped me.

[jtbandes]

I'm experiencing this issue as well... looks like by retrying a few times I managed to get past it. https://github.com/foxglove/mcap/actions/runs/8637068613/job/23679121390?pr=1153

[fwcd]

Ran into this pretty frequently in a new Swift package... would love to see @redsun82's fix upstream. Even if it doesn't work on Windows, we could just scope it to Unix-ish platforms for now. Anything would be better than having CI fail randomly.