apk update - ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/main: Permission denied

[ronnicek]

Hi,

sorry, if this bug is reported towards the wrong repo.

We hit following errors in edge:

Alpine latest:
[root@dockerserver ~]# docker run -it --rm alpine:latest /bin/sh
/ # export http_proxy=http://<proxyserver>:3128
/ # export https_proxy=http://<proxyserver>:3128
/ # apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
v3.12.0-160-g96d355a76e [http://dl-cdn.alpinelinux.org/alpine/v3.12/main]
v3.12.0-161-g4b08580d3e [http://dl-cdn.alpinelinux.org/alpine/v3.12/community]
OK: 12749 distinct packages available
/ #
 
Alpine Edge:
[root@dockerserver ~]# docker run -it --rm alpine:edge /bin/sh
/ # export http_proxy=http://<proxyserver>:3128
/ # export https_proxy=http://<proxyserver>:3128
/ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
< after milion years later >
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/main: Permission denied
WARNING: Ignoring APKINDEX.e37b76c2.tar.gz: No such file or directory
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
< after another milion years later >
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/community: Permission denied
WARNING: Ignoring APKINDEX.d022dfc8.tar.gz: No such file or directory
2 errors; 14 distinct packages available

I read thru gliderlabs/docker-alpine#191, but we are not using authentication on proxy, so we cannot use any username/password anything. I tried with HTTP_PROXY_AUTH, but getting still the same.

[msitruk]

Got exaclty the same issue ! working for alpine:latest who use http and not alpine:edge who use https ...

[stevenylai]

Instead of running apk, can you try wget from your Dockerfile and see if there's any error? wget https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz

[jpmorin]

I have the same issue. I was going to create a new issues but stumble upon yours first. Here is the solution:

I learned from this issue that dl-cdn.alpinelinux.org does not support TLS at all

alpine:3.12 (latest as of august 24 2020)

$cat /etc/apk/repositories 
http://dl-cdn.alpinelinux.org/alpine/edge/main
http://dl-cdn.alpinelinux.org/alpine/edge/community

alpine:edge

$cat /etc/apk/repositories 
https://dl-cdn.alpinelinux.org/alpine/edge/main
https://dl-cdn.alpinelinux.org/alpine/edge/community

APK repositories url should be http only.

[ronnicek]

@jpmorin Kudos! Well done, but then bug is in alpine:edge docker image, since I am not changing repositories to https and they are there by default.

So really.. I put there RUN sed -i 's/https/http/' /etc/apk/repositories and it's working now.. but I feel it's a workaround more then the solution.

[jpmorin]

@ronnicek Yes, I did the same sed command in my Dockerfile to make it works.

FROM alpine:edge

RUN sed -i 's/https/http/' /etc/apk/repositories

RUN apk add curl

The thing is every previous version of alpine images had plain http repository urls. To be honest I don't know if it's a simple mistake that introduce the https urls or if it is a deliberate change to apk configuration...

[garw]

I do appreciate switching to https in general and I don't think that just replacing it again with http is a reasonable course of action.
The root cause seems to be a bug in libfetch (that apk internally uses to fetch data from http): https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220468

The bug ticket even has a patch attached that at least one commenter claims works. However, there seems to be no progress upstream. Maybe alpine can apply this on their libfetch version as an intermediate solution or push for it to get merged?

It also seems that this is now becoming more important as https is now the default in alpine:latest since Friday.

[fabled]

There has been multiple issues with libfetch and proxy + https. Hopefully those issues are fixed now in the latest apk-tools release which is pushed to edge. Could someone verify if apk 2.12.1 works?

[seriouz]

Editing all of our pipelines to replace https with http ist no real solution. Please upgrade to the new libfetch.

[devonwarren]

I also ran into this issue behind a corporate http proxy, not using proxy authentication but https traffic is decrypted/re-encrypted using an internal CA so I can understand why that wouldn't be verified. Here are the 2 solutions I found:

• Copy/override the /etc/ssl/certs/ca-certificates.crt with your internal CA chain file
• Run sed -i 's/https\:\/\//http\:\/\//g' /etc/apk/repositories (this may stop working if they enforce https only at mirror level)

It would be nice if the ca-certificates apk was installed by default so you could use that import the certs

[dHannasch]

* Copy/override the /etc/ssl/certs/ca-certificates.crt with your internal CA chain file

* Run `sed -i 's/https\:\/\//http\:\/\//g' /etc/apk/repositories` (this may stop working if they enforce https only at mirror level)

I'm using https://gitlab.com/shell-bootstrap-scripts/shell-bootstrap-scripts/-/blob/master/before_script.yaml#L243 to bootstrap by manually adding the certificate if the image doesn't have ca-certificates installed.

    - cat $PROXY_CA_PEM >> /etc/ssl/certs/ca-certificates.crt
    - if [ "$(tail -n $PROXY_CA_PEM_NUM_LINES /etc/ssl/certs/ca-certificates.crt)" != "$(cat $PROXY_CA_PEM)" ]; then false; fi

This allows apk add to work without messing with /etc/apk/repositories on alpine:3.13.1, but not on alpine:edge.

[maurizio-lattuada]

FYI, the issue referenced above related to libfetch seems it has been solved, at least when using Alpine 3.14.0

[minecraftchest1]

I am still seeing this issue.

[benispeti]

I can still see the same issue on Alpine 3.14.0. I'm behind a corporate proxy which doesn't require any authentication. Adding the internal certificate to /etc/ssl/certs/ca-certificates.crt didn't help me.

My Dockerfile:

FROM alpine:3.14
RUN apk update

This is the log what I can see:

#5 [2/2] RUN apk update
#5 sha256:a1332c9af190e6718585316bf5b8f7a6f06f6207cc6c4d67c961fbdd95725e5d
#5 0.503 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
#5 5.509 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
#5 5.509 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/main: temporary error (try again later)
#5 5.509 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/main: No such file or directory
#5 10.52 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/community: temporary error (try again later)
#5 10.52 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/community: No such file or directory
#5 10.52 2 errors; 14 distinct packages available
#5 ERROR: executor failed running [/bin/sh -c apk update]: exit code: 2

[FLAGLORD]

I can still see the same issue on Alpine 3.14.0. I'm behind a corporate proxy which doesn't require any authentication. Adding the internal certificate to /etc/ssl/certs/ca-certificates.crt didn't help me.

My Dockerfile:

FROM alpine:3.14
RUN apk update

This is the log what I can see:

#5 [2/2] RUN apk update
#5 sha256:a1332c9af190e6718585316bf5b8f7a6f06f6207cc6c4d67c961fbdd95725e5d
#5 0.503 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
#5 5.509 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
#5 5.509 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/main: temporary error (try again later)
#5 5.509 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/main: No such file or directory
#5 10.52 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/community: temporary error (try again later)
#5 10.52 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/community: No such file or directory
#5 10.52 2 errors; 14 distinct packages available
#5 ERROR: executor failed running [/bin/sh -c apk update]: exit code: 2

Have you solved this problem?

[rubenmanzano]

I can still see the same issue on Alpine 3.14.0. I'm behind a corporate proxy which doesn't require any authentication. Adding the internal certificate to /etc/ssl/certs/ca-certificates.crt didn't help me.
My Dockerfile:

FROM alpine:3.14
RUN apk update

This is the log what I can see:

#5 [2/2] RUN apk update
#5 sha256:a1332c9af190e6718585316bf5b8f7a6f06f6207cc6c4d67c961fbdd95725e5d
#5 0.503 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
#5 5.509 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
#5 5.509 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/main: temporary error (try again later)
#5 5.509 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/main: No such file or directory
#5 10.52 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/community: temporary error (try again later)
#5 10.52 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/community: No such file or directory
#5 10.52 2 errors; 14 distinct packages available
#5 ERROR: executor failed running [/bin/sh -c apk update]: exit code: 2

Have you solved this problem?

FROM alpine:3.14
RUN sed -i 's/https/http/' /etc/apk/repositories
RUN apk update

[Paraphraser]

Still happening - see:

• Discord IOTstack 2021-09-17
• Discord IOTstack 2021-09-09

"sed" patch seems to work.

[benispeti]

Finally this thing looks to be working on my end.

FROM alpine:3.15
ARG PROXY="<your proxy URL comes here>"
RUN sed -i 's/https/http/' /etc/apk/repositories
RUN export HTTP_PROXY=$PROXY && export HTTPS_PROXY=$PROXY && \
    apk update

[pbertin-oneex]

On my end too, without any changes to my setups (same dockerfile, same docker version, same host config, tested on arch / ubuntu / alpine), fetch works today in http and https (without the use of proxy in my case).
Http server issue / configuration then?

[Coderdude112]

What did you use for the proxy server @benispeti?

I'm still getting the error

[benispeti]

What did you use for the proxy server @benispeti?

I'm still getting the error

I had this issue behind my corporate proxy, so I used it's URL here. You should use your company's proxy URL.

[Coderdude112]

Gotcha, yeah I'm trying to do this at home and still having problems. Is this thread exclusive to having this problem behind a corporate proxy?

[Gandulf78]

Gotcha, yeah I'm trying to do this at home and still having problems. Is this thread exclusive to having this problem behind a corporate proxy?

same issue here on my home network

[Paraphraser]

Are you running Buster? If yes, have you upgraded libseccomp2? If no, see instructions.

There's background info here.

Also, Mosquitto seemed to need the sed patch for https to http. See [Dockerfile line 5](https://github.com/SensorsIot/IOTstack/blob/master/.templates/mosquitto/Dockerfile]. That's mentioned above here too so I'd try both.

[Gandulf78]

Are you running Buster? If yes, have you upgraded libseccomp2? If no, see instructions.

Finally I succeeded to update my docker version. It is ok now.

[skaldesh]

I have this error now, needed to replace https with http

FROM alpine:3.15 AS final

# Install dependencies.
RUN sed -i 's/https/http/' /etc/apk/repositories
RUN apk add --no-cache \
    ca-certificates \
    docker

No proxy being used, but we build using docker buildx

[kunlong-luo]

Add some code to the dockerfile to solve this problem

RUN sed -i ‘s/https/http/’ /etc/apk/repositories
RUN apk add curl

[OJFord]

I have the same problem on Arch with libseccomp v2.5.4, all alpine versions that I tested (3.10 through 3.16), with and without TLS, with buildkit and not. With both docker v20.10.17 and podman 4.1.1.

Not on a corporate network, no proxy, I'm at home. It works in GH Actions though.

[molitona]

same as you arch too.. @OJFord

[LeonardoHQ]

Hello! I faced the same problem on alpine:3.16 with Fedora 36 as host.
I found the solution here https://shaik-tech.blogspot.com/2018/07/docker-error-warning-ignoring-httpdl.html

I just created the daemon.json file (in the fedora host) with the following content

{
    "dns": ["8.8.8.8"]
}

in /etc/docker/daemon.json then restarted the docker service (just in case) with sudo systemctl restart docker.service and all good.

My Dockerfile looks like:

FROM alpine:3.16
RUN apk upgrade
RUN apk add python3 py3-pip neovim
etc
etc

Note that I do not use the sed trick.

[drrobotic]

to everyone who has still this problem and all suggested solutions dont work, please check your firewall rules.
opening ports for dns(53) and https(443) alone via OUTPUT chain seems not enough, i also had to put the accept rules into the DOCKER-USER chain, after that https requests inside container worked again. its better to test container connectivity with ping/wget, i think the error message "temporary error (try again later)" from apk is very confusing. after allowing dns only there comes a much better hint "network error (check Internet connection and firewall)".

[kyberorg]

@drrobotic agreed. In my case it was rule that redirected all incoming traffic on port 443 to another port. Problem was solved by specifying WAN interface only.

[SamuelLHuber]

I was able to fix the issue in a microk8s Kubernetes Cluster by specifying dnsPolicy and hostNetwork in the CronJob spec.

While normal pods created with kubectl run had no issues to run apk update the pod created by the cronjob did. That fixed it for me. Haven't validated if dnsPolicy alone would be enough, but maybe you don't need the hostNetwork: true

To learn what is actually happening check the Kubernetes Docs for dns-pod-service

spec:
  schedule: {{ .Values.backup.schedule | quote }}
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: "OnFailure"
          hostNetwork: true
          dnsPolicy: ClusterFirstWithHostNet

[r2evans]

I'm getting the permission-denied error. My firewall is fine, my docker setup is unchanged over many years. This is reproducible across different networks, though I'm finding it fails more when the CDN resolves to the 146.* addresses for dl-cdn.alpinelinux.org but not for the 151.* resolution. The below are within the 146.* area.

This inconsistency to me suggests it is not necessarily with the internal libraries or ca-certificates (I tested after adding that package, too). Could it be something with the CDN?

All in-alpine commands below are based on this container and cmdline:

root@franz:~# docker images | grep -E 'REPO|alpine'
REPOSITORY                                                          TAG                            IMAGE ID       CREATED         SIZE
alpine                                                              latest                         05455a08881e   3 weeks ago     7.38MB
root@myhost:~# docker run -it --rm alpine sh

/ # cat /etc/os-release 
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.19.1
PRETTY_NAME="Alpine Linux v3.19"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"

Inconsistency

The behavior is inconsistent and changes on each invocation (even within the same container).

/ # apk add --no-cache  ca-certificates
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
WARNING: fetching https://dl-cdn.alpinelinux.org/alpine/v3.19/main: Permission denied
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
WARNING: fetching https://dl-cdn.alpinelinux.org/alpine/v3.19/community: Permission denied
ERROR: unable to select packages:
  ca-certificates (no such package):
    required by: world[ca-certificates]

/ # apk add --no-cache  ca-certificates
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/1) Installing ca-certificates (20230506-r0)
Executing busybox-1.36.1-r15.trigger
Executing ca-certificates-20230506-r0.trigger
OK: 8 MiB in 16 packages

No delay between attempts (there were a half-dozen attempts before the first shown).

Checking certs

/ # sed -i s/https/http/ /etc/apk/repositories
/ # apk add openssl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/1) Installing openssl (3.1.4-r5)
Executing busybox-1.36.1-r15.trigger
OK: 8 MiB in 16 packages

With that temporary hack (I don't yet accept going http:// as a secure and acceptable final solution):

/ # openssl version                                                                                      
OpenSSL 3.1.4 24 Oct 2023 (Library: OpenSSL 3.1.4 24 Oct 2023)  

/ # echo | openssl s_client -connect dl-cdn.alpinelinux.org:443 -showcerts
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 324 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
This TLS version forbids renegotiation.
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

And in the same container, if I wait a moment and try again,

/ # echo | openssl s_client -connect dl-cdn.alpinelinux.org:443 -showcerts
CONNECTED(00000003)                                                                                      
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1                                                                                          
depth=1 C = US, O = Let's Encrypt, CN = R3                                                               
verify return:1                                                                                          
depth=0 CN = dl-cdn.alpinelinux.org
verify return:1
---
Certificate chain
 0 s:CN = dl-cdn.alpinelinux.org
   i:C = US, O = Let's Encrypt, CN = R3
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jan 29 16:57:18 2024 GMT; NotAfter: Apr 28 16:57:17 2024 GMT
-----BEGIN CERTIFICATE-----
MIIE+TCCA+GgAwIBAgISA1hz6FwMJ3EYQna99ui1aKbaMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
...

(A completely-valid SSL connection.)

Changing DNS

Changing to quad-8 for DNS resolution does not fix it.

/ # sed  -E 's/nameserver.*/nameserver 8.8.8.8/g' /etc/resolv.conf > /tmp/resolv.conf
/ # cat /tmp/resolv.conf > /etc/resolv.conf 
/ # cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 8.8.8.8

/ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.19/main: Permission denied
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.19/community: Permission denied
4 unavailable, 0 stale; 15 distinct packages available

wget failure

wget reports that the connection is reset by the peer?

/ # wget -S https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
Connecting to dl-cdn.alpinelinux.org (146.75.30.132:443)
ssl_client: SSL_connect
wget: error getting response: Connection reset by peer

[lhartmann]

Got similar errors, along with many SSL failues on curl and wget, using alpine 3.18 and 3.19.

POTENTIAL CAUSE: Missing ca-certificates and symlink /usr/lib/ssl/certs -> /etc/ssl/certs.

The Error

~ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
489BE4F0E87F0000:error:80000002:system library:file_open:No such file or directory:providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs)
489BE4F0E87F0000:error:80000002:system library:file_open:No such file or directory:providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs)
489BE4F0E87F0000:error:80000002:system library:file_open:No such file or directory:providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs)
489BE4F0E87F0000:error:80000002:system library:file_open:No such file or directory:providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs)
489BE4F0E87F0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1889:
WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.19/main: Permission denied
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
489BE4F0E87F0000:error:80000002:system library:file_open:No such file or directory:providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs)
489BE4F0E87F0000:error:80000002:system library:file_open:No such file or directory:providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs)
489BE4F0E87F0000:error:80000002:system library:file_open:No such file or directory:providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs)
489BE4F0E87F0000:error:80000002:system library:file_open:No such file or directory:providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs)
489BE4F0E87F0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1889:
WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.19/community: Permission denied
4 unavailable, 0 stale; 73 distinct packages available

The Fix

• Change /etc/apk/repositories from HTTPS to HTTP
• Install the ca-certificates package
• Run update-ca-certificates
• Change /etc/apk/repositories back from HTTP to HTTPS
• Create /usr/lib/ssl/
• Create symlink /usr/lib/ssl/certs -> /etc/ssl/certs

~ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
v3.19.1-169-g5ed82648e0c [https://dl-cdn.alpinelinux.org/alpine/v3.19/main]
v3.19.1-180-g5674b95fd09 [https://dl-cdn.alpinelinux.org/alpine/v3.19/community]
OK: 23060 distinct packages available
~ # 

[lordloc]

Up until today August 2024, I still have this problem.
This is the context:

• Our application is deployed on Kubernetes Pod
• This issue rarely happens, like 1-2 fail for 100 success.
• This is the error logs

│ yatai-66b05443037de60b6d934bdb--startup-mon--1-nv7nk + apk add --no-cache curl                                                                                                                                   
│ yatai-66b05443037de60b6d934bdb--startup-mon--1-nv7nk fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/main/x86_64/APKINDEX.tar.gz                                                                               
│ yatai-66b05443037de60b6d934bdb--startup-mon--1-nv7nk fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/community/x86_64/APKINDEX.tar.gz                                                                          
│ yatai-66b05443037de60b6d934bdb--startup-mon--1-nv7nk WARNING: fetching https://dl-cdn.alpinelinux.org/alpine/v3.20/main: Permission denied

Base image is a mirror image of docker.io/library/bash:latest, I guess we are using up-to-date version of that image as well.

Because this happens intermittently, I think it's mostly related to tls certificate verification, but the real root cause is still unknown, because the complexity of the network layers in the company.

[danny-huang-openfind]

I’m using Alpine 3.17 (node:18.16.0-alpine3.17), but I still occasionally encounter this problem, though it doesn't happen every time.

#13 [builder 2/7] RUN apk add --no #-cache --verbose     bash     git     && corepack enable     && yarn global add zx -D
#13 0.230 fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz
#13 15.69 fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz
#13 15.69 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.17/main: Permission denied
#13 17.10 ERROR: unable to select packages:
#13 17.15   bash (no such package):
#13 17.15     required by: world[bash]
#13 17.15   git (no such package):
#13 17.15     required by: world[git]
#13 ERROR: process "/bin/sh -c apk add --no-cache --verbose     bash     git     && corepack enable     && yarn global add zx -D" did not complete successfully: exit code: 2

[sanjiv-16]

using alpine:latest under corporate network, is this issue fixed? or any other walkaround other that switching back to HTTP

/usr/local/share # apk add curl
fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/main/x86_64/APKINDEX.tar.gz
28BB7CAA0C7F0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:2091:
WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.20/main: Permission denied
fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/community/x86_64/APKINDEX.tar.gz
28BB7CAA0C7F0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:2091:
WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.20/community: Permission denied
OK: 99 MiB in 103 packages

/usr/local/share # curl -o /tmp/test.tar https://dl-cdn.alpinelinux.org/alpine/v3.20/main/x86_64/APKINDEX.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  461k  100  461k    0     0   584k      0 --:--:-- --:--:-- --:--:--  584k
/usr/local/share #

[MrNocTV]

@sanjiv-16
I had no other choices than including the curl package in the Dockerfile as well.
The most annoying thing about this issue is you cannot tell exactly what goes wrong.
It happens intermittently, so the safest approach is to just download it beforehand (in the Dockerfile).

[sanjiv-16]

@MrNocTV

FROM alpine:latest
USER root
RUN apk --no-cache add ca-certificates
COPY ca-bundle.crt /usr/local/share/ca-certificates/ca-bundle.crt
RUN update-ca-certificates
RUN rm -rf /var/cache/apk/*
RUN apk --no-cache add curl

 => ERROR [2/5] RUN apk --no-cache add ca-certificates                                                                          0.8s
------
 > [2/5] RUN apk --no-cache add ca-certificates:
0.085 fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
0.503 28EB64B31E7F0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:2091:
0.503 WARNING: fetching https://dl-cdn.alpinelinux.org/alpine/edge/main: Permission denied
0.503 fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
0.785 28EB64B31E7F0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:2091:
0.785 WARNING: fetching https://dl-cdn.alpinelinux.org/alpine/edge/community: Permission denied
0.785 ERROR: unable to select packages:
0.785   ca-certificates (no such package):
0.785     required by: world[ca-certificates]

facing same issue while building, need to add RUN sed -i 's|https://|http://|' /etc/apk/repositories to fix this issue, but what about security
:(

[r2evans]

facing same issue while building, need to add RUN sed -i 's|https://|http://|' /etc/apk/repositories to fix this issue, but what about security :(

Full disclaimer: I am no longer using alpine, but I think to address your question about security @sanjiv-16 ...

• you can check the basic status (if not the identity) of the ssl certs regardless of the repo contents, perhaps something like:

  / # grep '^http' /etc/apk/repositories | head -n 1 | xargs wget --spider 2> /dev/null
  / # _ret=$?
  / # test $_ret -ne 0 && echo bad
  / # echo $_ret
  0

  if you're curious, this is a known-bad (for testing) wget hit:

  / # echo 'https://self-signed.badssl.com' | xargs wget --spider 2> /dev/null
  / # _ret=$?
  / # test $_ret -ne 0 && echo bad
  bad
  / # echo $_ret
  123

  (This is running in /bin/sh and uses only wget and test which are available in stock alpine:latest. Alpine's curl does not afaict provide additional functionality.)

• perhaps you can deduce that if the SSL cert is present and valid, then the server is therefore good and can be trusted; because of that, the same domain name server serving files over http:// might arguably be trusted, so the sed -i ... hack should be as safe as we can expect

• disclaimer: I have not tested this in production nor in the presence of the failing connects that initiated this issue, so I don't know that it will be a perfect canary ...

• WARNING, this check is rather shallow: it does not check that it is the expected ssl cert, so a bad actor could conceivably take over the server and have an otherwise valid (though different) SSL cert

I'm still rather surprised that the repo does not serve over https (nor serve them reliably). While I recognize that serving files over TLS technically "costs" more, I find it difficult to believe that there is not a way to use SSL to provide assurance about the source (e.g., SSL cert fingerprint). If a bad actor takes over (or masks as) this repo, then package poisoning is trivial, and SSL is one step to providing a verifiable source. (Note: more steps are needed to really arm against it, since a valid SSL doesn't mean the packages are untouched. But lack of SSL seems like a show-stopper for any reassurance.)