Error installing from official repos onto Ubuntu #3219

ganeshgunasekaran · 2023-08-18T12:36:29Z

Hi,

I am tried installing the kubeadm in Ubuntu 22.4 LTS following the instructions given in the page https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
I was able follow the steps and successfully installed Containerd .
The commands worked fine till Installing kubeadm -> Debian-based distributions -> Add the appropriate Kubernetes apt repository
The next step showed error while running "sudo apt-get update"

Please find the command outputs attached as screenshots Step 1-2 and step 3-4.

This if my first issue. Please correct me if I have should have done something else.

k8s-ci-robot · 2023-08-18T12:36:36Z

This issue is currently awaiting triage.

SIG Docs takes a lead on issue triage for this website, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sftim · 2023-08-18T13:18:04Z

Thanks for reporting this
/retitle Error (missing repository signature) installing from official repos onto Ubuntu
/kind bug

/sig release
(and specifically @xmudrii) might like to know about this

sftim · 2023-08-18T13:20:02Z

@ganeshgunasekaran, what happens when you run this:
curl --verbose -i -L https://pkgs.k8s.io/core:/stable:/v1.28/deb/InRelease

The error output, if there is one, might help you understand what to fix.

ganeshgunasekaran · 2023-08-19T10:27:08Z

Thanks a lot for picking up this issue. @sftim .
Please find the output of the command pasted below

ubuntu1@ubuntu1:$ cat curl_run.sh
curl --verbose -i -L https://pkgs.k8s.io/core:/stable:/v1.28/deb/
ubuntu1@ubuntu1:$ ./curl_run.sh > curl_run.out 2>&1
ubuntu1@ubuntu1:~$ cat curl_run.out
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 34.107.204.206:443...

Connected to pkgs.k8s.io (34.107.204.206) port 443 (#0)
ALPN, offering h2
ALPN, offering http/1.1
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
TLSv1.0 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
TLSv1.2 (IN), TLS header, Certificate Status (22):
{ [5 bytes data]
TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
TLSv1.2 (IN), TLS header, Finished (20):
{ [5 bytes data]
TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [15 bytes data]
TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [5302 bytes data]
TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
TLSv1.2 (OUT), TLS header, Finished (20):
} [5 bytes data]
TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
ALPN, server accepted to use h2
Server certificate:
subject: CN=k8s.io
start date: Jul 27 22:56:09 2023 GMT
expire date: Oct 25 23:32:07 2023 GMT
subjectAltName: host "pkgs.k8s.io" matched cert's "pkgs.k8s.io"
issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1D4
SSL certificate verify ok.
Using HTTP2, server supports multiplexing
Connection state changed (HTTP/2 confirmed)
Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
Using Stream ID: 1 (easy handle 0x5602ae8bbe90)
TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]

GET /core:/stable:/v1.28/deb/ HTTP/2
Host: pkgs.k8s.io
user-agent: curl/7.81.0
accept: /

TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [267 bytes data]
TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [267 bytes data]
old SSL session ID is stale, removing
TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
< HTTP/2 302
< server: nginx
< date: Sat, 19 Aug 2023 10:26:18 GMT
< content-type: text/html
< content-length: 138
< location: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/
< via: 1.1 google
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
<
TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
Ignoring the response-body
{ [138 bytes data]
TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
100 138 100 138 0 0 363 0 --:--:-- --:--:-- --:--:-- 365
Connection #0 to host pkgs.k8s.io left intact
Issue another request to this URL: 'https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/'
Trying 13.224.181.102:443...
Connected to prod-cdn.packages.k8s.io (13.224.181.102) port 443 (Initial commit of release automation tooling. #1)
ALPN, offering h2
ALPN, offering http/1.1
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
TLSv1.0 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
TLSv1.2 (IN), TLS header, Certificate Status (22):
{ [5 bytes data]
TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
TLSv1.2 (IN), TLS header, Finished (20):
{ [5 bytes data]
TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [4972 bytes data]
TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
TLSv1.2 (OUT), TLS header, Finished (20):
} [5 bytes data]
TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
ALPN, server accepted to use h2
Server certificate:
subject: CN=prod-cdn.packages.k8s.io
start date: Jul 12 00:00:00 2023 GMT
expire date: Aug 9 23:59:59 2024 GMT
subjectAltName: host "prod-cdn.packages.k8s.io" matched cert's "prod-cdn.packages.k8s.io"
issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M02
SSL certificate verify ok.
Using HTTP2, server supports multiplexing
Connection state changed (HTTP/2 confirmed)
Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
Using Stream ID: 1 (easy handle 0x5602ae8bbe90)
TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]

GET /repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/ HTTP/2
Host: prod-cdn.packages.k8s.io
user-agent: curl/7.81.0
accept: /

TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [124 bytes data]
TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
< HTTP/2 403
< content-type: application/xml
< date: Sat, 19 Aug 2023 10:26:19 GMT
< server: AmazonS3
< x-cache: Error from cloudfront
< via: 1.1 291933b5bb7fbb03efd999a83bb9696a.cloudfront.net (CloudFront)
< x-amz-cf-pop: SYD1-C2
< x-amz-cf-id: uQ8T7zHTABIpTKNf3DqSYdISb8s1W96gMF5itC632lLxKQ0chyl3uQ==
<
{ [255 bytes data]
TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
100 255 0 255 0 0 126 0 --:--:-- 0:00:02 --:--:-- 2040
Connection Launch blockers website#1 to host prod-cdn.packages.k8s.io left intact
HTTP/2 302
server: nginx
date: Sat, 19 Aug 2023 10:26:18 GMT
content-type: text/html
content-length: 138
location: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/
via: 1.1 google
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

HTTP/2 403
content-type: application/xml
date: Sat, 19 Aug 2023 10:26:19 GMT
server: AmazonS3
x-cache: Error from cloudfront
via: 1.1 291933b5bb7fbb03efd999a83bb9696a.cloudfront.net (CloudFront)
x-amz-cf-pop: SYD1-C2
x-amz-cf-id: uQ8T7zHTABIpTKNf3DqSYdISb8s1W96gMF5itC632lLxKQ0chyl3uQ==

AccessDeniedAccess DeniedXB235ZQQ726WQQHVBV9V/HkvxgZtzhn+aRTfpP3Yh03zGoj3NEbm/dboQddDwSSXW1WTHX4GApdKt50YgIB/jFL2IaP1dMIBAsgQ6g==ubuntu1@ubuntu1:~$

sftim · 2023-08-19T17:25:59Z

Looks like the docs are correct.

/transfer kubernetes

sftim · 2023-08-19T17:27:32Z

/retitle Error (403 Forbidden) installing from official repos onto Ubuntu
/triage needs-information

sftim · 2023-08-19T17:28:07Z

@ganeshgunasekaran what happens when you run this exact command:
curl --verbose -i -L https://pkgs.k8s.io/core:/stable:/v1.28/deb/InRelease

?

ganeshgunasekaran · 2023-08-21T08:00:24Z

Hi @sftim ,
Sorry about the mistake. Please find the response below.
ubuntu1@ubuntu1:~$ curl --verbose -i -L https://pkgs.k8s.io/core:/stable:/v1.28/deb/InRelease

Trying 34.107.204.206:443...
Connected to pkgs.k8s.io (34.107.204.206) port 443 (#0)
ALPN, offering h2
ALPN, offering http/1.1
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
TLSv1.0 (OUT), TLS header, Certificate Status (22):
TLSv1.3 (OUT), TLS handshake, Client hello (1):
TLSv1.2 (IN), TLS header, Certificate Status (22):
TLSv1.3 (IN), TLS handshake, Server hello (2):
TLSv1.2 (IN), TLS header, Finished (20):
TLSv1.2 (IN), TLS header, Supplemental data (23):
TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
TLSv1.3 (IN), TLS handshake, Certificate (11):
TLSv1.3 (IN), TLS handshake, CERT verify (15):
TLSv1.3 (IN), TLS handshake, Finished (20):
TLSv1.2 (OUT), TLS header, Finished (20):
TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
TLSv1.2 (OUT), TLS header, Supplemental data (23):
TLSv1.3 (OUT), TLS handshake, Finished (20):
SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
ALPN, server accepted to use h2
Server certificate:
subject: CN=k8s.io
start date: Jul 27 22:56:09 2023 GMT
expire date: Oct 25 23:32:07 2023 GMT
subjectAltName: host "pkgs.k8s.io" matched cert's "pkgs.k8s.io"
issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1D4
SSL certificate verify ok.
Using HTTP2, server supports multiplexing
Connection state changed (HTTP/2 confirmed)
Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
TLSv1.2 (OUT), TLS header, Supplemental data (23):
TLSv1.2 (OUT), TLS header, Supplemental data (23):
TLSv1.2 (OUT), TLS header, Supplemental data (23):
Using Stream ID: 1 (easy handle 0x560f5e3b0e90)
TLSv1.2 (OUT), TLS header, Supplemental data (23):

GET /core:/stable:/v1.28/deb/InRelease HTTP/2
Host: pkgs.k8s.io
user-agent: curl/7.81.0
accept: /

TLSv1.2 (IN), TLS header, Supplemental data (23):
TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
old SSL session ID is stale, removing
TLSv1.2 (IN), TLS header, Supplemental data (23):
TLSv1.2 (OUT), TLS header, Supplemental data (23):
TLSv1.2 (IN), TLS header, Supplemental data (23):
TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 302
HTTP/2 302
< server: nginx
server: nginx
< date: Mon, 21 Aug 2023 07:56:26 GMT
date: Mon, 21 Aug 2023 07:56:26 GMT
< content-type: text/html
content-type: text/html
< content-length: 138
content-length: 138
< location: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/InRelease
location: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/InRelease
< via: 1.1 google
via: 1.1 google
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

<

TLSv1.2 (IN), TLS header, Supplemental data (23):
Ignoring the response-body
TLSv1.2 (IN), TLS header, Supplemental data (23):
Connection #0 to host pkgs.k8s.io left intact
Issue another request to this URL: 'https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/InRelease'
Trying 13.224.181.114:443...
Connected to prod-cdn.packages.k8s.io (13.224.181.114) port 443 (Initial commit of release automation tooling. #1)
ALPN, offering h2
ALPN, offering http/1.1
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
TLSv1.0 (OUT), TLS header, Certificate Status (22):
TLSv1.3 (OUT), TLS handshake, Client hello (1):
TLSv1.2 (IN), TLS header, Certificate Status (22):
TLSv1.3 (IN), TLS handshake, Server hello (2):
TLSv1.2 (IN), TLS header, Finished (20):
TLSv1.2 (IN), TLS header, Supplemental data (23):
TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
TLSv1.2 (IN), TLS header, Supplemental data (23):
TLSv1.3 (IN), TLS handshake, Certificate (11):
TLSv1.2 (IN), TLS header, Supplemental data (23):
TLSv1.3 (IN), TLS handshake, CERT verify (15):
TLSv1.2 (IN), TLS header, Supplemental data (23):
TLSv1.3 (IN), TLS handshake, Finished (20):
TLSv1.2 (OUT), TLS header, Finished (20):
TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
TLSv1.2 (OUT), TLS header, Supplemental data (23):
TLSv1.3 (OUT), TLS handshake, Finished (20):
SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
ALPN, server accepted to use h2
Server certificate:
subject: CN=prod-cdn.packages.k8s.io
start date: Jul 12 00:00:00 2023 GMT
expire date: Aug 9 23:59:59 2024 GMT
subjectAltName: host "prod-cdn.packages.k8s.io" matched cert's "prod-cdn.packages.k8s.io"
issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M02
SSL certificate verify ok.
Using HTTP2, server supports multiplexing
Connection state changed (HTTP/2 confirmed)
Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
TLSv1.2 (OUT), TLS header, Supplemental data (23):
TLSv1.2 (OUT), TLS header, Supplemental data (23):
TLSv1.2 (OUT), TLS header, Supplemental data (23):
Using Stream ID: 1 (easy handle 0x560f5e3b0e90)
TLSv1.2 (OUT), TLS header, Supplemental data (23):

GET /repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/InRelease HTTP/2
Host: prod-cdn.packages.k8s.io
user-agent: curl/7.81.0
accept: /

TLSv1.2 (IN), TLS header, Supplemental data (23):
TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
TLSv1.2 (IN), TLS header, Supplemental data (23):
Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
TLSv1.2 (OUT), TLS header, Supplemental data (23):
TLSv1.2 (IN), TLS header, Supplemental data (23):
TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
HTTP/2 200
< content-type: application/octet-stream
content-type: application/octet-stream
< content-length: 1186
content-length: 1186
< last-modified: Tue, 15 Aug 2023 17:06:48 GMT
last-modified: Tue, 15 Aug 2023 17:06:48 GMT
< x-amz-server-side-encryption: AES256
x-amz-server-side-encryption: AES256
< x-amz-meta-mtime: 1692119134.983891584
x-amz-meta-mtime: 1692119134.983891584
< accept-ranges: bytes
accept-ranges: bytes
< server: AmazonS3
server: AmazonS3
< date: Mon, 21 Aug 2023 02:06:07 GMT
date: Mon, 21 Aug 2023 02:06:07 GMT
< etag: "63d4fc87d6c0e8739c65f55f4d8aa600"
etag: "63d4fc87d6c0e8739c65f55f4d8aa600"
< vary: Accept-Encoding
vary: Accept-Encoding
< x-cache: Hit from cloudfront
x-cache: Hit from cloudfront
< via: 1.1 41f4e34e5d78c923aead0fa16ff91eb8.cloudfront.net (CloudFront)
via: 1.1 41f4e34e5d78c923aead0fa16ff91eb8.cloudfront.net (CloudFront)
< x-amz-cf-pop: SYD1-C2
x-amz-cf-pop: SYD1-C2
< x-amz-cf-id: jjbla7ya0KBxnHn2fnwxqV8vxcHGlaM-08_ZXBfnCFDLDslruglVwg==
x-amz-cf-id: jjbla7ya0KBxnHn2fnwxqV8vxcHGlaM-08_ZXBfnCFDLDslruglVwg==
< age: 21574
age: 21574

<
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Archive: deb
Codename: deb
Origin: obs://build.opensuse.org/isv:kubernetes:core:stable:v1.28/deb
Label: isv:kubernetes:core:stable:v1.28
Architectures: amd64 arm64 s390x ppc64el
Date: Tue Aug 15 17:05:34 2023
Description: Kubernetes v1.28 (Stable) (deb)
MD5Sum:
61f5b9d38a31b1f3213816c8dfb3e85a 11872 Packages
5804a31770caf87fbd6beb6c5c53920b 2759 Packages.gz
SHA1:
3245f8ed9874d24198d54f94bb5eca770f8cc11a 11872 Packages
e13e2873d2d11928b642410f7bd28db092aa7796 2759 Packages.gz
SHA256:
a8ec729af2342f13728bcd0e93b9dc3e512025972f6a8a3778f6cf8bd12c6c40 11872 Packages
0a9f6e3a6f234d021c1098a59f326f1abedc31f63b8d297c738a98cc4413bc8c 2759 Packages.gz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)

iQEVAwUBZNuwXiNGVNqaKWQ2AQiSRQf/V/9DWBbLR54XkMrtohORmPmrVdcBRTlS
LzzTjxC0jlwznQf4GpLwnruKdseruohFJZ8obEF9+7bst55gbRISs5iqa92UfduB
sM6K8tTpw9HLOvQbBTjYcn7G7LgF7qtbJHFRoPteCbZDcfQsFeCrw+3c3T4Vt2+A
XE2seMnvUgJFk8Pj5bD7182OTeK+wuH7j8F6VPhY+DWUTG8mtKPuRL3nfZX7HA1p
6Nxo9NQSdGRso8han1ocJkhUv90AYBfw3vKIZQRApKcvANIZvcVi0Iyd2q7saztg
j5ly3NRwPpGsu+ObDbdknzYpcXm+Prmg9Z1qX4L/wg3kkg7yHRUI1w==
=DV5/
-----END PGP SIGNATURE-----

TLSv1.2 (IN), TLS header, Supplemental data (23):
Connection Unit test coverage in Kubelet is lousy. (~30%) kubernetes#1 to host prod-cdn.packages.k8s.io left intact
ubuntu1@ubuntu1:~$

sftim · 2023-08-21T08:07:17Z

/retitle Error installing from official repos onto Ubuntu

Feels like a support query; I'm not sure
/remove-triage needs-information

(for now)

sftim · 2023-08-21T10:02:44Z

/retitle Error installing from official repos onto Ubuntu

xmudrii · 2023-08-23T15:26:26Z

/transfer release
/assign
I'll be taking a look into this issue.

xmudrii · 2023-09-05T13:04:42Z

/triage accepted

dominic-p · 2023-09-09T21:31:43Z

For what it's worth, I'm also observing the "Access Denied" issue above. I'm not very familiar with how an APT repo is supposed to work, but most of the ones I have used allow browsing available packages (e.g. https://apt.kubernetes.io/).

xmudrii · 2023-09-09T23:14:14Z

@dominic-p What cloud provider or infrastructure are you using? For example, AWS tends to block access from some Hetzner IP addresses and we're aware of that, but there's nothing that we can do about that. Speaking of browsing the repo, this is not supported via pkgs.k8s.io, but you can do that via download.opensuse.org: https://download.opensuse.org/repositories/isv:/kubernetes:/

dominic-p · 2023-09-10T21:20:17Z

Thanks for the quick reply. That is very interesting as I am using Hetzner, and I've been struggling with network issues in a lot of areas. I assumed it was a misconfiguration on my end, but maybe my IPs are blacklisted.

Just so I'm clear, it is expected behavior to get "Access Denied" when visiting https://pkgs.k8s.io/core:/stable:/v1.28/deb/ correct?

I originally came here because after switching from the Google repos to the community ones, my Hetzner load balancers no longer get any targets. The exact same config/version works when I install from the Google repos. I can open an issue on Hetzner's CCM, but I thought I would check here first. Are there any differences in the actual packages between the Google repos and the community ones?

xmudrii · 2023-09-11T08:34:57Z

Just so I'm clear, it is expected behavior to get "Access Denied" when visiting https://pkgs.k8s.io/core:/stable:/v1.28/deb/ correct?

Yes, it's expected. We don't have a file browser at pkgs.k8s.io.

Are there any differences in the actual packages between the Google repos and the community ones?

There should be no major differences. For both the Google repos and the community repos, we use the same binaries (e.g. kubelet, kubeadm...). Have you checked kubelet logs to make sure that kubelet is installed and running?

dominic-p · 2023-09-12T07:20:15Z

Thanks for the confirmation. Yes, the kubelet is running. I checked the logs, and I didn't see anything that looked out of the ordinary. It seems to have trouble connecting to the CRI-O unix socket for a bit at startup and then everything looks good. I'll take a look at the .deb files from each repo to see if I can see any differences.

xmudrii · 2023-09-12T08:12:07Z

@dominic-p Sounds good and please let us know if you find any difference. We tried our best to match these debs, and if there's any difference, it would be good to address it if it's possible.

dominic-p · 2023-09-12T20:37:52Z

Ok, I downloaded 1.28 kubelet debs from both the google repo and the community repo just now using a fresh Debian v12 container running on my cluster. I haven't looked deeply into the packages yet, but there are definitely some differences in the packages I downloaded.

$ tree
.
├── community
│   ├── etc
│   │   ├── kubernetes
│   │   │   └── manifests
│   │   └── sysconfig
│   │       └── kubelet
│   ├── lib
│   │   └── systemd
│   │       └── system
│   │           └── kubelet.service
│   ├── usr
│   │   ├── bin
│   │   │   └── kubelet
│   │   └── share
│   │       └── doc
│   │           └── kubelet
│   │               ├── LICENSE
│   │               └── README.md
│   └── var
│       └── lib
│           └── kubelet
└── google
    ├── lib
    │   └── systemd
    │       └── system
    │           └── kubelet.service
    └── usr
        └── bin
            └── kubelet

Here are the steps I did to get the debs:

Community

Create new Debian 12 container
Run apt update followed by apt install curl gpg
Add community apt repo:

curl -fsSL "https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key" | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list

Download kubelet: apt download kubelet: kubelet_1.28.1-1.1_amd64.deb

Google

Create new Debian 12 container
Run apt update followed by apt install curl gpg
Add Google apt repo:

curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list

Download kubelet: apt download "kubelet=1.28.*": kubelet_1.28.1-00_amd64.deb

xmudrii · 2023-09-12T22:06:03Z

@dominic-p The new packages have additional files that we didn't have in packages published to the legacy Google-hosted repositories. It's important that we don't have missing files, i.e. that we installed some file with old packages, but that we don't do it with new packages.

dominic-p · 2023-09-13T06:20:56Z

Thanks for the explanation there. Ok, I think I was able to find the issue. The Hetzner CCM currently requires --cloud-provider=external to be set in KUBELET_EXTRA_ARGS. My configuration script sets the env variable in /etc/default/kubelet and the community package includes a new file /etc/sysconfig/kubelet with the contents KUBELET_EXTRA_ARGS=. That resets the env variable and breaks the Hetzner CCM.

I guess that makes this a bug with my particular configuration (I've already worked around it), but it is strange to me that the env variable is set in the new deb package when it wasn't before.

xmudrii · 2023-09-15T17:09:29Z

There's another report relevant to /etc/default/kubelet: #3276
We'll be taking a look into this as soon as possible.

AnirudhPanchangam · 2023-09-21T12:28:37Z

Hi Team,

I am experiencing a similar issue.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb InRelease [1,186 B]
Err:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb InRelease
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 234654DA9A296436
Reading package lists... Done
W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 234654DA9A296436
E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.28/deb InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.

I've run the same commands that OP has run. However, i get NO_PUBKEY issue.
I tried running
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 234654DA9A296436
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
Executing: /tmp/apt-key-gpghome.1PwUILV6CQ/gpg.1.sh --keyserver keyserver.ubuntu.com --recv-keys 234654DA9A296436
gpg: key 234654DA9A296436: public key "isv:kubernetes OBS Project isv:[email protected]" imported
gpg: Total number processed: 1
gpg: imported: 1

However, i still get the same error upon running apt-get update
This is an error that i have started getting recently. It used to work just fine up until yesterday.

Please let me know if there is anything else i can try.

Thanks,
Anirudh

nethershaw · 2023-09-22T03:12:09Z

I'm seeing this 403 response both from within AWS on EC2 instances and from my own system at home on a standard cable ISP. Same error from CloudFront.

I can see it's just CloudFront -> S3. Consider checking your bucket policy, and remember that S3 returns HTTP 403 not just for access denied, but also when it would return HTTP 404. It does this on purpose so that unauthenticated users cannot use return codes to tell what files exist in the bucket. If you are looking for a permissions/policy problem, but it is actually a pathing problem, this behavior will conceal it.

Given the above... it seems the entire problem is this:

https://pkgs.k8s.io/core:/stable:/v1.28/deb/InRelease exists.
https://pkgs.k8s.io/core:/stable:/v1.28/deb/dists/jammy/InRelease does not exist.
Apt assumes the latter should exist when given https://pkgs.k8s.io/core:/stable:/v1.28/deb.

xmudrii · 2023-09-23T09:27:11Z

@AnirudhPanchangam @nethershaw At this time, please use the official instructions for adding the repository (e.g. https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl).

lucasmo · 2023-10-05T15:49:01Z

Hi Team,

I am experiencing a similar issue. Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb InRelease [1,186 B] Err:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 234654DA9A296436 Reading package lists... Done W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 234654DA9A296436 E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.28/deb InRelease' is not signed. N: Updating from such a repository can't be done securely, and is therefore disabled by default.

I've run the same commands that OP has run. However, i get NO_PUBKEY issue.

Are you running Ubuntu 20.04 by chance? I had to run these commands and now the NO_PUBKEY issue went away:

sudo chmod 755 /etc/apt/keyrings
sudo chmod 644 /etc/apt/keyrings/kubernetes-apt-keyring.gpg

A failure like "can't read signed-by key" or something would have been more helpful, apt.

xmudrii · 2023-10-11T08:33:08Z

Hey folks!

This issue now contains multiple distinct issues:

403 Forbidden trying to browse files: we don't have a file browser and the underlying S3 bucket is private, so accessing URLs of directories such as https://pkgs.k8s.io/core:/stable:/v1.28/deb is not going to work. Add file browser for pkgs.k8s.io #3317 has been created to track this
403 Forbidden trying to install packages: some cloud providers have IP addresses blocked by AWS. Hetzner is often affected by this. There is nothing that we can do about this, the same issue is affecting registry.k8s.io, and we can't make AWS to unblock the affected IP addresses. You can only try to get a new IP address that's not blocked or to mirror the repository somewhere else
The issue with /etc/default/kubelet reported by @dominic-p is already fixed as part of Fix EnvironmentFile path for kubeadm deb package #3279
For issues like can't read signed-by key, we strongly recommend setting up repositories as described in the official guidelines: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl
About Apt assumes the latter should exist when given https://pkgs.k8s.io/core:/stable:/v1.28/deb reported by @nethershaw: we couldn't reproduce this issue, please try setting up the repository as described in the document that I linked previously. If the issue still appears, please create a new issue in this repository

Given that this issue contains multiple different reports and is hard to navigate, I'll go ahead and lock it. If you run into any issue and it's not already covered by this issue/comment, please create a new issue in this repository.

k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Aug 18, 2023

k8s-ci-robot changed the title ~~Installing kubeadm~~ Error (missing repository signature) installing from official repos onto Ubuntu Aug 18, 2023

k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. sig/release Categorizes an issue or PR as relevant to SIG Release. labels Aug 18, 2023

k8s-ci-robot transferred this issue from kubernetes/website Aug 19, 2023

k8s-ci-robot changed the title ~~Error (missing repository signature) installing from official repos onto Ubuntu~~ Error (403 Forbidden) installing from official repos onto Ubuntu Aug 19, 2023

k8s-ci-robot added the triage/needs-information Indicates an issue needs more information in order to work on it. label Aug 19, 2023

k8s-ci-robot changed the title ~~Error (403 Forbidden) installing from official repos onto Ubuntu~~ Error installing from official repos onto Ubuntu Aug 21, 2023

k8s-ci-robot assigned xmudrii Aug 23, 2023

k8s-ci-robot transferred this issue from kubernetes/kubernetes Aug 23, 2023

k8s-ci-robot added the needs-priority label Aug 23, 2023

AverageMarcus mentioned this issue Sep 4, 2023

Install Kubernetes artifacts from pkgs.k8s.io kubernetes-sigs/image-builder#1280

Merged

xmudrii added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Sep 5, 2023

k8s-ci-robot removed the needs-priority label Sep 5, 2023

k8s-ci-robot added the triage/accepted Indicates an issue or PR is ready to be actively worked on. label Sep 5, 2023

xmudrii added this to SIG Release - Packaging Sep 5, 2023

xmudrii moved this to 🔖 Backlog in SIG Release - Packaging Sep 5, 2023

This was referenced Sep 13, 2023

CA: Hetzner cloud-init example may need to be updated for new community packages kubernetes/autoscaler#6106

Closed

Readme update for new community pacakges hetznercloud/hcloud-cloud-controller-manager#513

Closed

xmudrii mentioned this issue Sep 15, 2023

deb packages for kubelet on pkgs.k8s.io seem to include a systemd service definition for redhat machines #3276

Closed

xmudrii mentioned this issue Sep 18, 2023

Fix EnvironmentFile path for kubeadm deb package #3279

Merged

xmudrii mentioned this issue Oct 11, 2023

Add file browser for pkgs.k8s.io #3317

Open

kubernetes locked as resolved and limited conversation to collaborators Oct 11, 2023

xmudrii closed this as completed Oct 11, 2023

github-project-automation bot moved this from 🔖 Backlog to ✅ Done in SIG Release - Packaging Oct 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error installing from official repos onto Ubuntu #3219

Error installing from official repos onto Ubuntu #3219

ganeshgunasekaran commented Aug 18, 2023

k8s-ci-robot commented Aug 18, 2023

sftim commented Aug 18, 2023

sftim commented Aug 18, 2023 •

edited

Loading

ganeshgunasekaran commented Aug 19, 2023

sftim commented Aug 19, 2023

sftim commented Aug 19, 2023

sftim commented Aug 19, 2023

ganeshgunasekaran commented Aug 21, 2023

sftim commented Aug 21, 2023 •

edited

Loading

sftim commented Aug 21, 2023

xmudrii commented Aug 23, 2023

xmudrii commented Sep 5, 2023

dominic-p commented Sep 9, 2023

xmudrii commented Sep 9, 2023

dominic-p commented Sep 10, 2023 •

edited

Loading

xmudrii commented Sep 11, 2023

dominic-p commented Sep 12, 2023

xmudrii commented Sep 12, 2023

dominic-p commented Sep 12, 2023 •

edited

Loading

xmudrii commented Sep 12, 2023

dominic-p commented Sep 13, 2023

xmudrii commented Sep 15, 2023

AnirudhPanchangam commented Sep 21, 2023 •

edited

Loading

nethershaw commented Sep 22, 2023 •

edited

Loading

xmudrii commented Sep 23, 2023

lucasmo commented Oct 5, 2023

xmudrii commented Oct 11, 2023

Error installing from official repos onto Ubuntu #3219

Error installing from official repos onto Ubuntu #3219

Comments

ganeshgunasekaran commented Aug 18, 2023

k8s-ci-robot commented Aug 18, 2023

sftim commented Aug 18, 2023

sftim commented Aug 18, 2023 • edited Loading

ganeshgunasekaran commented Aug 19, 2023

sftim commented Aug 19, 2023

sftim commented Aug 19, 2023

sftim commented Aug 19, 2023

ganeshgunasekaran commented Aug 21, 2023

sftim commented Aug 21, 2023 • edited Loading

sftim commented Aug 21, 2023

xmudrii commented Aug 23, 2023

xmudrii commented Sep 5, 2023

dominic-p commented Sep 9, 2023

xmudrii commented Sep 9, 2023

dominic-p commented Sep 10, 2023 • edited Loading

xmudrii commented Sep 11, 2023

dominic-p commented Sep 12, 2023

xmudrii commented Sep 12, 2023

dominic-p commented Sep 12, 2023 • edited Loading

Community

Google

xmudrii commented Sep 12, 2023

dominic-p commented Sep 13, 2023

xmudrii commented Sep 15, 2023

AnirudhPanchangam commented Sep 21, 2023 • edited Loading

nethershaw commented Sep 22, 2023 • edited Loading

xmudrii commented Sep 23, 2023

lucasmo commented Oct 5, 2023

xmudrii commented Oct 11, 2023

sftim commented Aug 18, 2023 •

edited

Loading

sftim commented Aug 21, 2023 •

edited

Loading

dominic-p commented Sep 10, 2023 •

edited

Loading

dominic-p commented Sep 12, 2023 •

edited

Loading

AnirudhPanchangam commented Sep 21, 2023 •

edited

Loading

nethershaw commented Sep 22, 2023 •

edited

Loading