curl command is not working in k3s container #763

linusenthu · 2019-08-26T19:29:09Z

Describe the bug
curl command does not work in k3s container

alexellis · 2019-08-27T21:22:34Z

I have this too, I couldn't even install curl due to not being able to access apk over HTTP.

alexellis · 2019-08-27T21:24:02Z

Install k3s (on Civo instance)
Run alpine:3.10
Type in apk add --no-cache curl

^ hangs, then gives i/o timeout.

alexellis · 2019-08-27T23:16:19Z

@erikwilson what do you need from us on this one? It's blocking my use of k3s.

erikwilson · 2019-08-27T23:38:43Z

Hmmm, I feel like it might be helpful for us to have a bug reporting tool to collect system info.

Is there a firewall enabled? (if yes, does temporarily disabling the firewall resolve the issue?)

otherwise...

How is k3s installed?
What flags are used for running k3s?
What is the OS & version of the Civo instance?
What is the output of iptables --version?
What is the output of iptables-save?
What are the network interfaces? (eg output of ip a or ifconfig)
What are the routes? (output of route)

warmchang · 2019-08-28T05:55:26Z

@alexellis I can't reproduce this on Ubuntu 18.04.2

$ uname -a
Linux minikube 4.15.0-51-generic #55-Ubuntu SMP Wed May 15 14:27:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$
$ iptables --version
iptables v1.6.1
$
$ k3s kubectl run busybox --image=alpine --replicas=1 --command -- sh -c 'echo Hello K3S! && sleep 3600'
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/busybox created
$ k3s kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6-k3s.1", GitCommit:"4cd85f14854d942e9016cc15f399785c103242e9", GitTreeState:"clean", BuildDate:"2019-08-19T16:12+00:00Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6-k3s.1", GitCommit:"4cd85f14854d942e9016cc15f399785c103242e9", GitTreeState:"clean", BuildDate:"2019-08-19T16:12+00:00Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
$
$ k3s kubectl get pod
NAME READY STATUS RESTARTS AGE
busybox-b7b8d5dc4-74ktq 1/1 Running 0 13s
$
$ k3s kubectl exec -ti busybox-b7b8d5dc4-74ktq -- sh
/ # apk add --no-cache curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/4) Installing ca-certificates (20190108-r0)
(2/4) Installing nghttp2-libs (1.39.2-r0)
(3/4) Installing libcurl (7.65.1-r0)
(4/4) Installing curl (7.65.1-r0)
Executing busybox-1.30.1-r2.trigger
Executing ca-certificates-20190108-r0.trigger
OK: 7 MiB in 18 packages
/ # curl --version
curl 7.65.1 (x86_64-alpine-linux-musl) libcurl/7.65.1 OpenSSL/1.1.1c zlib/1.2.11 nghttp2/1.39.2
Release-Date: 2019-06-05
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS HTTP2 HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL TLS-SRP UnixSockets
/ #

andyjeffries · 2019-08-28T09:21:02Z

I've just been thinking about this and it may be "Docker + OpenStack" specific. (I'm not normally network focused, so some of the following terminology may be wrong) OpenStack uses some overhead in the packet size, so we need to reconfigure Docker on OpenStack instances to use a 50 byte smaller MTU. For some history on the bug:

https://medium.com/@sylwit/how-we-spent-a-full-day-figuring-out-a-mtu-issue-with-docker-4d81fdfe2caf
https://mlohr.com/docker-mtu/

For the fix for Civo (and other OpenStack, and where affected, virtualised instances) is:

https://www.civo.com/learn/fixing-networking-for-docker

I'll put this fix in to our managed K3s service, but if you're installing manually or using k3sup you'll need to apply the fix manually.

@alexellis hope this fixes it for you.

andyjeffries · 2019-08-28T09:24:59Z

Actually, I've just realised K3s doesn't use Docker - so this may be a complete red herring or the fix may be similar?

andyjeffries · 2019-08-28T10:00:16Z

There's some info on setting the MTU for flannel at flannel-io/flannel#841 (and again saying use networkMTU - 50), but I don't know how this applies to K3s.

alexellis · 2019-08-28T10:03:25Z

alext:civo alex$ export KUBECONFIG=./cluster2
alext:civo alex$ kubectl run -t -i alpine-curl-2 --image alpine:3.10
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.
/ # apk add --no-cache curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
  curl (missing):
    required by: world[curl]
/ # Session ended, resume using 'kubectl attach alpine-curl-2-54c77d6b89-rdjnv -c alpine-curl-2 -i -t' command when the pod is running
alext:civo alex$ kubectl get node -o wide
NAME    STATUS   ROLES    AGE   VERSION         INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k3s-1   Ready    master   15h   v1.14.6-k3s.1   172.31.2.245   <none>        Ubuntu 18.04.3 LTS   4.15.0-58-generic   containerd://1.2.7-k3s1
alext:civo alex$

alexellis · 2019-08-28T10:04:15Z

From the host:

alext:civo alex$ ssh [email protected]
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-58-generic x86_64)

root@k3s-1:~# curl http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output 
Warning: <FILE>" to save to a file.
root@k3s-1:~#

alexellis · 2019-08-28T10:05:18Z

Sorry for the verbose output 😄

root@k3s-1:~# which iptables
/sbin/iptables
root@k3s-1:~# iptables --version
iptables v1.6.1
root@k3s-1:~# iptables-save
# Generated by iptables-save v1.6.1 on Wed Aug 28 11:04:39 2019
*filter
:INPUT ACCEPT [508:140930]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [489:140398]
:KUBE-EXTERNAL-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A FORWARD -s 10.42.0.0/16 -j ACCEPT
-A FORWARD -d 10.42.0.0/16 -j ACCEPT
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -s 10.42.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 10.42.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Wed Aug 28 11:04:39 2019
# Generated by iptables-save v1.6.1 on Wed Aug 28 11:04:39 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [5:300]
:POSTROUTING ACCEPT [5:300]
:CNI-DN-3c10d74c99659033b0660 - [0:0]
:CNI-HOSTPORT-DNAT - [0:0]
:CNI-HOSTPORT-MASQ - [0:0]
:CNI-HOSTPORT-SETMARK - [0:0]
:KUBE-FW-IKNZCF5XJQBTG3KZ - [0:0]
:KUBE-FW-X3WUOHPTYIG7AA3Q - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-5OH45RWWUFBIURRS - [0:0]
:KUBE-SEP-CO33EDX5JB6XGH4C - [0:0]
:KUBE-SEP-J5U7JDFDTOWHXPVR - [0:0]
:KUBE-SEP-JXLNNH6TRVMJLTK5 - [0:0]
:KUBE-SEP-K5XXJLQU5KR42YSX - [0:0]
:KUBE-SEP-OWVYFKC6LI5UONET - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-IKNZCF5XJQBTG3KZ - [0:0]
:KUBE-SVC-JD5MR3NA4I4DYORP - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
:KUBE-SVC-X3WUOHPTYIG7AA3Q - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A POSTROUTING -m comment --comment "CNI portfwd requiring masquerade" -j CNI-HOSTPORT-MASQ
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 10.42.0.0/16 -d 10.42.0.0/16 -j RETURN
-A POSTROUTING -s 10.42.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
-A POSTROUTING ! -s 10.42.0.0/16 -d 10.42.0.0/24 -j RETURN
-A POSTROUTING ! -s 10.42.0.0/16 -d 10.42.0.0/16 -j MASQUERADE
-A CNI-DN-3c10d74c99659033b0660 -s 10.42.0.6/32 -p tcp -m tcp --dport 80 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-3c10d74c99659033b0660 -s 127.0.0.1/32 -p tcp -m tcp --dport 80 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-3c10d74c99659033b0660 -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.42.0.6:80
-A CNI-DN-3c10d74c99659033b0660 -s 10.42.0.6/32 -p tcp -m tcp --dport 443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-3c10d74c99659033b0660 -s 127.0.0.1/32 -p tcp -m tcp --dport 443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-3c10d74c99659033b0660 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.42.0.6:443
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"cbr0\" id: \"ffa075ebcbed414f42777fb76e115bd4e5774a836e36eff20b7836de3c1345d8\"" -m multiport --dports 80,443 -j CNI-DN-3c10d74c99659033b0660
-A CNI-HOSTPORT-MASQ -m mark --mark 0x2000/0x2000 -j MASQUERADE
-A CNI-HOSTPORT-SETMARK -m comment --comment "CNI portfwd masquerade mark" -j MARK --set-xmark 0x2000/0x2000
-A KUBE-FW-IKNZCF5XJQBTG3KZ -m comment --comment "kube-system/traefik:https loadbalancer IP" -j KUBE-MARK-MASQ
-A KUBE-FW-IKNZCF5XJQBTG3KZ -m comment --comment "kube-system/traefik:https loadbalancer IP" -j KUBE-SVC-IKNZCF5XJQBTG3KZ
-A KUBE-FW-IKNZCF5XJQBTG3KZ -m comment --comment "kube-system/traefik:https loadbalancer IP" -j KUBE-MARK-DROP
-A KUBE-FW-X3WUOHPTYIG7AA3Q -m comment --comment "kube-system/traefik:http loadbalancer IP" -j KUBE-MARK-MASQ
-A KUBE-FW-X3WUOHPTYIG7AA3Q -m comment --comment "kube-system/traefik:http loadbalancer IP" -j KUBE-SVC-X3WUOHPTYIG7AA3Q
-A KUBE-FW-X3WUOHPTYIG7AA3Q -m comment --comment "kube-system/traefik:http loadbalancer IP" -j KUBE-MARK-DROP
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODEPORTS -p tcp -m comment --comment "kube-system/traefik:https" -m tcp --dport 32338 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "kube-system/traefik:https" -m tcp --dport 32338 -j KUBE-SVC-IKNZCF5XJQBTG3KZ
-A KUBE-NODEPORTS -p tcp -m comment --comment "kube-system/traefik:http" -m tcp --dport 30142 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "kube-system/traefik:http" -m tcp --dport 30142 -j KUBE-SVC-X3WUOHPTYIG7AA3Q
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-5OH45RWWUFBIURRS -s 10.42.0.5/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-5OH45RWWUFBIURRS -p tcp -m tcp -j DNAT --to-destination 10.42.0.5:80
-A KUBE-SEP-CO33EDX5JB6XGH4C -s 10.42.0.5/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-CO33EDX5JB6XGH4C -p tcp -m tcp -j DNAT --to-destination 10.42.0.5:443
-A KUBE-SEP-J5U7JDFDTOWHXPVR -s 10.42.0.4/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-J5U7JDFDTOWHXPVR -p udp -m udp -j DNAT --to-destination 10.42.0.4:53
-A KUBE-SEP-JXLNNH6TRVMJLTK5 -s 172.31.2.245/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-JXLNNH6TRVMJLTK5 -p tcp -m tcp -j DNAT --to-destination 172.31.2.245:6443
-A KUBE-SEP-K5XXJLQU5KR42YSX -s 10.42.0.4/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-K5XXJLQU5KR42YSX -p tcp -m tcp -j DNAT --to-destination 10.42.0.4:53
-A KUBE-SEP-OWVYFKC6LI5UONET -s 10.42.0.4/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-OWVYFKC6LI5UONET -p tcp -m tcp -j DNAT --to-destination 10.42.0.4:9153
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-SVC-JD5MR3NA4I4DYORP
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.19.233/32 -p tcp -m comment --comment "kube-system/traefik:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.19.233/32 -p tcp -m comment --comment "kube-system/traefik:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-IKNZCF5XJQBTG3KZ
-A KUBE-SERVICES -d 172.31.2.245/32 -p tcp -m comment --comment "kube-system/traefik:https loadbalancer IP" -m tcp --dport 443 -j KUBE-FW-IKNZCF5XJQBTG3KZ
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.19.233/32 -p tcp -m comment --comment "kube-system/traefik:http cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.19.233/32 -p tcp -m comment --comment "kube-system/traefik:http cluster IP" -m tcp --dport 80 -j KUBE-SVC-X3WUOHPTYIG7AA3Q
-A KUBE-SERVICES -d 172.31.2.245/32 -p tcp -m comment --comment "kube-system/traefik:http loadbalancer IP" -m tcp --dport 80 -j KUBE-FW-X3WUOHPTYIG7AA3Q
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-ERIFXISQEP7F7OF4 -j KUBE-SEP-K5XXJLQU5KR42YSX
-A KUBE-SVC-IKNZCF5XJQBTG3KZ -j KUBE-SEP-CO33EDX5JB6XGH4C
-A KUBE-SVC-JD5MR3NA4I4DYORP -j KUBE-SEP-OWVYFKC6LI5UONET
-A KUBE-SVC-NPX46M4PTMTKRN6Y -j KUBE-SEP-JXLNNH6TRVMJLTK5
-A KUBE-SVC-TCOU7JCQXEZGVUNU -j KUBE-SEP-J5U7JDFDTOWHXPVR
-A KUBE-SVC-X3WUOHPTYIG7AA3Q -j KUBE-SEP-5OH45RWWUFBIURRS
COMMIT
# Completed on Wed Aug 28 11:04:39 2019
root@k3s-1:~# ifconfig
cni0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1400
        inet 10.42.0.1  netmask 255.255.255.0  broadcast 10.42.0.255
        inet6 fe80::3c21:27ff:fe43:c7d2  prefixlen 64  scopeid 0x20<link>
        ether 3e:21:27:43:c7:d2  txqueuelen 1000  (Ethernet)
        RX packets 97748  bytes 8722921 (8.7 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 105993  bytes 26227643 (26.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.0.0.5  netmask 255.0.0.0  broadcast 10.255.255.255
        inet6 fe80::f816:3eff:fe46:a98  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:46:0a:98  txqueuelen 1000  (Ethernet)
        RX packets 269  bytes 24402 (24.4 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 205  bytes 20997 (20.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.31.2.245  netmask 255.255.0.0  broadcast 172.31.255.255
        inet6 fe80::f816:3eff:fef6:408f  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:f6:40:8f  txqueuelen 1000  (Ethernet)
        RX packets 2387028  bytes 253988623 (253.9 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 73863  bytes 10409496 (10.4 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1400
        inet 10.42.0.0  netmask 255.255.255.255  broadcast 10.42.0.0
        inet6 fe80::2c3f:4aff:feed:2481  prefixlen 64  scopeid 0x20<link>
        ether 2e:3f:4a:ed:24:81  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 30 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 1188924  bytes 314172136 (314.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1188924  bytes 314172136 (314.1 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth083a2867: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1400
        inet6 fe80::a4ed:42ff:fe38:b0e8  prefixlen 64  scopeid 0x20<link>
        ether a6:ed:42:38:b0:e8  txqueuelen 0  (Ethernet)
        RX packets 62989  bytes 6366926 (6.3 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 67483  bytes 6044252 (6.0 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth1499f87b: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1400
        inet6 fe80::c36:50ff:fe64:2ecc  prefixlen 64  scopeid 0x20<link>
        ether 0e:36:50:64:2e:cc  txqueuelen 0  (Ethernet)
        RX packets 30  bytes 2168 (2.1 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 144  bytes 10152 (10.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth2f3a6561: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1400
        inet6 fe80::a0af:73ff:fe6d:2926  prefixlen 64  scopeid 0x20<link>
        ether a2:af:73:6d:29:26  txqueuelen 0  (Ethernet)
        RX packets 33222  bytes 3431599 (3.4 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 36357  bytes 3183468 (3.1 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethccea6bcd: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1400
        inet6 fe80::d436:29ff:fec1:9583  prefixlen 64  scopeid 0x20<link>
        ether d6:36:29:c1:95:83  txqueuelen 0  (Ethernet)
        RX packets 174  bytes 12389 (12.3 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 444  bytes 696530 (696.5 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethe61b4511: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1400
        inet6 fe80::300a:96ff:fe65:fcb5  prefixlen 64  scopeid 0x20<link>
        ether 32:0a:96:65:fc:b5  txqueuelen 0  (Ethernet)
        RX packets 143  bytes 10298 (10.2 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 255  bytes 683150 (683.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@k3s-1:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.31.255.254  0.0.0.0         UG    0      0        0 ens4
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 ens3
10.42.0.0       0.0.0.0         255.255.255.0   U     0      0        0 cni0
169.254.169.254 172.31.255.254  255.255.255.255 UGH   0      0        0 ens4
172.31.0.0      0.0.0.0         255.255.0.0     U     0      0        0 ens4
root@k3s-1:~#

andyjeffries · 2019-08-28T10:13:14Z

We at Civo use a 172.31.0.0 address for our "pseudo-public" interface. There's more description around this networking style and why we did it at - https://www.civo.com/blog/changes-in-ip-address-usage-at-civo

erikwilson · 2019-08-28T14:55:07Z

I wonder if ens3 routing is conflicting with the pod & service cidr configured for k3s.

erikwilson · 2019-08-28T15:37:54Z

starting k3s with --cluster-cidr=192.168.0.0/17 --service-cidr=192.168.128.0/17 might help, it may be that we need to cowardly refuse to start if our cidrs are part of another interface.

running traceroute 1.1.1.1 from within an alpine pod might help to determine where traffic is going also.

andyjeffries · 2019-08-28T15:57:09Z

I've just switched our managed K3s service to install with those flags (as we aren't in close beta yet). Below is some more debug:

root@kube-master-ec7d:~# ps axwu | grep k3s
root      2629  5.8  5.9 341296 239540 ?       Ssl  16:46   0:18 /usr/local/bin/k3s server --tls-san 91.211.152.232 --cluster-cidr=192.168.0.0/17 --service-cidr=192.168.128.0/17
# ...

Process is running with flags, so network check:

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:75:8a:bd brd ff:ff:ff:ff:ff:ff
    inet 10.148.137.54/24 brd 10.148.137.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe75:8abd/64 scope link 
       valid_lft forever preferred_lft forever
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:89:15:02 brd ff:ff:ff:ff:ff:ff
    inet 172.31.2.59/16 brd 172.31.255.255 scope global ens4
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe89:1502/64 scope link 
       valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default 
    link/ether 6a:58:59:05:bc:90 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.0/32 brd 192.168.0.0 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::6858:59ff:fe05:bc90/64 scope link 
       valid_lft forever preferred_lft forever
5: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default qlen 1000
    link/ether e6:ff:6b:de:ec:8b brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/24 brd 192.168.0.255 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::e4ff:6bff:fede:ec8b/64 scope link 
       valid_lft forever preferred_lft forever
7: veth43f03e73@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default 
    link/ether f2:b4:46:0a:9c:14 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::f0b4:46ff:fe0a:9c14/64 scope link 
       valid_lft forever preferred_lft forever
8: veth9503aeb9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default 
    link/ether e2:f7:6f:b9:35:6d brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::e0f7:6fff:feb9:356d/64 scope link 
       valid_lft forever preferred_lft forever
9: veth9e772eca@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default 
    link/ether a6:16:80:10:ef:c4 brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet6 fe80::a416:80ff:fe10:efc4/64 scope link 
       valid_lft forever preferred_lft forever
10: veth3323e369@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master cni0 state UP group default 
    link/ether 06:7b:b1:29:88:fd brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::47b:b1ff:fe29:88fd/64 scope link 
       valid_lft forever preferred_lft forever

Checking to see if we can download files over HTTP:

$ kubectl run -t -i alpine-curl-2 --image alpine:3.10                                                                                                                                                                                                                  ⎈ test-net-1:default
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.
/ # apk add --no-cache curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz: network error (check Internet connection and firewall)
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
  curl (missing):
    required by: world[curl]

I remembered from when I've done stuff like this (I don't do it that often) that sometimes you need to apk update first, but that will also download over HTTP...

/ # apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
v3.10.2-19-g264df00ff8 [http://dl-cdn.alpinelinux.org/alpine/v3.10/main]
v3.10.2-20-ga23f3d183a [http://dl-cdn.alpinelinux.org/alpine/v3.10/community]
OK: 10334 distinct packages available
/ # apk add --no-cache curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/4) Installing ca-certificates (20190108-r0)
(2/4) Installing nghttp2-libs (1.39.2-r0)
(3/4) Installing libcurl (7.65.1-r0)
(4/4) Installing curl (7.65.1-r0)
Executing busybox-1.30.1-r2.trigger
Executing ca-certificates-20190108-r0.trigger
OK: 7 MiB in 18 packages

So it's all working? Just for extra stuff, I did the traceroute too...

/ # traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 46 byte packets
 1  192.168.0.1 (192.168.0.1)  0.010 ms  0.011 ms  0.005 ms
 2  172.31.255.254 (172.31.255.254)  0.702 ms  0.513 ms  0.513 ms
 3  185.136.235.252 (185.136.235.252)  4.252 ms  13.899 ms  16.888 ms
 4  31.24.108.181 (31.24.108.181)  0.562 ms  0.944 ms  2.297 ms
 5  ve159.er3.sdc.as50056.net (178.18.119.25)  0.760 ms  0.842 ms  0.861 ms
 6  178.18.119.97 (178.18.119.97)  2.591 ms  4.470 ms  14.275 ms
 7  185.120.204.66 (185.120.204.66)  3.465 ms  178.18.119.90 (178.18.119.90)  31.215 ms  185.120.204.66 (185.120.204.66)  5.418 ms
 8  185.120.204.70 (185.120.204.70)  3.164 ms  *  *
 9  *  *  185.120.204.73 (185.120.204.73)  2.029 ms
10  *  *  *
11  *  *  *
12  *  *  *
13  *  *  *
14  *  *  *
15  *  *  *
16  *  *  *
17  *  *  *
18  *  *  *

and it's off in to the ether of never coming back, but it certainly left our network and the apk update and subsequent apk add worked. So, all good @alexellis ?

erikwilson · 2019-08-28T16:06:40Z

Interesting, looks like it is working! Thought about the need for apk update also, but tried the same process and didn't need to update.

andyjeffries · 2019-08-28T16:10:27Z

Weirdly the second (and further nodes) aren't connecting to the cluster now though. I'm installing them with (effectively):

K3S_URL=https://some.ip.add.ress:6443 K3S_TOKEN=my_token INSTALL_K3S_VERSION=v0.8.1 \
 INSTALL_K3S_EXEC='agent --cluster-cidr=192.168.0.0/17 --service-cidr=192.168.128.0/17'

Is that right? Before now it didn't have an INSTALL_K3S_EXEC for the nodes just for the master.

andyjeffries · 2019-08-28T16:51:45Z

Never mind, realised the agent doesn't need those flags (I assume it pulls them from the server).

andyjeffries · 2019-08-28T17:02:30Z

Just launched a new cluster with the new CIDR and it all seems OK for me - @alexellis ?

18:01 ~ $ kubectl get nodes                                                                                                                              NAME               STATUS   ROLES    AGE     VERSION
kube-master-3dbe   Ready    master   2m39s   v1.14.6-k3s.1
kube-node-c349     Ready    worker   16s     v1.14.6-k3s.1
18:01 ~ $ kubectl run -t -i alpine-curl-2 --image alpine:3.10                                                                                            kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.
/ # apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
apk add curlfetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
v3.10.2-19-g264df00ff8 [http://dl-cdn.alpinelinux.org/alpine/v3.10/main]
v3.10.2-20-ga23f3d183a [http://dl-cdn.alpinelinux.org/alpine/v3.10/community]
OK: 10334 distinct packages available
/ # apk add curl
(1/4) Installing ca-certificates (20190108-r0)
(2/4) Installing nghttp2-libs (1.39.2-r0)
(3/4) Installing libcurl (7.65.1-r0)
(4/4) Installing curl (7.65.1-r0)
Executing busybox-1.30.1-r2.trigger
Executing ca-certificates-20190108-r0.trigger
OK: 7 MiB in 18 packages
/ # Session ended, resume using 'kubectl attach alpine-curl-2-54c77d6b89-fwk8k -c alpine-curl-2 -i -t' command when the pod is running

alexellis · 2019-08-28T17:42:27Z

The apk update command is not needed, apk add --no-cache curl already updates and then doesn't store the data. Can you try this one please?

Here's a bare-metal Intel NUC at my house with Ubuntu 18.04:

alex@nuc7:~$ uname -a
Linux nuc7 4.15.0-50-generic #54-Ubuntu SMP Mon May 6 18:46:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
alex@nuc7:~$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
alex@nuc7:~$ kubectl run -t -i alpine-curl-2 --image alpine:3.10
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.
/ # 
/ # apk add --no-cache curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/4) Installing ca-certificates (20190108-r0)
(2/4) Installing nghttp2-libs (1.39.2-r0)
(3/4) Installing libcurl (7.65.1-r0)
(4/4) Installing curl (7.65.1-r0)
Executing busybox-1.30.1-r2.trigger
Executing ca-certificates-20190108-r0.trigger
OK: 7 MiB in 18 packages
/ #

(I believe it's using Weave net.)

alexellis · 2019-08-28T17:45:47Z

With no changes to the Civo/OpenStack VM, it worked at least once:

$ kubectl run -t -i alpine-curl-3 --image alpine:3.10
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.

/ # 
/ # apk add --no-cache curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/4) Installing ca-certificates (20190108-r0)
(2/4) Installing nghttp2-libs (1.39.2-r0)
(3/4) Installing libcurl (7.65.1-r0)
(4/4) Installing curl (7.65.1-r0)
Executing busybox-1.30.1-r2.trigger
Executing ca-certificates-20190108-r0.trigger
OK: 7 MiB in 18 packages
/ #

The second time it failed:

kubectl run -t -i alpine-curl-4 --image alpine:3.10
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.
/ # kubectl run -t -i alpine-curl-3 --image alpine:3.10
/bin/sh: kubectl: not found
/ # apk add --no-cache curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)
ERROR: unsatisfiable constraints:
  curl (missing):
    required by: world[curl]
/ #

Someone asked about firewalls / security groups. There is only a security group on incoming traffic to block VXLAN from the outside.

~~I wonder if the error depends on whether the pod is scheduled on the worker or the server node?~~ ignore that comment, it's a single-node.

alexellis · 2019-08-28T17:48:42Z

@erikwilson in light of this issue from the Civo blog about MTUs, which Andy mentioned, can we configure an MTU setting for containerd?

Perhaps since networking is out of scope for containerd, this would actually be done in CNI with Flannel?

alexellis · 2019-08-28T17:53:04Z

I'm seeing this:

root@k3s-1:~# cat /run/flannel/subnet.env   
FLANNEL_NETWORK=10.42.0.0/16                
FLANNEL_SUBNET=10.42.0.1/24                 
FLANNEL_MTU=1400                            
FLANNEL_IPMASQ=true                         
root@k3s-1:~#

andyjeffries · 2019-08-28T17:55:36Z

With those values though, it looks like this is an old cluster without the 192 change made this afternoon?

…

On 28 Aug 2019, 18:53 +0100, Alex Ellis ***@***.***>, wrote: I'm seeing this: ***@***.***:~# cat /run/flannel/subnet.env FLANNEL_NETWORK=10.42.0.0/16 FLANNEL_SUBNET=10.42.0.1/24 FLANNEL_MTU=1400 FLANNEL_IPMASQ=true ***@***.***:~# — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

erikwilson · 2019-08-28T17:56:09Z

The MTU stuff is being setup in flannel and is applying a -50, looking at the network configurations I think it is okay. I would expect it to fail consistently.

The weird thing is the error temporary error (try again later), I wonder if dl-cdn.alpinelinux.org is actually returning a 503 for whatever reason.

alexellis · 2019-08-28T17:56:36Z

I edited the file to 1500, then when I rebooted the value came back again.

alexellis · 2019-08-28T17:59:31Z

You guys can use my sample curl image to try out other sites:

kubectl run -t -i curl --image alexellis/curl:0.1.1 -- /bin/sh

~ $ time curl 'https://api6.ipify.org?format=json'
{"ip":"91.211.152.153"}real	0m 5.64s
user	0m 0.02s
sys	0m 0.00s
~ $ time curl 'https://api6.ipify.org?format=json'
{"ip":"91.211.152.153"}real	0m 3.25s
user	0m 0.02s
sys	0m 0.00s
~ $

From my home network this takes 0.7s only.

erikwilson · 2019-08-28T18:04:41Z

Ah, I think the temporary error message is just the default message from apk. Should double check that DNS is resolving consistently also.

andyjeffries · 2019-08-28T18:56:01Z

I've just checked from a container and it seems to take a while to resolve a DNS entry and then consistently resolves it. However, I just checked the underlying instance and it shows the same problem. I'll have our engineers look in to it tomorrow, I have a hacky fix for it but would like them to try to properly fix it first.

erikwilson · 2019-08-28T20:28:18Z

Might be worth checking for DNSSEC errors in /var/log/syslog, re #544

I think there is a 5s timeout for DNS queries, so also could be upstream DNS response times.

jmichalek132 · 2020-04-17T14:11:05Z

Hi I am experiencing the same issue. Did you manage to solve it?

andyjeffries · 2020-04-20T08:54:23Z

We're not experiencing it any more, but unfortunately I can't remember what changed. Sorry I can't help.

jmichalek132 · 2020-04-20T14:34:41Z

Okay should I create separate issue for this or piggyback on this one?

jmichalek132 · 2020-04-20T21:07:52Z

Okay in my case I didn't really solve the issue but when using cilium instead of flannel it's not happening.

lvandyk · 2021-03-19T07:34:56Z

@jmichalek132 Thank you, using cilium solved all the timeouts for me.

andyjeffries · 2021-05-12T13:59:31Z

I think this can be closed, certainly hasn't been an issue for a while.

stale · 2021-11-08T14:46:06Z

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

davidnuzik changed the title ~~curl command is not working in k3s container.~~ curl command is not working in k3s container Aug 26, 2019

erikwilson added the status/more-info label Aug 27, 2019

warmchang mentioned this issue Aug 30, 2019

CoreDNS CrashLoopBackOff #776

Closed

lvandyk mentioned this issue Mar 19, 2021

cURL to external internet API timeouts every few requests within Pods but works perfectly on node. #2928

Closed

consideRatio mentioned this issue Oct 5, 2021

Test latest (1.22) and oldest supported and testable (1.18) k8s version jupyterhub/binderhub#1392

Merged

stale bot added the status/stale label Nov 8, 2021

stale bot closed this as completed Nov 22, 2021

apiening mentioned this issue Mar 30, 2022

Can't reach internet from pod / container #5349

Closed

curl command is not working in k3s container #763

curl command is not working in k3s container #763

Comments

linusenthu commented Aug 26, 2019 • edited by davidnuzik Loading

alexellis commented Aug 27, 2019

alexellis commented Aug 27, 2019

alexellis commented Aug 27, 2019

erikwilson commented Aug 27, 2019

warmchang commented Aug 28, 2019

andyjeffries commented Aug 28, 2019

andyjeffries commented Aug 28, 2019

andyjeffries commented Aug 28, 2019

alexellis commented Aug 28, 2019

alexellis commented Aug 28, 2019

alexellis commented Aug 28, 2019

andyjeffries commented Aug 28, 2019

erikwilson commented Aug 28, 2019

erikwilson commented Aug 28, 2019

andyjeffries commented Aug 28, 2019

erikwilson commented Aug 28, 2019

andyjeffries commented Aug 28, 2019 • edited Loading

andyjeffries commented Aug 28, 2019

andyjeffries commented Aug 28, 2019

alexellis commented Aug 28, 2019

alexellis commented Aug 28, 2019 • edited Loading

alexellis commented Aug 28, 2019 • edited Loading

alexellis commented Aug 28, 2019

andyjeffries commented Aug 28, 2019 via email

erikwilson commented Aug 28, 2019

alexellis commented Aug 28, 2019

alexellis commented Aug 28, 2019 • edited Loading

erikwilson commented Aug 28, 2019

andyjeffries commented Aug 28, 2019

erikwilson commented Aug 28, 2019

jmichalek132 commented Apr 17, 2020

andyjeffries commented Apr 20, 2020

jmichalek132 commented Apr 20, 2020

jmichalek132 commented Apr 20, 2020

lvandyk commented Mar 19, 2021

andyjeffries commented May 12, 2021

stale bot commented Nov 8, 2021

linusenthu commented Aug 26, 2019 •

edited by davidnuzik

Loading

andyjeffries commented Aug 28, 2019 •

edited

Loading

alexellis commented Aug 28, 2019 •

edited

Loading

alexellis commented Aug 28, 2019 •

edited

Loading

alexellis commented Aug 28, 2019 •

edited

Loading