Too slow downloads from cache #8502

viric · 2015-06-25T14:08:10Z

Quite often the downloads from cache.nixos.org do not go beyond 20KB/s. Restarting the https download sometimes improves it, sometimes not. It seems that there is some chance of getting a normal download (>2MB/s) or a stalled one (<20KB/s).

This makes it very annoying to run almost any nix-env operation. A simple installation of a program that requires, let's say, 20 downloads, may require 5 or 6 Ctrl-C and restart of the nix-env operation.

Here is the log of several attempts in a row to download a very specific file. I tried to stop the download after ~30s, when it was running at stalled speed. Never a "stalled connection" recovers to normal speed, neither a normal connection becomes "stalled". The character of the connection seems determined since its start, and never changes:

[user@machine:~/nixpkgs]$ curl https://cache.nixos.org/nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz > /dev/null 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 46 8187k   46 3784k    0     0  2398k      0  0:00:03  0:00:01  0:00:02 2399k^C

[user@machine:~/nixpkgs]$ curl https://cache.nixos.org/nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz > /dev/null 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8187k  100 8187k    0     0  1336k      0  0:00:06  0:00:06 --:--:-- 1840k

[user@machine:~/nixpkgs]$ curl https://cache.nixos.org/nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz > /dev/null 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8187k  100 8187k    0     0  4783k      0  0:00:01  0:00:01 --:--:-- 4788k

[user@machine:~/nixpkgs]$ curl https://cache.nixos.org/nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz > /dev/null 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  6 8187k    6  510k    0     0  17044      0  0:08:11  0:00:30  0:07:41 20660^C

[user@machine:~/nixpkgs]$ curl https://cache.nixos.org/nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz > /dev/null 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8187k  100 8187k    0     0  5311k      0  0:00:01  0:00:01 --:--:-- 5313k

[user@machine:~/nixpkgs]$ curl https://cache.nixos.org/nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz > /dev/null 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8187k  100 8187k    0     0  2976k      0  0:00:02  0:00:02 --:--:-- 2976k

[user@machine:~/nixpkgs]$ curl https://cache.nixos.org/nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz > /dev/null 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8187k  100 8187k    0     0  6914k      0  0:00:01  0:00:01 --:--:-- 6921k

[user@machine:~/nixpkgs]$ curl https://cache.nixos.org/nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz > /dev/null 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  8 8187k    8  655k    0     0  21520      0  0:06:29  0:00:31  0:05:58 22536^C

[user@machine:~/nixpkgs]$ curl https://cache.nixos.org/nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz > /dev/null 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  8 8187k    8  661k    0     0  13877      0  0:10:04  0:00:48  0:09:16 10167^C

[user@machine:~/nixpkgs]$ curl https://cache.nixos.org/nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz > /dev/null 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 26 8187k   26 2205k    0     0  22774      0  0:06:08  0:01:39  0:04:29 24474^C

[user@machine:~/nixpkgs]$ curl https://cache.nixos.org/nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz > /dev/null 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8187k  100 8187k    0     0  6828k      0  0:00:01  0:00:01 --:--:-- 6834k

Our big set of NixOS computers are behind this IP address: 84.89.61.89. There we experience this problem since more than one month ago.

I cannot reproduce the behaviour from some other IPs of the same city, like: 47.61.242.149.

The text was updated successfully, but these errors were encountered:

viric · 2015-06-25T14:46:39Z

This problem happens also from this IP, city at ~50km: 188.77.24.141

vcunat · 2015-06-25T16:07:47Z

I now tried ~8 attempts total from ~4 URLs above from CZ and I'm getting ~8 MB/s, but I must admit the university connection is very good, IMHO.

viric · 2015-07-03T12:28:12Z

There is only one URL. In the "normal" cases we get 7MB/s, but we fall for most attempts into some network trap.

I just tested the same URL, a single attempt.... ~20KB/s.

viric · 2015-07-03T12:49:08Z

For what is worth, @edolstra already noted it, but we have this problem also downloading anyting else from CloudFront, like debian images. It seems related to our world region.

cpages · 2015-07-09T09:48:35Z

@rbvermaa now that we're a proper foundation ;) did you find the time to poke amazon about this? I'm hitting this continually

rbvermaa · 2015-07-09T09:55:22Z

@cpages @viric can you let us know what DNS you are using, and the location of your machines?

viric · 2015-07-09T10:00:26Z

At one site, we are using now 80.58.61.250 and 80.58.61.254 from Telefonica, our ISP. @cpages had problems at home too, so he can later answer about his DNS.

cpages · 2015-07-09T10:08:09Z

Mine is 87.216.1.65, from jazztel.

viric · 2015-07-09T10:12:06Z

I have tried a new DNS of a close infrastructure (CESCA), 84.88.0.3, and I get a similar outcome.

I have checked "netstat -tnp | grep 443" to see what IPs have slow https, and what have a fast one, and it seems unrelated to the IP. I had connections to 54.230.60.122:443 at 7MB/s and some at the same IP at 15KB/s.

viric · 2015-07-09T10:27:40Z

I'm trying the partial solution of getting our local DNS to return always the same cloudfront IP: 54.230.219.42. This one is not very fast (~1MB/s), but seems reliable. It does not get stuck like most returned by the DNS.

viric · 2015-07-09T10:53:43Z

Our office and @cpages home have a similar set of IPs returned by the DNS, for cache.nixos.org. My home, where all is not that fast but more stable, has a different set.

The annoying set that most of the time gets stuck, but sometimes works at 7MB/s are:

cache.nixos.org.        719     IN      CNAME   d3m36hgdyp4koz.cloudfront.net.
d3m36hgdyp4koz.cloudfront.net. 60 IN    A       54.230.60.227
d3m36hgdyp4koz.cloudfront.net. 60 IN    A       54.192.60.50
d3m36hgdyp4koz.cloudfront.net. 60 IN    A       54.230.61.177
d3m36hgdyp4koz.cloudfront.net. 60 IN    A       54.230.60.122
d3m36hgdyp4koz.cloudfront.net. 60 IN    A       54.192.61.247
d3m36hgdyp4koz.cloudfront.net. 60 IN    A       54.230.62.218
d3m36hgdyp4koz.cloudfront.net. 60 IN    A       54.230.61.109
d3m36hgdyp4koz.cloudfront.net. 60 IN    A       54.230.63.229

The set that work at ~1MB/s but reliable are:

cache.nixos.org.        2450    IN      CNAME   d3m36hgdyp4koz.cloudfront.net.
d3m36hgdyp4koz.cloudfront.net. 60 IN    A       54.192.219.6
d3m36hgdyp4koz.cloudfront.net. 60 IN    A       54.192.219.242
d3m36hgdyp4koz.cloudfront.net. 60 IN    A       54.230.219.35
d3m36hgdyp4koz.cloudfront.net. 60 IN    A       54.230.219.73
d3m36hgdyp4koz.cloudfront.net. 60 IN    A       54.230.219.11
d3m36hgdyp4koz.cloudfront.net. 60 IN    A       54.230.219.42
d3m36hgdyp4koz.cloudfront.net. 60 IN    A       54.230.219.17
d3m36hgdyp4koz.cloudfront.net. 60 IN    A       54.230.218.247

rbvermaa · 2015-07-09T12:34:11Z

Similar issue on AWS forum https://forums.aws.amazon.com/thread.jspa?threadID=111346

viric · 2015-07-09T12:49:38Z

Report based on http://s3.amazonaws.com/aws-cloudfront-testing/CustomerTesting.html :

# dig identity.cloudfront.net

; <<>> DiG 9.7.2-P3 <<>> identity.cloudfront.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64984
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;identity.cloudfront.net.       IN      A

;; ANSWER SECTION:
identity.cloudfront.net. 60     IN      A       54.240.150.22

;; Query time: 38 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Jul  9 12:40:55 2015
;; MSG SIZE  rcvd: 57

# dig resolver-identity.cloudfront.net

; <<>> DiG 9.7.2-P3 <<>> resolver-identity.cloudfront.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42501
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;resolver-identity.cloudfront.net. IN   A

;; ANSWER SECTION:
resolver-identity.cloudfront.net. 10 IN A       85.62.229.81

;; Query time: 39 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Jul  9 12:41:43 2015
;; MSG SIZE  rcvd: 66

# traceroute d7uri8nf7uskq.cloudfront.net
traceroute to d7uri8nf7uskq.cloudfront.net (54.230.61.50), 30 hops max, 60 byte packets
 1  picdelesbruixes.xxxxxxx.intern (192.168.33.1)  0.401 ms  0.541 ms  0.709 ms
 2  192.168.128.254 (192.168.128.254)  2.970 ms  3.020 ms  3.066 ms
 3  anella-parcudg.cesca.es (84.89.2.253)  4.915 ms  5.024 ms  8.589 ms
 4  94.229.195.1 (94.229.195.1)  6.193 ms  6.876 ms  7.344 ms
 5  212.31.32.14 (212.31.32.14)  7.392 ms  7.437 ms  7.487 ms
 6  84.76.248.5 (84.76.248.5)  7.516 ms  5.746 ms  5.832 ms
 7  10.34.220.17 (10.34.220.17)  5.362 ms  5.426 ms  5.537 ms
 8  * * *
 9  10.34.34.225 (10.34.34.225)  10.286 ms  10.399 ms  8.219 ms
10  81.52.188.113 (81.52.188.113)  7.155 ms  7.979 ms  7.973 ms
11  tengige0-6-0-1.barcr4.Barcelona.opentransit.net (193.251.132.122)  6.920 ms tengige0-7-0-10.barcr4.Barcelona.opentransit.net (193.251.242.30)  9.039 ms tengige0-6-0-5.barcr4.Barcelona.opentransit.net (193.251.240.53)  10.658 ms
12  tengige0-5-0-9.madtr1.Madrid.opentransit.net (193.251.128.40)  18.799 ms  16.365 ms  16.224 ms
13  gigabitethernet4-0-2.madcr3.Madrid.opentransit.net (193.251.131.246)  14.733 ms  14.686 ms  14.698 ms
14  mad-b2-link.telia.net (213.248.78.121)  13.399 ms  13.776 ms  13.843 ms
15  mad-b2-link.telia.net (62.115.139.187)  13.847 ms mad-b2-link.telia.net (62.115.139.133)  13.836 ms mad-b2-link.telia.net (62.115.139.119)  13.991 ms
16  a100row-ic-153783-mad-b1.c.telia.net (80.239.195.94)  15.140 ms  14.985 ms  15.203 ms
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

# curl -w "time_namelookup: %{time_namelookup}\ntime_connect: %{time_connect}\ntime_appconnect: %{time_appconnect}\ntime_pretransfer: %{time_pretransfer}\ntime_redirect: %{time_redirect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n" -v https://cache.nixos.org/nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz -o /dev/null
*   Trying 54.230.62.218...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to cache.nixos.org (54.230.62.218) port 443 (#0)
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-bundle.crt
  CApath: none
* TLSv1.2, TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2, TLS handshake, Server hello (2):
{ [74 bytes data]
* NPN, negotiated HTTP1.1
* TLSv1.2, TLS handshake, CERT (11):
{ [2318 bytes data]
* TLSv1.2, TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2, TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2, TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2, TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2, TLS handshake, Unknown (67):
} [36 bytes data]
* TLSv1.2, TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2, TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2, TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*        subject: serialNumber=752rFkZx9v83NlhSL4X71NxpvGYjbLUO; OU=GT33120842; OU=See www.rapidssl.com/resources/cps (c)14; OU=Domain Control Validated - RapidSSL(R); CN=cache.nixos.org
*        start date: 2014-10-21 19:51:27 GMT
*        expire date: 2015-10-25 00:40:09 GMT
*        subjectAltName: cache.nixos.org matched
*        issuer: C=US; O=GeoTrust, Inc.; CN=RapidSSL CA
*        SSL certificate verify ok.
> GET /nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz HTTP/1.1
> Host: cache.nixos.org
> User-Agent: curl/7.42.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: binary/octet-stream
< Content-Length: 8384088
< Connection: keep-alive
< Date: Thu, 09 Jul 2015 10:04:31 GMT
< Last-Modified: Wed, 10 Jun 2015 21:59:05 GMT
< ETag: "3093a156fbce3aae2de004f94d570727"
< Accept-Ranges: bytes
< Server: AmazonS3
< Age: 9594
< X-Cache: Hit from cloudfront
< Via: 1.1 c8ebee7efad3aecca9e7749a90284b36.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: yfKmoJh9VoApJkCBtcX7tmjd-V26ZvIMjtIkaNk5aXgyY_jX-y2ggA==
< 
{ [15933 bytes data]
 22 8187k   22 1805k    0     0   8281      0  0:16:52  0:03:43  0:13:09  9978
^C  # I Ctrl-C at 3:43. 1800KB downloaded, ~10KB/s, with some stalls, sometimes reaching 20KB/s.

# Retrying again, and showing the 'fast' case.

# curl -w "time_namelookup: %{time_namelookup}\ntime_connect: %{time_connect}\ntime_appconnect: %{time_appconnect}\ntime_pretransfer: %{time_pretransfer}\ntime_redirect: %{time_redirect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n" -v https://cache.nixos.org/nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz -o /dev/null
*   Trying 54.230.63.229...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to cache.nixos.org (54.230.63.229) port 443 (#0)
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-bundle.crt
  CApath: none
* TLSv1.2, TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2, TLS handshake, Server hello (2):
{ [74 bytes data]
* NPN, negotiated HTTP1.1
* TLSv1.2, TLS handshake, CERT (11):
{ [2318 bytes data]
* TLSv1.2, TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2, TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2, TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2, TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2, TLS handshake, Unknown (67):
} [36 bytes data]
* TLSv1.2, TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2, TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2, TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*        subject: serialNumber=752rFkZx9v83NlhSL4X71NxpvGYjbLUO; OU=GT33120842; OU=See www.rapidssl.com/resources/cps (c)14; OU=Domain Control Validated - RapidSSL(R); CN=cache.nixos.org
*        start date: 2014-10-21 19:51:27 GMT
*        expire date: 2015-10-25 00:40:09 GMT
*        subjectAltName: cache.nixos.org matched
*        issuer: C=US; O=GeoTrust, Inc.; CN=RapidSSL CA
*        SSL certificate verify ok.
> GET /nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz HTTP/1.1
> Host: cache.nixos.org
> User-Agent: curl/7.42.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: binary/octet-stream
< Content-Length: 8384088
< Connection: keep-alive
< Date: Thu, 09 Jul 2015 10:04:31 GMT
< Last-Modified: Wed, 10 Jun 2015 21:59:05 GMT
< ETag: "3093a156fbce3aae2de004f94d570727"
< Accept-Ranges: bytes
< Server: AmazonS3
< Age: 9851
< X-Cache: Hit from cloudfront
< Via: 1.1 fe90d926e9e3c9e262718a54401ef0d0.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: gbb7bU9TOOjCs_4-41AZSlzDmdZbx9df3Ya7kIskFnA3xJ4fsMuTAg==
< 
{ [6450 bytes data]
100 8187k  100 8187k    0     0  1116k      0  0:00:07  0:00:07 --:--:-- 1101k
* Connection #0 to host cache.nixos.org left intact
time_namelookup: 0,001
time_connect: 0,016
time_appconnect: 0,092
time_pretransfer: 0,093
time_redirect: 0,000
time_starttransfer: 0,116
time_total: 7,332

viric · 2015-07-09T13:21:57Z

Here is a full slow download:

[root@comanegra:~]# curl -w "time_namelookup: %{time_namelookup}\ntime_connect: %nect}\ntime_appconnect: %{time_appconnect}\ntime_pretransfer: %{time_pretransfer}direct: %{time_redirect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: tal}\n" -v https://cache.nixos.org/nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgyi.nar.xz -o /dev/null
*   Trying 54.230.62.218...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Cto cache.nixos.org (54.230.62.218) port 443 (#0)
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-bundle.crt
  CApath: none
* TLSv1.2, TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2, TLS handshake, Server hello (2):
{ [74 bytes data]
* NPN, negotiated HTTP1.1
* TLSv1.2, TLS handshake, CERT (11):
{ [2318 bytes data]
* TLSv1.2, TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2, TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2, TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2, TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2, TLS handshake, Unknown (67):
} [36 bytes data]
* TLSv1.2, TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2, TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2, TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*        subject: serialNumber=752rFkZx9v83NlhSL4X71NxpvGYjbLUO; OU=GT33120842; O.rapidssl.com/resources/cps (c)14; OU=Domain Control Validated - RapidSSL(R); CN=os.org
*        start date: 2014-10-21 19:51:27 GMT
*        expire date: 2015-10-25 00:40:09 GMT
*        subjectAltName: cache.nixos.org matched
*        issuer: C=US; O=GeoTrust, Inc.; CN=RapidSSL CA
*        SSL certificate verify ok.
> GET /nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz HTTP/1.1
> Host: cache.nixos.org
> User-Agent: curl/7.42.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: binary/octet-stream
< Content-Length: 8384088
< Connection: keep-alive
< Date: Thu, 09 Jul 2015 10:04:31 GMT
< Last-Modified: Wed, 10 Jun 2015 21:59:05 GMT
< ETag: "3093a156fbce3aae2de004f94d570727"
< Accept-Ranges: bytes
< Server: AmazonS3
< Age: 10020
< X-Cache: Hit from cloudfront
< Via: 1.1 9596a84020c6947d4035fe9c12c2b966.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: c-1RYJcbi9qRu-eCGIFLzNAcMmN8_-BrwO_5F3NQL9LAl0bfnLeYPw==
< 
{ [13688 bytes data]
100 8187k  100 8187k    0     0   5449      0  0:25:38  0:25:38 --:--:-- 17746
* Connection #0 to host cache.nixos.org left intact
time_namelookup: 0,001
time_connect: 0,019
time_appconnect: 0,113
time_pretransfer: 0,113
time_redirect: 0,000
time_starttransfer: 0,240
time_total: 1538,483

cpages · 2015-07-15T19:56:20Z

@viric I just noticed that what you pasted as the 'fast case' connects to 54.230.63.229, which belongs to the pool of 'sometimes gets stuck' in your previous comment. I just found out when trying this workaround at home, and indeed it gets stuck ;)

rbvermaa · 2015-08-19T12:37:43Z

Amazon requested some more recent info:

ClientIP Address
dig resolver-identity.cloudfront.net
curl -v https://cache.nixos.org/nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz > /dev/null (or some other asset from your distribution)
MTR to your CloudFront distribution (let run for a few hundred samples)

viric · 2015-08-19T13:08:22Z

example client ip: 84.89.61.89
------------------
$ dig resolver-identity.cloudfront.net

; <<>> DiG 9.7.2-P3 <<>> resolver-identity.cloudfront.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34859
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;resolver-identity.cloudfront.net. IN   A

;; ANSWER SECTION:
resolver-identity.cloudfront.net. 10 IN A       85.62.233.81

;; Query time: 38 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Aug 19 14:42:00 2015
;; MSG SIZE  rcvd: 66

---------
$ curl -v https://cache.nixos.org/nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz > /dev/null
*   Trying 54.230.61.109...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to cache.nixos.org (54.230.61.109) port 443 (#0)
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-bundle.crt
  CApath: none
* TLSv1.2, TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2, TLS handshake, Server hello (2):
{ [74 bytes data]
* NPN, negotiated HTTP1.1
* TLSv1.2, TLS handshake, CERT (11):
{ [2318 bytes data]
* TLSv1.2, TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2, TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2, TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2, TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2, TLS handshake, Unknown (67):
} [36 bytes data]
* TLSv1.2, TLS handshake, Finished (20):
} [16 bytes data]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* TLSv1.2, TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2, TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*        subject: serialNumber=752rFkZx9v83NlhSL4X71NxpvGYjbLUO; OU=GT33120842; OU=See www.rapidssl.com/resources/cps (c)14; OU=Domain Control Validated - RapidSSL(R); CN=cache.nixos.org
*        start date: 2014-10-21 19:51:27 GMT
*        expire date: 2015-10-25 00:40:09 GMT
*        subjectAltName: cache.nixos.org matched
*        issuer: C=US; O=GeoTrust, Inc.; CN=RapidSSL CA
*        SSL certificate verify ok.
> GET /nar/00gq5dxn728g6557w3jrs39mn43q709jsasd363wgy9fg8rcisgi.nar.xz HTTP/1.1
> Host: cache.nixos.org
> User-Agent: curl/7.42.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: binary/octet-stream
< Content-Length: 8384088
< Connection: keep-alive
< Date: Wed, 19 Aug 2015 12:43:40 GMT
< Last-Modified: Wed, 10 Jun 2015 21:59:05 GMT
< ETag: "3093a156fbce3aae2de004f94d570727"
< Accept-Ranges: bytes
< Server: AmazonS3
< Age: 10
< X-Cache: Hit from cloudfront
< Via: 1.1 1b0e2c12e43b797479be5ac44b57db2c.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: 7DB7xdIilTKdtg7XEh1KzPhMCifJv9zBEDC6DikZW9QYS64wu-hkpQ==
< 
{ [13694 bytes data]
100 8187k  100 8187k    0     0  12627      0  0:11:03  0:11:03 --:--:-- 37046
* Connection #0 to host cache.nixos.org left intact

-------------
# mtr -n -c 400 -r 54.230.61.109
HOST: comanegra                   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.33.1               0.0%   400    0.4   0.4   0.4   1.3   0.1
  2.|-- 192.168.128.254            0.0%   400    0.7   0.6   0.6   3.6   0.2
  3.|-- 84.89.2.253                0.0%   400    6.7   6.0   3.6  19.5   1.9
  4.|-- 94.229.195.1               0.0%   400    4.1   7.8   3.9 197.3  17.4
  5.|-- 212.31.32.14               0.0%   400    5.5  16.4   3.8 201.8  37.7
  6.|-- 84.76.248.5                0.0%   400    4.3  10.5   4.0 376.7  30.9
  7.|-- 10.34.220.17               0.0%   400    5.4   6.1   4.4  60.0   4.2
  8.|-- ???                       100.0   400    0.0   0.0   0.0   0.0   0.0
  9.|-- 10.34.34.225               0.0%   400   11.2   7.1   5.1  25.5   2.8
 10.|-- 81.52.188.113              8.2%   400    5.8   6.1   4.9  37.8   2.6
 11.|-- 193.251.242.154            0.0%   400    7.0   7.4   4.7  17.1   2.3
 12.|-- 193.251.242.93             0.0%   400   21.5  15.6  12.7  24.7   2.2
 13.|-- 81.52.200.206              0.0%   400   17.4  15.6  12.8  24.5   2.2
 14.|-- 193.251.243.33             0.0%   400   15.9  31.8  13.4 151.0  32.2
 15.|-- 213.248.78.121             0.0%   400   13.3  14.7  13.0  29.2   2.8
 16.|-- 62.115.139.191             0.0%   400   13.8  15.1  13.6  28.9   2.5
 17.|-- 80.239.195.94              3.2%   400   14.1  15.4  14.0  51.8   3.5
 18.|-- 54.230.61.109              3.2%   400   14.2  15.3  14.0  23.8   2.2

viric · 2015-08-26T07:08:12Z

Dr Pepper said on irc that Amazon claims the issue to be fixed (problems in Madrid).

I confirm that all works more normal now.

rbvermaa · 2015-08-26T08:33:49Z

Amazon has fixed the issue, which turned out to be related to the edge location in Madrid. @viric confirmed that his downloads are fast again.

kirelagin · 2015-09-21T19:39:38Z

I also have surpisingly low download speed from cache.nixos.org. Tight now it’s around 300 Kb/s, but sometimes gets as low as 20 Kb/s. And this happens both at my home and at the office. I guess, I first notices this last Friday, but I’m not sure, things might be this way for longer time.

dlukes · 2015-11-18T16:28:30Z

I'm in Prague, Czech Republic, running the NixOS installer, and I can't confirm that the issue has been resolved :( I keep restarting nixos-install, every now and then a few packages get downloaded at the expected speed, followed by several attempts where the first download on the list slows to a crawl or stalls. Anything I can do to help? My public IP is 195.113.53.126.

vcunat · 2015-11-18T17:25:49Z

@dlukes: I just tried from 195.113.19.46 (haf.ms.mff.cuni.cz), wget http://cache.nixos.org/nar/1r7qx8fvm0nnppjpc9f09mi40gr4c9djmavb3b8ss9k8gi70h5sc.nar.xz (28 MB) for ~6.5 MB/s. That's not too good, as from a different UK network I'm ~10 MB/s on the same command.

vcunat · 2015-11-18T17:30:57Z

Hmm, perhaps it was some transient problem, as now I'm getting ~16 MB/s on haf.

dlukes · 2015-11-18T17:49:00Z

@vcunat yes, the speed's also got better here in the last 30 minutes, and especially more consistent, though less impressive :) I guess it was only a transient issue, sorry for making a fuss.

y-usuzumi · 2016-10-07T07:25:39Z

I'm from China. The download sometimes just drops to 0kb/s.

I'm new to NixOS. There might be some possible mirroring mechanism no?

Mic92 · 2016-10-07T22:30:13Z

vpn?

mkawalec · 2016-10-21T08:12:38Z

I'm routinely getting downloads at 160kb/s even though I have a 150mbit connection.

milahu · 2022-05-04T11:32:12Z

50 KByte = 0.4 Mbit on a 50 Mbit downlink. vpn via madrid, spain. normally much faster
probably just this one file. package is jre → openjdk-17.0.3+7

curl -O https://cache.nixos.org/nar/01vxcvf7l3dc1b0rny68zs8gviygfaiz0cgbqs831v15d20dzywi.nar.xz

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0  337M    0 1163k    0     0  74237      0  1:19:22  0:00:16  1:19:06 40456
  0  337M    0 1347k    0     0  71694      0  1:22:11  0:00:19  1:21:52 51594
  0  337M    0 1855k    0     0  63066      0  1:33:25  0:00:30  1:32:55 55141

dig +short cache.nixos.org
dualstack.v2.shared.global.fastly.net.
151.101.134.217

edit: download finished fast. just the start was slow

vcunat · 2022-05-04T11:46:21Z

@milahu: please don't extend this thread. Nowadays it's even a different CDN...

rbvermaa self-assigned this Jul 3, 2015

rbvermaa closed this as completed Aug 26, 2015

cpages mentioned this issue Dec 14, 2015

cache.nixos.org is slow #11669

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Too slow downloads from cache #8502

Too slow downloads from cache #8502

viric commented Jun 25, 2015

viric commented Jun 25, 2015

vcunat commented Jun 25, 2015

viric commented Jul 3, 2015

viric commented Jul 3, 2015

cpages commented Jul 9, 2015

rbvermaa commented Jul 9, 2015

viric commented Jul 9, 2015

cpages commented Jul 9, 2015

viric commented Jul 9, 2015

viric commented Jul 9, 2015

viric commented Jul 9, 2015

rbvermaa commented Jul 9, 2015

viric commented Jul 9, 2015

viric commented Jul 9, 2015

cpages commented Jul 15, 2015

rbvermaa commented Aug 19, 2015

viric commented Aug 19, 2015

viric commented Aug 26, 2015

rbvermaa commented Aug 26, 2015

kirelagin commented Sep 21, 2015

dlukes commented Nov 18, 2015

vcunat commented Nov 18, 2015

vcunat commented Nov 18, 2015

dlukes commented Nov 18, 2015

y-usuzumi commented Oct 7, 2016

Mic92 commented Oct 7, 2016

mkawalec commented Oct 21, 2016

milahu commented May 4, 2022 •

edited

Loading

vcunat commented May 4, 2022

Too slow downloads from cache #8502

Too slow downloads from cache #8502

Comments

viric commented Jun 25, 2015

viric commented Jun 25, 2015

vcunat commented Jun 25, 2015

viric commented Jul 3, 2015

viric commented Jul 3, 2015

cpages commented Jul 9, 2015

rbvermaa commented Jul 9, 2015

viric commented Jul 9, 2015

cpages commented Jul 9, 2015

viric commented Jul 9, 2015

viric commented Jul 9, 2015

viric commented Jul 9, 2015

rbvermaa commented Jul 9, 2015

viric commented Jul 9, 2015

viric commented Jul 9, 2015

cpages commented Jul 15, 2015

rbvermaa commented Aug 19, 2015

viric commented Aug 19, 2015

viric commented Aug 26, 2015

rbvermaa commented Aug 26, 2015

kirelagin commented Sep 21, 2015

dlukes commented Nov 18, 2015

vcunat commented Nov 18, 2015

vcunat commented Nov 18, 2015

dlukes commented Nov 18, 2015

y-usuzumi commented Oct 7, 2016

Mic92 commented Oct 7, 2016

mkawalec commented Oct 21, 2016

milahu commented May 4, 2022 • edited Loading

vcunat commented May 4, 2022

milahu commented May 4, 2022 •

edited

Loading