[NuGet.org Bug]: api.nuget.org is slow from some geographic locations

[mjolka]

Impact

It's more difficult to complete my work

Describe the bug

api.nuget.org is slow from some geographic locations, including where I'm based (Australia).

To test this, I created DigitalOcean Droplets in different regions and ran tests using ApacheBench.

This table shows the time taken to complete 100 requests for a resource on api.nuget.org, using HTTP Keep Alive.

Location	Time taken for tests (s)	Time taken for tests, relative to New York
New York	3.04	1.00
San Francisco	5.627	1.85
London	10.518	3.46
Singapore	22.089	7.27

When using NuGet Package Manager UI, the Updates and Installed tabs can make several hundred API requests. This combined with the slow response time degrades the user experience for people in certain locations.

From my home machine, I'm seeing times comparable to Singapore in the table above.

Detailed results:

New York

This is ApacheBench, Version 2.3 <$Revision: 1901567 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking api.nuget.org (be patient).....done


Server Software:        Windows-Azure-Blob/1.0
Server Hostname:        api.nuget.org
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Server Temp Key:        ECDH P-256 256 bits
TLS Server Name:        api.nuget.org

Document Path:          /v3/registration5-gz-semver2/microsoft.aspnetcore.mvc.testing/index.json
Document Length:        18694 bytes

Concurrency Level:      1
Time taken for tests:   3.040 seconds
Complete requests:      100
Failed requests:        0
Keep-Alive requests:    100
Total transferred:      1955600 bytes
HTML transferred:       1869400 bytes
Requests per second:    32.89 [#/sec] (mean)
Time per request:       30.402 [ms] (mean)
Time per request:       30.402 [ms] (mean, across all concurrent requests)
Transfer rate:          628.18 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.2      0      12
Processing:    28   30   7.9     29     105
Waiting:       28   30   7.9     29     105
Total:         28   30   9.1     29     117

Percentage of the requests served within a certain time (ms)
  50%     29
  66%     29
  75%     30
  80%     30
  90%     31
  95%     32
  98%     48
  99%    117
 100%    117 (longest request)

San Francisco

This is ApacheBench, Version 2.3 <$Revision: 1901567 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking api.nuget.org (be patient).....done


Server Software:        Windows-Azure-Blob/1.0
Server Hostname:        api.nuget.org
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Server Temp Key:        ECDH P-256 256 bits
TLS Server Name:        api.nuget.org

Document Path:          /v3/registration5-gz-semver2/microsoft.aspnetcore.mvc.testing/index.json
Document Length:        18694 bytes

Concurrency Level:      1
Time taken for tests:   5.627 seconds
Complete requests:      100
Failed requests:        0
Keep-Alive requests:    100
Total transferred:      1955600 bytes
HTML transferred:       1869400 bytes
Requests per second:    17.77 [#/sec] (mean)
Time per request:       56.272 [ms] (mean)
Time per request:       56.272 [ms] (mean, across all concurrent requests)
Transfer rate:          339.38 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.0      0      10
Processing:    55   56   2.0     56      75
Waiting:       55   56   2.0     56      75
Total:         55   56   2.4     56      75

Percentage of the requests served within a certain time (ms)
  50%     56
  66%     56
  75%     56
  80%     56
  90%     57
  95%     58
  98%     70
  99%     75
 100%     75 (longest request)

London

This is ApacheBench, Version 2.3 <$Revision: 1901567 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking api.nuget.org (be patient).....done


Server Software:        Windows-Azure-Blob/1.0
Server Hostname:        api.nuget.org
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Server Temp Key:        ECDH P-256 256 bits
TLS Server Name:        api.nuget.org

Document Path:          /v3/registration5-gz-semver2/microsoft.aspnetcore.mvc.testing/index.json
Document Length:        18694 bytes

Concurrency Level:      1
Time taken for tests:   10.518 seconds
Complete requests:      100
Failed requests:        0
Keep-Alive requests:    100
Total transferred:      1955600 bytes
HTML transferred:       1869400 bytes
Requests per second:    9.51 [#/sec] (mean)
Time per request:       105.182 [ms] (mean)
Time per request:       105.182 [ms] (mean, across all concurrent requests)
Transfer rate:          181.57 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.9      0       9
Processing:   100  105  29.7    101     394
Waiting:      100  105  29.7    101     394
Total:        100  105  30.6    101     403

Percentage of the requests served within a certain time (ms)
  50%    101
  66%    101
  75%    101
  80%    101
  90%    102
  95%    113
  98%    140
  99%    403
 100%    403 (longest request)

Singapore

This is ApacheBench, Version 2.3 <$Revision: 1901567 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking api.nuget.org (be patient).....done


Server Software:        Windows-Azure-Blob/1.0
Server Hostname:        api.nuget.org
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Server Temp Key:        ECDH P-256 256 bits
TLS Server Name:        api.nuget.org

Document Path:          /v3/registration5-gz-semver2/microsoft.aspnetcore.mvc.testing/index.json
Document Length:        18694 bytes

Concurrency Level:      1
Time taken for tests:   22.089 seconds
Complete requests:      100
Failed requests:        0
Keep-Alive requests:    100
Total transferred:      1955600 bytes
HTML transferred:       1869400 bytes
Requests per second:    4.53 [#/sec] (mean)
Time per request:       220.891 [ms] (mean)
Time per request:       220.891 [ms] (mean, across all concurrent requests)
Transfer rate:          86.46 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.8      0       8
Processing:   214  221  63.7    214     851
Waiting:      214  221  63.7    214     851
Total:        214  221  64.5    214     860

Percentage of the requests served within a certain time (ms)
  50%    214
  66%    214
  75%    215
  80%    215
  90%    215
  95%    216
  98%    219
  99%    860
 100%    860 (longest request)

Repro Steps

1. Install ApacheBench

apt-get install -y apache2-utils

2. Run tests

ab -n 100 -k https://api.nuget.org/v3/registration5-gz-semver2/microsoft.aspnetcore.mvc.testing/index.json

Expected Behavior

Response times should not vary so dramatically with geographic location.

Screenshots

No response

Additional Context and logs

After further testing, I found that this does not apply to https://api.nuget.org/v3/index.json.

For /v3/index.json, the Server response header value (ECAcc (mbw/47ED)) shows that it is being served from an Edgio PoP, while for other requests the value is Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0.

[joelverhagen]

Hey @mjolka, thanks for the detailed report. tl;dr: this is a known issue but we don't have a date we can share of when it will be improved. That being said, on the client side, we've increase parallelism in VS per NuGet/Home#11923 (which I see you've also originally posted, thank you for the focus on performance!).

This is a known issue primarily affecting the performance of the Package Manager UI, as you've stated. There are other JSON endpoints with the same symptom you're seeing (slower latency across geos) but, based on end-to-end performance tests (e.g. total dotnet restore duration) the slower response time does not significantly impact the overall experience. Unfortunately, this disconnect between endpoint latency and overall operation latency is not the case with the package metadata endpoint (/v3/registration5-gz-semver2-like URLs) and the lower latency across geos indeed impacts the responsiveness of the UI (again, confirming what you're saying).

Generally speaking, for the "api.nuget.org" domain, we have a variety of caching mechanisms and several geographic locations backing the CDN nodes which serve requests. For example, we have a dedicated storage CDN and storage backend in China yielding superior performance in this case. The "https://api.nuget.org/v3/index.json" URL is highly cached at the CDN level which is why the performance is good across geos. Similarly, .nupkg and .nuspec URLs are highly cached too.

We've been unable to cache JSON endpoints in most cases due to concerns about data consistency across multiple URLs that the client interacts with in a session but it's an area we're experimenting with for both performance and availability reasons. I don't have an ETA for you, however.

For the particular user scenario of the Browse tab in the PM UI, we have #7297 planned which will alleviate the need for so many HTTP requests, thus reducing the impact of lower latency across geos. Unfortunately, this work will not help the Installed or Updates tab which need to pull information about a set of IDs known only to the client (compared to the Browse tab where the set of IDs is provided by the server).

Thanks for the high-quality issue report. I'll leave this issue open since it is indeed a problem but I can't provide you a timeline of when it will get better.

[murshex]

This is still a problem. from Bangkok

[tyeth]

@joelverhagen has there been any movement on the CDN being slow in certain regions?

I'm seeing unreasonable slowness in the UK getting this package:
https://api.nuget.org/v3-flatcontainer/syncfusion.blazor.themes/27.2.2/syncfusion.blazor.themes.27.2.2.nupkg
It's not just slow, but unbelievably slow (less than 5kb/s on an 8gig connection). I've also switched UK locations via a vpn, but no real difference, the downloads speed along for the first bit of cached (at the CDN) data, then come at treacle speed.

It's also like the first 8mb has been cached (by my previous attempts) but the remainder is coming at 15kb/s with pauses. I'm up to 9mb now, but you can imagine how long this has taken. Other files didn't appear to have the same issue when I installed syncfusion.blazor.inputs, presumably due to smaller sizes.
Jump on a VPN and get into the UK, try to download that file, I've cancelled mine so the cache doesn't finish the file.
It might be worth noting I'm using 1.1.1.1(cloudflare) for DNS on my home router.

[zhhyu]

Hi! @tyeth My apologies for the inconvenience! Multiple customers reported that CDN is slow in UK. We have switched the traffic to another CDN service. Feel free to reach out to support (https://www.nuget.org/policies/Contact) if the issue is still ongoing.