Very slow HTTP connection to crates.io

[dometto]

Locally, the rust specs are taking a looong time to run. This is because the RustLang#last_version method is doing an HTTP request to crates.io which is slow, and sometimes times out, causing the specs to fail.

Weirdly, I am:

• Not seeing this issue when I just curl e.g. https://crates.io/api/v1/crates/log
• Am seeing when I use Ruby's Net::HTTP instead of http-rb.
• Not seeing it on Travis

Possibly some low-level Ruby issue on my system? @bartkamphorst could you see how it works locally for you?

As mentioned in #17, the specs should be mocking the http requests anyway. But this issue has got me thinking: what should we do in case a last_version HTTP request times out? Currently it just returns nil. Should we be trying to cache last_version calls (possibly in a separate Moneta store)? If so, how long should it take for a cached result to expire?

[dometto]

Can confirm this must be a ruby issue of some kind, as I don't get the timeouts when running the app with jruby.

Incidentally, the app takes up between 600-700MB of RAM with JRuby. 😆 (But runs nice and fast!)
(EDIT: actually, telling JRuby to use less memory works fine -- bundle exec jruby -J-Xmx200m bin/launch.rb causes only ~320MB memory use. Presumably JRuby is just utilizing as much memory as possible for the workers when its allowed.)

[dometto]

what should we do in case a last_version HTTP request times out? Currently it just returns nil. Should we be trying to cache last_version calls (possibly in a separate Moneta store)? If so, how long should it take for a cached result to expire?

In last_version, we are currently rescuing timeouts for Python and Rust, and rescuing StandardError for Ruby. A StandarError will result if, due to a timeouts, the Gem::SpecFetcher fails to return a spec. So we're all good on this front.