Unable to find gem on remote

[indirect]

This is a kind of weird and specific situation:

If, and only if, some version of a gem is installed into system gems
(e.g. rack 1.0), then trying to resolve a different version of that gem
from a specific remote source (e.g. rack 2.0.1.1.forked) will fail, even
though that gem is in fact available from that soure. Uninstalling the
unrelated version of rack from system gems completely solves this
problem. Alternately, running bundle update rack instead of bundle install also completely solves this problem.

From this I speculate that we are unfairly withholding the rack
2.0.1.1.forked spec from the resolver, even though it was provided by
the source.

I have written a failing spec, available in the branch named aa-missing-specs.

Update: after looking in to the problem at a bit more length, it seems that we are in fact supplying the relevant specs to the resolver. Here's some debug output that might help point to the problem. Maybe.

Failing case debug output

Resolver#start `requirements` ["rack (= 2.0.1.1.forked)", "thor (= 0.19.1.1.forked)", "rack-obama", "ruby\u0000", "rubygems\u0000 (= 2.6.7)"]

Resolver#start `index.search('rack')` ["rack-0.9.1", "rack-1.0.0", "rack-2.0.1.1.forked"]

Resolver#start `@base_dg` Bundler::Molinillo::DependencyGraph:[Bundler::Molinillo::DependencyGraph::Vertex:rack(#<Bundler::DepProxy:0x007ff6a7913e80 @dep=<Bundler::Dependency type=:runtime name="rack" requirements="= 2.0.1.1.forked" prerelease=ok>, @__platform="ruby">), Bundler::Molinillo::DependencyGraph::Vertex:rack-obama(#<Bundler::DepProxy:0x007ff6a7913778 @dep=<Bundler::Dependency type=:runtime name="rack-obama" requirements="= 1.0">, @__platform="ruby">), Bundler::Molinillo::DependencyGraph::Vertex:thor(#<Bundler::DepProxy:0x007ff6a7913228 @dep=<Bundler::Dependency type=:runtime name="thor" requirements="= 0.19.1.1.forked" prerelease=ok>, @__platform="ruby">)]
Resolving dependencies...

Starting resolution (2016-10-13 20:42:53 -0700)
Resolving dependencies...Creating possibility state for rack (= 2.0.1.1.forked) (1 remaining)
  Attempting to activate rack (2.0.1.1.forked)
  Activated rack at rack (2.0.1.1.forked)
  Requiring nested dependencies ()
  Creating possibility state for thor (= 0.19.1.1.forked) (1 remaining)
    Attempting to activate thor (0.19.1.1.forked)
    Activated thor at thor (0.19.1.1.forked)
    Requiring nested dependencies ()
    Creating possibility state for rack-obama (1 remaining)
      Attempting to activate rack-obama (1.0)
      Activated rack-obama at rack-obama (1.0)
      Requiring nested dependencies (rack)
      Creating possibility state for rack (0 remaining)
        Unwinding for conflict: rack

Finished resolution (4 steps) (Took 0.003739 seconds) (2016-10-13 20:42:53 -0700)

#<Bundler::Molinillo::VersionConflict: Unable to satisfy the following requirements:

- `rack (= 2.0.1.1.forked)` required by `Gemfile`
- `rack (= 2.0.1.1.forked)` required by `Gemfile.lock`
- `rack` required by `rack-obama (1.0)`>

Success case debug output

Resolver#start `requirements` ["rack (= 2.0.1.1.forked)", "thor (= 0.19.1.1.forked)", "rack-obama", "ruby\u0000", "rubygems\u0000 (= 2.6.7)"]

Resolver#start `index.search('rack')` ["rack-0.9.1", "rack-1.0.0", "rack-2.0.1.1.forked"]

Resolver#start `@base_dg` Bundler::Molinillo::DependencyGraph:[Bundler::Molinillo::DependencyGraph::Vertex:rack(#<Bundler::DepProxy:0x007f848b15f448 @dep=<Bundler::Dependency type=:runtime name="rack" requirements="= 2.0.1.1.forked" prerelease=ok>, @__platform="ruby">), Bundler::Molinillo::DependencyGraph::Vertex:rack-obama(#<Bundler::DepProxy:0x007f848b15ec50 @dep=<Bundler::Dependency type=:runtime name="rack-obama" requirements="= 1.0">, @__platform="ruby">), Bundler::Molinillo::DependencyGraph::Vertex:thor(#<Bundler::DepProxy:0x007f848b15e2f0 @dep=<Bundler::Dependency type=:runtime name="thor" requirements="= 0.19.1.1.forked" prerelease=ok>, @__platform="ruby">)]

Starting resolution (2016-10-13 20:44:51 -0700)
Creating possibility state for rack (= 2.0.1.1.forked) (1 remaining)
  Attempting to activate rack (2.0.1.1.forked)
  Activated rack at rack (2.0.1.1.forked)
  Requiring nested dependencies ()
  Creating possibility state for thor (= 0.19.1.1.forked) (1 remaining)
    Attempting to activate thor (0.19.1.1.forked)
    Activated thor at thor (0.19.1.1.forked)
    Requiring nested dependencies ()
    Creating possibility state for rack-obama (1 remaining)
      Attempting to activate rack-obama (1.0)
      Activated rack-obama at rack-obama (1.0)
      Requiring nested dependencies (rack)
      Creating possibility state for rack (1 remaining)
        Attempting to activate rack (2.0.1.1.forked)
        Found existing spec (rack (2.0.1.1.forked))
        Creating possibility state for ruby (1 remaining)
          Attempting to activate ruby (2.3.1.112)
          Activated ruby at ruby (2.3.1.112)
          Requiring nested dependencies ()
          Creating possibility state for rubygems (= 2.6.7) (1 remaining)
            Attempting to activate rubygems (2.6.7)
            Activated rubygems at rubygems (2.6.7)
            Requiring nested dependencies ()
Finished resolution (6 steps) (Took 0.001092 seconds) (2016-10-13 20:44:51 -0700)
Unactivated:
Activated: rack, thor, rack-obama, ruby, rubygems

[indirect]

/cc @segiddins here is the ticket that goes with the branch with the failing spec 👍

[chrismo]

is this the one you mentioned where the GEM_PATH plays a role in the problem?

[indirect]

@chrismo yup! as shown in the test, if rack is installed in system gems, this fails. if rack is not installed in system gems, this succeeds. I think maybe this means there is a bug either in Index.search or in the way that Molinillo looks inside the Index object for specs.

[indirect]

(The reason that setting GEM_PATH "fixed" the bug is that it meant Bundler could no longer see the already-installed copy of rack, and so the problem went away.)

[chrismo]

and this is broken on master? do you know what released versions this works with / fails with?

and ... double-checking ... yeah, this code is put back the way it was: https://github.com/bundler/bundler/blob/aa-missing-specs/lib/bundler.rb#L496 - so probably not that.

[indirect]

According to git bisect, this commit (1.8.5) is the first failing commit: 8571963.

[indirect]

So somehow... all 1.8.x releases, and the 1-8-stable branch, do not have this bug. But this bug arrived on master when the tag v1.8.5 was merged in. 😬

[indirect]

Ahh, crap, I copied and pasted the wrong SHA. The first failing commit is the merge commit that brought 8571963 onto master, and that SHA (the real first failing SHA) is ca746cb.

[segiddins]

I have a one-line fix for this that proves my suspicion about this being pre-release related 😝