Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix corner case in cached dependency resolution #362

Merged
merged 2 commits into from
Feb 22, 2017

Conversation

butlern
Copy link
Contributor

@butlern butlern commented Feb 17, 2017

Ref: #178

The error is as follows:

  1. Two resolvables, one pinned, the other not.
  2. First pex run with clean cache works.
  3. Change the pinned version.
  4. Second pex run fails with Unsatisfiable.

The problem is that the non-pinned resolvable locates the cached dep
and only returns it. Since that version doesn't match the pinned
version, we fail. This fixes that behavior by returning both the
cached version and all uncached pypi packages for unpinned
resolvables.

We change the pinned version to test the use case where you have a
cached version of a dep and then you transitively pull in another
requirement that either has a tighter bound or a pin on a version not
found in the cache.

  Ref: pex-tool#178

  The error is as follows:

  1. Two resolvables, one pinned, the other not.
  2. First pex run with clean cache works.
  3. Change the pinned version.
  4. Second pex run fails with Unsatisfiable.

  The problem is that the non-pinned resolvable locates the cached dep
  and only returns it. Since that version doesn't match the pinned
  version, we fail. This fixes that behavior by returning both the
  cached version and all uncached pypi packages for unpinned
  resolvables.

  We change the pinned version to test the use case where you have a
  cached version of a dep and then you transitively pull in another
  requirement that either has a tighter bound or a pin on a version not
  found in the cache.
@butlern butlern force-pushed the fix_cached_dependency_resolution_bug branch from 15ccd05 to f18543b Compare February 18, 2017 00:03
Copy link
Contributor

@kwlzn kwlzn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm w/ one minor thought. thanks for fixing this bug!

pex/resolver.py Outdated
# packages because the cached version might not match another resolvable which has a tighter
# bound or an exact version.
return packages + super(CachingResolver, self).package_iterator(resolvable,
existing=existing)

return super(CachingResolver, self).package_iterator(resolvable, existing=existing)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assuming packages == [] in the non-cached case, how would you feel about killing the slight bit of super().package_iterator() call duplication here (lines 255 + 258) by always returning a combined iterable that includes packages - whether empty or not - in the main return statement?

e.g.

  if self.__cache_ttl:
    packages = ...

return itertools.chain(packages, super(...).package_iterator(...))

Copy link
Contributor Author

@butlern butlern Feb 21, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So something like this?

  def package_iterator(self, resolvable, existing=None):
    iterator = Iterator(fetchers=[Fetcher([self.__cache])])
    packages = self.filter_packages_by_interpreter(
        resolvable.compatible(iterator), self._interpreter, self._platform)

    if packages:
      if resolvable.exact:
        return packages

      if self.__cache_ttl:
        packages = self.filter_packages_by_ttl(packages, self.__cache_ttl)
      else:
        packages = []

    return itertools.chain(packages, super(CachingResolver, self).package_iterator(resolvable, existing=existing))

Copy link
Contributor

@kwlzn kwlzn Feb 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, you could go a step further and guard the population of packages altogether by checking for the conditions in which it'd be used - was sort of imagining something along the lines of:

  def package_iterator(self, resolvable, existing=None):
    packages = []
    if self.__cache_ttl or resolvable.exact:
      iterator = Iterator(fetchers=[Fetcher([self.__cache])])
      packages = self.filter_packages_by_interpreter(
        resolvable.compatible(iterator),
        self._interpreter,
        self._platform
      )

      if packages:
        if resolvable.exact:
          return packages

        if self.__cache_ttl:
          packages = self.filter_packages_by_ttl(packages, self.__cache_ttl)

    return itertools.chain(
      packages,
      super(CachingResolver, self).package_iterator(resolvable, existing=existing)
    )

but I'm probably just splitting hairs at this point - fine with any of the above.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha, yah that looks good to me. I'll push up the change. For now I'll do a separate autosquashable commit unless you'd prefer it be a separate commit. Local tests run fine with that change btw :)

@kwlzn kwlzn merged commit 731d124 into pex-tool:master Feb 22, 2017
@kwlzn
Copy link
Contributor

kwlzn commented Feb 22, 2017

merged - thanks for the PR @butlern !

@kwlzn kwlzn mentioned this pull request Feb 22, 2017
@butlern butlern deleted the fix_cached_dependency_resolution_bug branch February 22, 2017 01:48
butlern added a commit to butlern/pex that referenced this pull request Jun 30, 2017
Fixes pex-tool#178

The error is as follows:

Two resolvables, one pinned, the other not.
First pex run with clean cache works.
Change the pinned version.
Second pex run fails with Unsatisfiable.
The problem is that the non-pinned resolvable locates the cached dep
and only returns it. Since that version doesn't match the pinned
version, we fail. This fixes that behavior by returning both the
cached version and all uncached pypi packages for unpinned
resolvables.

We change the pinned version to test the use case where you have a
cached version of a dep and then you transitively pull in another
requirement that either has a tighter bound or a pin on a version not
found in the cache.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants