Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pex does not resolve packages using -i switch when using pypi-server #270

Closed
petmakris opened this issue May 26, 2016 · 6 comments
Closed

pex does not resolve packages using -i switch when using pypi-server #270

petmakris opened this issue May 26, 2016 · 6 comments

Comments

@petmakris
Copy link

petmakris commented May 26, 2016 •
edited
Loading

When using as ~/.pip/pip.conf

[global]
index-url = http://localhost:8080/simple

pip resolves packages correctly.
If using the -i switch to declare this particular index with pex the following happens:

Traceback (most recent call last):tributions :: Packaging acet-tracing-steward
  File "/opt/python/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/opt/python/lib/python2.6/site-packages/pex/bin/pex.py", line 540, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/opt/python/lib/python2.6/site-packages/pex/bin/pex.py", line 489, in build_pex
    resolveds = resolver.resolve(resolvables)
  File "/opt/python/lib/python2.6/site-packages/pex/resolver.py", line 200, in resolve
    dist = self.build(package, resolvable.options)
  File "/opt/python/lib/python2.6/site-packages/pex/resolver.py", line 253, in build
    package = Package.from_href(options.get_context().fetch(package, into=self.__cache))
  File "/opt/python/lib/python2.6/site-packages/pex/http.py", line 111, in fetch
    shutil.copyfileobj(in_fp, out_fp)
  File "/opt/python/lib/python2.6/shutil.py", line 28, in copyfileobj
    buf = fsrc.read(length)
  File "/opt/python/lib/python2.6/site-packages/pex/http.py", line 169, in read
    self._validate()
  File "/opt/python/lib/python2.6/site-packages/pex/http.py", line 179, in _validate
    raise Context.Error('%s failed checksum!' % (self._link.url))
pex.http.Error: http://localhost:8080/packages/f0/07/26b519e6ebb03c2a74989f7571e6ae6b82e9d7d81b8de6fcdbfc643c7b58/simplejson-3.8.2.tar.gz#md5=53b1371bbf883b129a12d594a97e9a18 failed checksum!

At this point the contents of ~/.pex/build will be

-rw-r--r--. 1 pmakris pmakris    0 May 26 11:33 simplejson-3.8.2.tar.gz.a9fa70be-c37b-4010-bb5b-c15aec8cb638

This works fine with artifactory as a pypi server.
@petmakris
Copy link
Author

Works with localshop and artifactory. pypiserver is not actually fetching packages on disk and mirroring maybe with a redirect or don't know why, and maybe that's why it's failing. Maybe it could be closed if no-one is interested in using 'pypiserver' a local pypi mirror.

@kwlzn
Copy link
Contributor

kwlzn commented May 26, 2016

does adding a trailing slash improve things at all? e.g.

index-url = http://localhost:8080/simple/

if not, repro steps on how to setup a similar pypiserver instance would be helpful.

@petmakris
Copy link
Author

petmakris commented May 26, 2016 •
edited
Loading

Installed pypi-server version 1.1.10:

$ pypi-server --version
pypiserver 1.1.10

Started pypi server using:

/opt/python26/bin/pypi-server -v --overwrite -p 8080 -P /home/user/applications/pypi-packages/htpasswd.txt --log-file /home/user/applications/pypi-packages/pypi-server.log /home/user/applications/pypi-packages >/dev/null 2>&1 &

Using this .pypirc

[distutils]
index-servers =
    pypi
    local

[pypi]
username:
password:

[local]
repository: http://localhost:8080
username: user
password: secret

Using the following ~/.pip/pip.conf:

[global]
index-url = http://localhost:8080/simple/

Until now an upload to local works and I can fetch any package with pip install, even the failing ioloop package.

But when I try to execute pex:

pex -v . -r requirements.text -i http://localhost:8080/simple -c esmon.app -o products/esmon.app

with requirements.text:

addict==0.4.0
backports-abc==0.4
backports.ssl-match-hostname==3.5.0.1
certifi==2016.2.28
colorama==0.3.7
ioloop==0.1a
Jinja2==2.8
logutils==0.3.3
MarkupSafe==0.23
pex==1.1.4
PyYAML==3.11
rainbow-logging-handler==2.2.2
requests==2.9.1
schedule==0.3.2
simplejson==3.8.0
singledispatch==3.4.0.3
six==1.10.0
tornado==4.3

I get:

Traceback (most recent call last):
  File "/home/pmakris/projects/acet-tracing-admin-future/src/.esmon/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/home/pmakris/projects/acet-tracing-admin-future/src/.esmon/lib/python2.6/site-packages/pex/bin/pex.py", line 509, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/home/pmakris/projects/acet-tracing-admin-future/src/.esmon/lib/python2.6/site-packages/pex/bin/pex.py", line 471, in build_pex
    resolveds = resolver.resolve(resolvables)
  File "/home/pmakris/projects/acet-tracing-admin-future/src/.esmon/lib/python2.6/site-packages/pex/resolver.py", line 200, in resolve
    dist = self.build(package, resolvable.options)
  File "/home/pmakris/projects/acet-tracing-admin-future/src/.esmon/lib/python2.6/site-packages/pex/resolver.py", line 253, in build
    package = Package.from_href(options.get_context().fetch(package, into=self.__cache))
  File "/home/pmakris/projects/acet-tracing-admin-future/src/.esmon/lib/python2.6/site-packages/pex/http.py", line 111, in fetch
    shutil.copyfileobj(in_fp, out_fp)
  File "/opt/python/lib/python2.6/shutil.py", line 28, in copyfileobj
    buf = fsrc.read(length)
  File "/home/pmakris/projects/acet-tracing-admin-future/src/.esmon/lib/python2.6/site-packages/pex/http.py", line 169, in read
    self._validate()
  File "/home/pmakris/projects/acet-tracing-admin-future/src/.esmon/lib/python2.6/site-packages/pex/http.py", line 179, in _validate
    raise Context.Error('%s failed checksum!' % (self._link.url))
pex.http.Error: http://localhost:8080/packages/96/ac/d55158ea1150d71bf3a6287c6d95fe4ee1cbeb9ea7143a1999b279b8da47/ioloop-0.1a.tar.gz#md5=5ee57d607ba033faa4633927a2bbb7c4 failed checksum!

Note that I am using python 2.6 as I need to create pex files for my production env which is really old but cannot be updated.

$ python --version
Python 2.6.6

If I exclude the -i http:// from pex's command line pex succeeds with downloading from global pypi.

The trailing "/" makes no difference.

@petmakris
Copy link
Author

Interesting is that if I don't have requests in my path:

pex -v . -r requirements.text -i http://localhost:8080/simple/ -c esmon.app -o products/esmon.app
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
Traceback (most recent call last):
  File "/opt/python/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/opt/python/lib/python2.6/site-packages/pex/bin/pex.py", line 540, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/opt/python/lib/python2.6/site-packages/pex/bin/pex.py", line 489, in build_pex
    resolveds = resolver.resolve(resolvables)
  File "/opt/python/lib/python2.6/site-packages/pex/resolver.py", line 200, in resolve
    dist = self.build(package, resolvable.options)
  File "/opt/python/lib/python2.6/site-packages/pex/resolver.py", line 253, in build
    package = Package.from_href(options.get_context().fetch(package, into=self.__cache))
  File "/opt/python/lib/python2.6/site-packages/pex/http.py", line 109, in fetch
    with contextlib.closing(self.open(link)) as in_fp:
  File "/opt/python/lib/python2.6/site-packages/pex/http.py", line 121, in open
    return urllib_request.urlopen(link.url)
  File "/opt/python/lib/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/opt/python/lib/python2.6/urllib2.py", line 397, in open
    response = meth(req, response)
  File "/opt/python/lib/python2.6/urllib2.py", line 510, in http_response
    'http', request, response, code, msg, hdrs)
  File "/opt/python/lib/python2.6/urllib2.py", line 435, in error
    return self._call_chain(*args)
  File "/opt/python/lib/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/opt/python/lib/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found
make: *** [esmon.app] Error 1

The pypi-server reports:

<LocalRequest: GET http://localhost:8080/simple/addict/>
200 OK
<LocalRequest: GET http://localhost:8080/simple/backports-abc/>
200 OK
<LocalRequest: GET http://localhost:8080/simple/backports.ssl-match-hostname/>
200 OK
<LocalRequest: GET http://localhost:8080/simple/certifi/>
200 OK
<LocalRequest: GET http://localhost:8080/simple/colorama/>
200 OK
<LocalRequest: GET http://localhost:8080/simple/ioloop/>
200 OK
<LocalRequest: GET http://localhost:8080/simple/Jinja2/>
200 OK
<LocalRequest: GET http://localhost:8080/simple/logutils/>
200 OK
<LocalRequest: GET http://localhost:8080/simple/MarkupSafe/>
200 OK
<LocalRequest: GET http://localhost:8080/simple/PyYAML/>
200 OK
<LocalRequest: GET http://localhost:8080/simple/rainbow-logging-handler/>
200 OK
<LocalRequest: GET http://localhost:8080/simple/schedule/>
200 OK
<LocalRequest: GET http://localhost:8080/simple/simplejson/>
200 OK
<LocalRequest: GET http://localhost:8080/simple/singledispatch/>
200 OK
<LocalRequest: GET http://localhost:8080/simple/six/>
200 OK
<LocalRequest: GET http://localhost:8080/simple/tornado/>
200 OK
<LocalRequest: GET http://localhost:8080/packages/96/ac/d55158ea1150d71bf3a6287c6d95fe4ee1cbeb9ea7143a1999b279b8da47/ioloop-0.1a.tar.gz%23md5%3D5ee57d607ba033faa4633927a2bbb7c4>

And if I install requests:

/opt/python/lib/python2.6/site-packages/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/opt/python/lib/python2.6/site-packages/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
Traceback (most recent call last):
  File "/opt/python/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/opt/python/lib/python2.6/site-packages/pex/bin/pex.py", line 540, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/opt/python/lib/python2.6/site-packages/pex/bin/pex.py", line 489, in build_pex
    resolveds = resolver.resolve(resolvables)
  File "/opt/python/lib/python2.6/site-packages/pex/resolver.py", line 200, in resolve
    dist = self.build(package, resolvable.options)
  File "/opt/python/lib/python2.6/site-packages/pex/resolver.py", line 253, in build
    package = Package.from_href(options.get_context().fetch(package, into=self.__cache))
  File "/opt/python/lib/python2.6/site-packages/pex/http.py", line 111, in fetch
    shutil.copyfileobj(in_fp, out_fp)
  File "/opt/python/lib/python2.6/shutil.py", line 28, in copyfileobj
    buf = fsrc.read(length)
  File "/opt/python/lib/python2.6/site-packages/pex/http.py", line 169, in read
    self._validate()
  File "/opt/python/lib/python2.6/site-packages/pex/http.py", line 179, in _validate
    raise Context.Error('%s failed checksum!' % (self._link.url))
pex.http.Error: http://localhost:8080/packages/96/ac/d55158ea1150d71bf3a6287c6d95fe4ee1cbeb9ea7143a1999b279b8da47/ioloop-0.1a.tar.gz#md5=5ee57d607ba033faa4633927a2bbb7c4 failed checksum!

and pypi-server reports:

<LocalRequest: GET http://localhost:8080/packages/96/ac/d55158ea1150d71bf3a6287c6d95fe4ee1cbeb9ea7143a1999b279b8da47/ioloop-0.1a.tar.gz>

@petmakris
Copy link
Author

petmakris commented May 28, 2016 •
edited
Loading

After lots of testing I understand the following:

Pex does not resolve packages from local repositories, at least the .tgz sources do not behave well. I understand the complexity of the resolver.py & http.py, and the work is much appreciated, but the problem is there, maybe a tighter integration with pip could solve the problem (use pip's api for fetching packages, or not use fetch but get from pips cache).

Modern workflows require private pypi repos, pex should work with them :/

My current workflow is not to use -i for pex (download everything from public pypi) and copy my wheels in ~/.pex/build (pex's cache).

Any suggestions please welcomed.

@petmakris
Copy link
Author

Just for people that read this thread, you may work with pex and private repos by using pip for package resolving then load wheel dependencies from local cache

pip wheel -w wheel_dir ...
pex --disable-cache  -f wheel_dir --no-index ...

kxepal added a commit to kxepal/pex that referenced this issue Feb 14, 2017
@kxepal
Follow redirects on HTTP requests
1ea9e82 
Lack of redirects support is notable when local PyPI server is used
with fallback to official one. For example, that's how pypiserver
implementation works.

On the PyPI pages all the package links are relative, not absolute.
What means that we cannot use original request url to construct full
package url to fetch it, we should respect the url to where we were
redirected.

Without redirects resolution we end with the urls which our pypiserver
cannot process and the HTTP 404 response obliviously will never match
the md5 package digest, so we'll fail on verification stage in anyway.

This fixes pex-tool#200 pex-tool#270
@kxepal kxepal mentioned this issue Feb 14, 2017
Merged
kxepal added a commit to kxepal/pex that referenced this issue Feb 14, 2017
@kxepal
Follow redirects on HTTP requests
e28f520 
Lack of redirects support is notable when local PyPI server is used
with fallback to official one. For example, that's how pypiserver
implementation works.

On the PyPI pages all the package links are relative, not absolute.
What means that we cannot use original request url to construct full
package url to fetch it, we should respect the url to where we were
redirected.

Without redirects resolution we end with the urls which our pypiserver
cannot process and the HTTP 404 response obliviously will never match
the md5 package digest, so we'll fail on verification stage in anyway.

This fixes pex-tool#200 pex-tool#270
kxepal added a commit to kxepal/pex that referenced this issue Feb 14, 2017
@kxepal
Follow redirects on HTTP requests
4e13f2f 
Lack of redirects support is notable when local PyPI server is used
with fallback to official one. For example, that's how pypiserver
implementation works.

On the PyPI pages all the package links are relative, not absolute.
What means that we cannot use original request url to construct full
package url to fetch it, we should respect the url to where we were
redirected.

Without redirects resolution we end with the urls which our pypiserver
cannot process and the HTTP 404 response obliviously will never match
the md5 package digest, so we'll fail on verification stage in anyway.

This fixes pex-tool#200 pex-tool#270
@kwlzn kwlzn closed this as completed in 6ed76cb Feb 14, 2017
@npwolf npwolf mentioned this issue May 1, 2017
Closed
@jdevilla20 jdevilla20 mentioned this issue Jan 31, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kwlzn @petmakris