Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

distribute data_files directory with package regardless of installation method (setup.py/requirements.txt/wheel) [$250] #142

Closed
arne-cl opened this issue Apr 30, 2016 · 9 comments
Closed

distribute data_files directory with package regardless of installation method (setup.py/requirements.txt/wheel) [$250] #142

arne-cl opened this issue Apr 30, 2016 · 9 comments
Labels
bug data help wanted

Comments

@arne-cl
Copy link
Owner

arne-cl commented Apr 30, 2016

The discoursegraphs package needs to distribute a directory (including subdirectories) called data that contains non-Python files . The package structure looks like this:

├── data
│   └── ...
├── MANIFEST.in
├── requirements.txt
├── setup.py
├── src
│   └── discoursegraphs
│       ├── ...

I managed to get this working for installations via setup.py and requirements.txt, but not
for sdist/wheel packages that I can upload to PyPI. (sdist seems to work, but never versions
of pip call bdist_wheel after downloading an sdist package.)

I need a clean solution for this seemingly simple problem. People should be able to
install this package via PyPI (with a current version of pip) as well as via setup.py
and requirements.txt (after getting the sources from github.)

Installation using setup.py

python setup.py install
python
>>> import discoursegraphs
>>> discoursegraphs.DATA_ROOT_DIR
'/home/arne/.virtualenvs/test/lib/python2.7/site-packages/discoursegraphs-0.3.0-py2.7.egg/data'
>>> len(discoursegraphs.corpora.pcc)
176

Installation using requirements.txt

pip install -r requirements.txt
python
>>> import discoursegraphs
>>> discoursegraphs.DATA_ROOT_DIR
'/home/arne/repos/discoursegraphs/data'
>>> len(discoursegraphs.corpora.pcc)
176

Installation using wheel

People installing the package via PyPI will run into the same problem.

python setup.py sdist bdist_wheel
pip install dist/discoursegraphs-0.3.0-py2-none-any.whl
python
>>> import discoursegraphs
>>> discoursegraphs.DATA_ROOT_DIR # this directory does not exist
'/home/arne/.virtualenvs/test/lib/python2.7/data'
>>> len(discoursegraphs.corpora.pcc) # therefore, we can't find the files ...
0
@hernan-erasmo hernan-erasmo mentioned this issue May 4, 2016
Closed
@hernan-erasmo
Copy link
Contributor

hernan-erasmo commented May 4, 2016 •
edited
Loading

Hi,

We'll just have to wait and see what wheel mantainers do about this and that, but in the meantime perhaps this fix might help.

I realized that the code inside discoursegraphs/src/init.py wasn't handling correctly the different paths generated by different methods of installation (using setup.py, requirements.txt or wheel).

I'm specifically talking about this method which handles correctly the cases where the user installs discoursegraphs using setup.py and requirements.txt, but fails when using wheel (although the files ARE there) because DATA_ROOT_DIR ends up pointing to a wrong location within the package tree.

The data dir, the same one that is pointed to by DATA_ROOT_DIR when installing with requirements.txt, is just a couple more dirs above in the path.

Cheers!

@arne-cl
Copy link
Owner Author

arne-cl commented May 7, 2016

Hello Hernán,

thank you for your pull request! It works for installing the package from a local wheel file,
but not when installing from PyPI.

I uploaded the package to PyPI using these steps:

~/repos/discoursegraphs $ rm dist/*
~/repos/discoursegraphs $ python setup.py sdist bdist_wheel
~/repos/discoursegraphs $ twine upload dist/*

Here are my installation results:

installing from PyPI without sudo

$ python --version
Python 2.7.6

$ pip --version
pip 8.1.1 from /usr/local/lib/python2.7/dist-packages (python 2.7)

$ python -c "import discoursegraphs"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named discoursegraphs

$ pip install discoursegraphs --user
Collecting discoursegraphs
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached discoursegraphs-0.3.1-py2-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): networkx in /usr/local/lib/python2.7/dist-packages (from discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): nltk in /usr/local/lib/python2.7/dist-packages (from discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): pydotplus in /usr/local/lib/python2.7/dist-packages/pydotplus-2.0.2-py2.7.egg (from discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): pygraphviz in /usr/local/lib/python2.7/dist-packages (from discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): unidecode in /usr/local/lib/python2.7/dist-packages (from discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): pydot2 in /usr/local/lib/python2.7/dist-packages/pydot2-1.0.33-py2.7.egg (from discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/local/lib/python2.7/dist-packages (from discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): brewer2mpl in /usr/local/lib/python2.7/dist-packages/brewer2mpl-1.4.1-py2.7.egg (from discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): decorator>=3.4.0 in /usr/local/lib/python2.7/dist-packages (from networkx->discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): pyparsing>=2.0.1 in /usr/local/lib/python2.7/dist-packages (from pydotplus->discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/dist-packages (from pydot2->discoursegraphs)
Installing collected packages: discoursegraphs
Successfully installed discoursegraphs

$ python -c "import discoursegraphs as dg; print dg.SRC_ROOT_DIR; print dg.DATA_ROOT_DIR; print len(dg.corpora.pcc)"
/home/arne/.local/lib/python2.7/site-packages/discoursegraphs
/home/data
0

installing from PyPI with sudo

$ python --version
Python 2.7.6

$ pip --version
pip 8.1.1 from /usr/local/lib/python2.7/dist-packages (python 2.7)

$ python -c "import discoursegraphs as dg; print dg.DATA_ROOT_DIR; print len(dg.corpora.pcc)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named discoursegraphs

$ sudo pip install discoursegraphs
The directory '/home/arne/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/arne/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting discoursegraphs
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading discoursegraphs-0.3.1-py2-none-any.whl (2.7MB)
    100% |████████████████████████████████| 2.7MB 258kB/s 
Requirement already satisfied (use --upgrade to upgrade): networkx in /usr/local/lib/python2.7/dist-packages (from discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): nltk in /usr/local/lib/python2.7/dist-packages (from discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): pydotplus in /usr/local/lib/python2.7/dist-packages/pydotplus-2.0.2-py2.7.egg (from discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): pygraphviz in /usr/local/lib/python2.7/dist-packages (from discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): unidecode in /usr/local/lib/python2.7/dist-packages (from discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): pydot2 in /usr/local/lib/python2.7/dist-packages/pydot2-1.0.33-py2.7.egg (from discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/local/lib/python2.7/dist-packages (from discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): brewer2mpl in /usr/local/lib/python2.7/dist-packages/brewer2mpl-1.4.1-py2.7.egg (from discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): decorator>=3.4.0 in /usr/local/lib/python2.7/dist-packages (from networkx->discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): pyparsing>=2.0.1 in /usr/local/lib/python2.7/dist-packages (from pydotplus->discoursegraphs)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/dist-packages (from pydot2->discoursegraphs)
Installing collected packages: discoursegraphs
Successfully installed discoursegraphs-0.3.1

$ python -c "import discoursegraphs as dg; print dg.SRC_ROOT_DIR; print dg.DATA_ROOT_DIR; print len(dg.corpora.pcc)"
/usr/local/lib/python2.7/dist-packages/discoursegraphs
/usr/local/lib/python2.7/data
0

installing from PyPI in a virtualenv

$ mkvirtualenv dgtest
New python executable in dgtest/bin/python
Installing setuptools, pip, wheel...done.

(dgtest)$ which python
/home/arne/.virtualenvs/dgtest/bin/python

(dgtest)$ python --version
Python 2.7.6

(dgtest)$ which pip
/home/arne/.virtualenvs/dgtest/bin/pip

(dgtest)$ pip --version
pip 7.1.0 from /home/arne/.virtualenvs/dgtest/local/lib/python2.7/site-packages (python 2.7)

(dgtest)$ python -c "import discoursegraphs"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named discoursegraphs

(dgtest)$ pip install discoursegraphs
/home/arne/.virtualenvs/dgtest/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
You are using pip version 7.1.0, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting discoursegraphs
/home/arne/.virtualenvs/dgtest/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached discoursegraphs-0.3.1-py2-none-any.whl
Collecting networkx (from discoursegraphs)
  Using cached networkx-1.11-py2.py3-none-any.whl
Collecting nltk (from discoursegraphs)
Collecting pydotplus (from discoursegraphs)
Collecting pygraphviz (from discoursegraphs)
Collecting unidecode (from discoursegraphs)
Collecting pydot2 (from discoursegraphs)
Collecting lxml (from discoursegraphs)
Collecting brewer2mpl (from discoursegraphs)
  Using cached brewer2mpl-1.4.1-py2.py3-none-any.whl
Collecting decorator>=3.4.0 (from networkx->discoursegraphs)
  Using cached decorator-4.0.9-py2.py3-none-any.whl
Collecting pyparsing>=2.0.1 (from pydotplus->discoursegraphs)
  Using cached pyparsing-2.1.1-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./.virtualenvs/dgtest/lib/python2.7/site-packages (from pydot2->discoursegraphs)
Installing collected packages: decorator, networkx, nltk, pyparsing, pydotplus, pygraphviz, unidecode, pydot2, lxml, brewer2mpl, discoursegraphs
Successfully installed brewer2mpl-1.4.1 decorator-4.0.9 discoursegraphs-0.3.1 lxml-3.6.0 networkx-1.11 nltk-3.2.1 pydot2-1.0.33 pydotplus-2.0.2 pygraphviz-1.3.1 pyparsing-2.1.1 unidecode-0.4.19

(dgtest)$ python -c "import discoursegraphs as dg; print dg.SRC_ROOT_DIR; print dg.DATA_ROOT_DIR; print len(dg.corpora.pcc)"
/home/arne/.virtualenvs/dgtest/local/lib/python2.7/site-packages/discoursegraphs
/home/arne/.virtualenvs/data
0

Best regards,
Arne

@arne-cl arne-cl closed this as completed in 37f57d4 May 7, 2016
@hernan-erasmo
Copy link
Contributor

Hi Arne, thank you for your comments.

Would you consider re-opening this issue? Since it seems the fix I provided didn't work :(
I'm going to keep working on that PR.

@arne-cl arne-cl reopened this May 9, 2016
@arne-cl
Copy link
Owner Author

arne-cl commented May 9, 2016

Hello Hernán,

sorry, I didn't realize that I automatically closed the issue. I just reopened it.
Thank you for working on this!

Best,
Arne

@hernan-erasmo
Copy link
Contributor

Hi Arne,

Sorry for taking so long to make the changes. I believe it's fixed now, but when you have time please check the latest commits I pushed to the feature branch and see if it works for you.

Thank you,

@hernan-erasmo
Copy link
Contributor

hernan-erasmo commented May 29, 2016 •
edited
Loading

Hi @arne-cl ,

I see you've merged #147

Did it work? Do you think this issue should be closed now, or is more work needed?

@arne-cl
Copy link
Owner Author

arne-cl commented May 30, 2016

Dear Hernan,

I will test the code manually in a virtualenv, upload it to pypi and let you know.

Kind regards,
Arne

@arne-cl
Copy link
Owner Author

arne-cl commented May 30, 2016

Dear Hernan,

thanks a lot for the effort! You seem to have solved the issue. I would have preferred a solution using data_files instead of package_data, but I guess there's not much we can do about it right now.

Best regards,
Arne

@arne-cl arne-cl closed this as completed May 30, 2016
@hernan-erasmo
Copy link
Contributor

I'm glad it worked. I tried to make as little modifications as I could to make the change less drastic, but it seems that they're moving away from data_files as you can see in this comment and this other one.

In any case, I'm happy this solution works for you.

Best regards,
Hernán.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug data help wanted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@arne-cl @hernan-erasmo