This repository has been archived by the owner on Sep 11, 2020. It is now read-only.
Add requirements.txt #27
Comments
nateraw
added
help wanted
|
Would really like some help with this...I don't have any experience setting up the requirements.txt file for a package. Here are the versions of the packages I'm using to run this: Also - Note that by default, the tokenizer uses spacy's "en_core_web_sm" language model. It would be nice if on install from pip, it automatically would install spacy as well as the "en_core_web_sm" model or prompted the user somehow to do so. |
|
@dbl001 in my dev branch I did everything under a new clean environment. I think this is exactly what I needed to do in order to set it up (if you have conda). If you don't have conda, you'll have to translate out parts of this to pip. If you get the chance to checkout the commit I tagged you in, maybe try to set up this environment using these commands and the updated requirements.txt file under the dev branch. ### Create new conda environemnt
conda create --name lda2vec_test python=3.5
### Activate new environment
source activate lda2vec_test
### Add conda forge so you can install spacy
conda config --add channels conda-forge
### Install my spacy version
conda install spacy=2.0.11
### Install spacy language model
sudo pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-2.0.0/en_core_web_lg-2.0.0.tar.gz --trusted-host github.com
### Install the rest of the requirements
sudo pip install -r requirements.txt |
|
Note - Windows users would say |
|
@dbl001 were you able to set up the environment successfully and run it? If not, I may probe a little more to assure it works before updating readme with instal directions. Also, if this all works out - I may take the package off of pypi. Doesn't seem relevant to pip install this, as users probably want to play around with parameters. What do you think about that? |
|
On mac I had to take This wasn't an issue on Linux. Not sure how to mitigate this issue without Dockerizing the repo. |
|
I ran the script on my Mac. Not sure why iPython started as python 2.76 |
|
Doesn't look like you ever actually ran requirements.txt for some reason 😅 . Also - my guess is that on mac, installing requirements.txt will break for you when you go to install it. You'll have to do what I said in the earlier comment about removing tensorflow from requirements.txt and running that other command manually. It's only Mac that seems to have that issue...strange. Works fine on my Linux box. |
|
conda and pip don’t always ‘play nicely together’.
After I ran:
sudo pip install -r requirements.txt
load_20newsgroups.py got exceptions trying to import:
pandas, scikit-learn, tears, and pyLDAvis
pip installed pandas==0.21.1
E.g.
…
Collecting pandas==0.21.1 (from -r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/54/3e/816df3ff52b805038743c8e15a48e67524ecad9f9b597e2d10c61073cc7a/pandas-0.21.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (14.9MB)
100% |████████████████████████████████| 14.9MB 1.2MB/s
…
$ python load_20newsgroups.py
Traceback (most recent call last):
File "load_20newsgroups.py", line 1, in <module>
import pandas as pd
ImportError: No module named 'pandas'
I had to install it with conda:
$ conda install pandas
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.5.13
latest version: 4.6.11
Please update conda by running
$ conda update -n base -c defaults conda
## Package Plan ##
environment location: /Users/davidlaxer/anaconda/envs/lda2vec_test
added / updated specs:
- pandas
The following packages will be downloaded:
package | build
---------------------------|-----------------
python-dateutil-2.8.0 | py_0 219 KB conda-forge
pandas-0.23.4 | py35hf8a1672_0 10.1 MB conda-forge
pytz-2018.9 | py_0 229 KB conda-forge
------------------------------------------------------------
Total: 10.6 MB
The following NEW packages will be INSTALLED:
pandas: 0.23.4-py35hf8a1672_0 conda-forge
python-dateutil: 2.8.0-py_0 conda-forge
pytz: 2018.9-py_0 conda-forge
Proceed ([y]/n)? y
Downloading and Extracting Packages
python-dateutil-2.8. | 219 KB | ##################################### | 100%
pandas-0.23.4 | 10.1 MB | ##################################### | 100%
pytz-2018.9 | 229 KB | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Same issue with scikit-learn, keras and ldaPYvis.
Anyway, it’s running now …
$ python load_20newsgroups.py
/Users/davidlaxer/anaconda/envs/lda2vec_test/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: compiletime version 3.6 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.5
return f(*args, **kwds)
Using TensorFlow backend.
…---------- Tokenizing Texts ----------
879it [01:00, 15.31s/it]
On Apr 4, 2019, at 9:56 PM, Nathan Raw ***@***.***> wrote:
Doesn't look like you ever actually ran requirements.txt for some reason 😅 . Also - my guess is that on mac, installing requirements.txt will break for you when you go to install it. You'll have to do what I said in the earlier comment about removing tensorflow from requirements.txt and running that other command manually.
It's only Mac that seems to have that issue...strange. Works fine on my Linux box.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#27 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC9i26wOLrh-xxtv44xzvpwhnmgFBgg8ks5vdteMgaJpZM4bgObT>.
|
|
FileNotFoundError: [Errno 2] No such file or directory: '/media/dlmachine/SSD_2/embeddings/glove.6B.300d.txt'
… On Apr 4, 2019, at 11:12 PM, David Laxer ***@***.***> wrote:
conda and pip don’t always ‘play nicely together’.
After I ran:
sudo pip install -r requirements.txt
load_20newsgroups.py got exceptions trying to import:
pandas, scikit-learn, tears, and pyLDAvis
pip installed pandas==0.21.1
E.g.
…
Collecting pandas==0.21.1 (from -r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/54/3e/816df3ff52b805038743c8e15a48e67524ecad9f9b597e2d10c61073cc7a/pandas-0.21.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl <https://files.pythonhosted.org/packages/54/3e/816df3ff52b805038743c8e15a48e67524ecad9f9b597e2d10c61073cc7a/pandas-0.21.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl> (14.9MB)
100% |████████████████████████████████| 14.9MB 1.2MB/s
…
$ python load_20newsgroups.py
Traceback (most recent call last):
File "load_20newsgroups.py", line 1, in <module>
import pandas as pd
ImportError: No module named 'pandas'
I had to install it with conda:
$ conda install pandas
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.5.13
latest version: 4.6.11
Please update conda by running
$ conda update -n base -c defaults conda
## Package Plan ##
environment location: /Users/davidlaxer/anaconda/envs/lda2vec_test
added / updated specs:
- pandas
The following packages will be downloaded:
package | build
---------------------------|-----------------
python-dateutil-2.8.0 | py_0 219 KB conda-forge
pandas-0.23.4 | py35hf8a1672_0 10.1 MB conda-forge
pytz-2018.9 | py_0 229 KB conda-forge
------------------------------------------------------------
Total: 10.6 MB
The following NEW packages will be INSTALLED:
pandas: 0.23.4-py35hf8a1672_0 conda-forge
python-dateutil: 2.8.0-py_0 conda-forge
pytz: 2018.9-py_0 conda-forge
Proceed ([y]/n)? y
Downloading and Extracting Packages
python-dateutil-2.8. | 219 KB | ##################################### | 100%
pandas-0.23.4 | 10.1 MB | ##################################### | 100%
pytz-2018.9 | 229 KB | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Same issue with scikit-learn, keras and ldaPYvis.
Anyway, it’s running now …
$ python load_20newsgroups.py
/Users/davidlaxer/anaconda/envs/lda2vec_test/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: compiletime version 3.6 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.5
return f(*args, **kwds)
Using TensorFlow backend.
---------- Tokenizing Texts ----------
879it [01:00, 15.31s/it]
> On Apr 4, 2019, at 9:56 PM, Nathan Raw ***@***.*** ***@***.***>> wrote:
>
> Doesn't look like you ever actually ran requirements.txt for some reason 😅 . Also - my guess is that on mac, installing requirements.txt will break for you when you go to install it. You'll have to do what I said in the earlier comment about removing tensorflow from requirements.txt and running that other command manually.
>
> It's only Mac that seems to have that issue...strange. Works fine on my Linux box.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <#27 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC9i26wOLrh-xxtv44xzvpwhnmgFBgg8ks5vdteMgaJpZM4bgObT>.
>
|
|
This worked: |
|
Warnings:
/Users/davidlaxer/Lda2vec-Tensorflow/lda2vec/nlppipe.py:129: FutureWarning: arrays to stack must be passed as a "sequence" type such as list or tuple. Support for non-sequence iterables such as generators is deprecated as of NumPy 1.16 and will raise an error in the future.
all_embs = np.stack(embeddings_index.values())
/Users/davidlaxer/anaconda/envs/lda2vec_test/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: compiletime version 3.6 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.5
return f(*args, **kwds)
… On Apr 5, 2019, at 2:45 AM, David Laxer ***@***.***> wrote:
FileNotFoundError: [Errno 2] No such file or directory: '/media/dlmachine/SSD_2/embeddings/glove.6B.300d.txt'
> On Apr 4, 2019, at 11:12 PM, David Laxer ***@***.*** ***@***.***>> wrote:
>
> conda and pip don’t always ‘play nicely together’.
>
> After I ran:
> sudo pip install -r requirements.txt
>
> load_20newsgroups.py got exceptions trying to import:
>
> pandas, scikit-learn, tears, and pyLDAvis
>
>
> pip installed pandas==0.21.1
> E.g.
> …
>
> Collecting pandas==0.21.1 (from -r requirements.txt (line 2))
> Downloading https://files.pythonhosted.org/packages/54/3e/816df3ff52b805038743c8e15a48e67524ecad9f9b597e2d10c61073cc7a/pandas-0.21.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl <https://files.pythonhosted.org/packages/54/3e/816df3ff52b805038743c8e15a48e67524ecad9f9b597e2d10c61073cc7a/pandas-0.21.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl> (14.9MB)
> 100% |████████████████████████████████| 14.9MB 1.2MB/s
> …
> $ python load_20newsgroups.py
> Traceback (most recent call last):
> File "load_20newsgroups.py", line 1, in <module>
> import pandas as pd
> ImportError: No module named 'pandas'
>
> I had to install it with conda:
>
> $ conda install pandas
> Solving environment: done
>
>
> ==> WARNING: A newer version of conda exists. <==
> current version: 4.5.13
> latest version: 4.6.11
>
> Please update conda by running
>
> $ conda update -n base -c defaults conda
>
>
>
> ## Package Plan ##
>
> environment location: /Users/davidlaxer/anaconda/envs/lda2vec_test
>
> added / updated specs:
> - pandas
>
>
> The following packages will be downloaded:
>
> package | build
> ---------------------------|-----------------
> python-dateutil-2.8.0 | py_0 219 KB conda-forge
> pandas-0.23.4 | py35hf8a1672_0 10.1 MB conda-forge
> pytz-2018.9 | py_0 229 KB conda-forge
> ------------------------------------------------------------
> Total: 10.6 MB
>
> The following NEW packages will be INSTALLED:
>
> pandas: 0.23.4-py35hf8a1672_0 conda-forge
> python-dateutil: 2.8.0-py_0 conda-forge
> pytz: 2018.9-py_0 conda-forge
>
> Proceed ([y]/n)? y
>
>
> Downloading and Extracting Packages
> python-dateutil-2.8. | 219 KB | ##################################### | 100%
> pandas-0.23.4 | 10.1 MB | ##################################### | 100%
> pytz-2018.9 | 229 KB | ##################################### | 100%
> Preparing transaction: done
> Verifying transaction: done
> Executing transaction: done
>
>
> Same issue with scikit-learn, keras and ldaPYvis.
> Anyway, it’s running now …
>
> $ python load_20newsgroups.py
> /Users/davidlaxer/anaconda/envs/lda2vec_test/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: compiletime version 3.6 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.5
> return f(*args, **kwds)
> Using TensorFlow backend.
>
> ---------- Tokenizing Texts ----------
> 879it [01:00, 15.31s/it]
>
>> On Apr 4, 2019, at 9:56 PM, Nathan Raw ***@***.*** ***@***.***>> wrote:
>>
>> Doesn't look like you ever actually ran requirements.txt for some reason 😅 . Also - my guess is that on mac, installing requirements.txt will break for you when you go to install it. You'll have to do what I said in the earlier comment about removing tensorflow from requirements.txt and running that other command manually.
>>
>> It's only Mac that seems to have that issue...strange. Works fine on my Linux box.
>>
>> —
>> You are receiving this because you were mentioned.
>> Reply to this email directly, view it on GitHub <#27 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC9i26wOLrh-xxtv44xzvpwhnmgFBgg8ks5vdteMgaJpZM4bgObT>.
>>
>
|
|
Whoops - thought I sent a reply to you earlier but must not have hit send. The embeddings path was left in there by accident. That's where I keep my embeddings on my computer so I don't have to keep copying them to every repo I use them in. I'll make sure to push a quick switch for that. As for the mac vs windows install stuff, I know the issues on both, I just don't know the best way to solve the problem. Looking into some solutions. Main solution would probably be to just use a newer version of TF instead of the outdated 1.5.0. Problem is my hardware restricts me to 1.5.0 due to my CPU not being compatible with AVX. Will think on this a bit more... |
|
My CPU on my Mac doesn’t support AVX either.
I’ve running Tensorflow 1.12 and I think also 1.18.
I think I installed it from Conda. Stay tuned.
… On Apr 5, 2019, at 3:21 PM, Nathan Raw ***@***.***> wrote:
Whoops - thought I sent a reply to you earlier but must not have hit send. The embeddings path was left in there by accident. That's where I keep my embeddings on my computer so I don't have to keep copying them to every repo I use them in. I'll make sure to push a quick switch for that.
As for the mac vs windows install stuff, I know the issues on both, I just don't know the best way to solve the problem. Looking into some solutions. Main solution would probably be to just use a newer version of TF instead of the outdated 1.5.0. Problem is my hardware restricts me to 1.5.0 due to my CPU not being compatible with AVX. Will think on this a bit more...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
Interesting...did you have to pass flags to make it work?? Maybe the AVX stuff is built into that mac specific installer. When v1.6 got released I looked into how to install it but couldn't get it to work, so I haven't tried since. CPU only is obviously not an option. |
|
I may have built Tensorflow from source on my Mac.
What are the 2-3 character tokens in the word to topic output that are not words and do appear in the original document file?
… On Apr 5, 2019, at 3:54 PM, Nathan Raw ***@***.***> wrote:
Interesting...did you have to pass flags to make it work?? Maybe the AVX stuff is built into that mac specific installer. When v1.6 got released I looked into how to install it but couldn't get it to work, so I haven't tried since. CPU only is obviously not an option.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
No idea. Haven't had time to look. Will be able to tonight/tomorrow. |
|
This works for me and should solve all of our problems. The following are instructions for building a Docker environment on Ubuntu 16.04. If you're on a different OS, you'll have to navigate to the instructions for your relevant OS (from the sidebars in the links provided) First, install docker CE. I'm on Ubuntu, so I followed these instructions Next, if you're on Linux and want to use GPU, you have to install nvidia-docker by following the instructions in the readme. Once you're finished, you can build/run the tensorflow 1.5 GPU Docker Image from tensorflow/tensorflow After you have this working, we should be able to standardize the install across platforms. |
This was referenced Apr 15, 2019
|
pip install -r requirements.txt |
|
Is this happening in Docker?
… On Jul 3, 2019, at 12:01 AM, SUDHEER NAIDU ***@***.***> wrote:
pip install -r requirements.txt
ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#27?email_source=notifications&email_token=AAXWFW2AKIXKS4FA42E3X43P5RFD5A5CNFSM4G4A43J2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZDPPGI#issuecomment-507967385>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAXWFWZ4JY4QFY7IDEFN3JTP5RFD5ANCNFSM4G4A43JQ>.
|
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
No description provided.
The text was updated successfully, but these errors were encountered: