osx build with unittest #12

gyanesh-m · 2018-02-28T11:51:18Z

Trial to fix build for osx.
Fixes : #1901

conda-forge-linter · 2018-02-28T11:51:23Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

Selectors are suggested to take a <two spaces>#<one space>[<expression>] form. See lines [63]

conda-forge-linter · 2018-02-28T11:55:40Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

…feedstock into osx-build-test

menshikh-iv · 2018-02-28T12:36:58Z

@gyanesh-m hey, great, you reproduced a problem (with stuck of Doc2Vec test), now you need to investigate, what's exactly happens and why.

gyanesh-m · 2018-02-28T13:14:39Z

@menshikh-iv Yea sure , I will get started with it. Also is it advisable to edit .travis.yml file manually even though it is generated automatically ?

menshikh-iv · 2018-02-28T13:16:18Z

@gyanesh-m that's not really good, but for debugging - OK

gyanesh-m · 2018-02-28T18:24:04Z

@menshikh-iv Hey, its been 5 hours but job in travis-ci is not getting scheduled. Any ideas ?

menshikh-iv · 2018-02-28T18:26:08Z

@gyanesh-m sometimes Travis stuck (or have no free MacOSX machines), this is the standard situation.
I re-run your build, I hope that this will help.

menshikh-iv · 2018-02-28T18:27:19Z

.travis.yml

+      source run_conda_forge_build_setup
+
+script:
+  - travis_wait conda build ./recipe


this doesn't look like the solution, but you can try (in debugging propose).

souravsingh · 2018-02-28T19:52:32Z

@gyanesh-m conda-forge can have problems in requisitioning machines when many packages run through CI services. I suggest waiting for a while.

souravsingh · 2018-02-28T19:59:42Z

Travis Wait command fails. I would suggest looking for other solutions.

menshikh-iv · 2018-03-01T04:04:15Z

I propose

check "FAST_VERSION" before run tests (should be != -1)
run ONLY doc2vec tests with max verbosity (logging.DEBUG)

menshikh-iv · 2018-03-01T04:16:39Z

@souravsingh good news, I added SKIP_NETWORK_TESTS flag (for cases when CI have some problems with SSL or any network issues).

gyanesh-m · 2018-03-01T04:22:23Z

@menshikh-iv Ok, I will give that a try. I have one doubt, are you talking about the logging level mentioned here test_doc2vec.py. If so, then it is already set to DEBUG. Also, can we split the test file of doc2vec for parallel testing method so that we can know more precisely which part consumes more time ?

menshikh-iv · 2018-03-01T04:25:24Z

@gyanesh-m not quiet, when you run it in current manner - no debug logs, for more pricely testing - python -m unittest -v gensim.test.<test_file>.<class_name>.<method_name>

You can also run it as file python path/to/doc2vec/tests, in this case you'll see logs.

gyanesh-m · 2018-03-01T04:56:14Z

@menshikh-iv I have written the test file for fast version check, so should I add it here or in the main repo ?

menshikh-iv · 2018-03-01T04:57:26Z

@gyanesh-m you can check it in one line here (no need to add it as distinct test to main repo)

…le changes

gyanesh-m · 2018-03-01T13:46:50Z

@menshikh-iv In the travis build log I can't find the line which shows the test for fast version check mentioned in test_fast.py. Any idea why it didn't execute ?

menshikh-iv · 2018-03-01T14:20:58Z

@gyanesh-m maybe you should modify https://github.com/conda-forge/gensim-feedstock/blob/2f18b2fca0e96457e8b67e060de3c3448891bd12/recipe/forge_test.sh ?

menshikh-iv · 2018-03-01T14:22:12Z

@gyanesh-m or this one https://github.com/conda-forge/gensim-feedstock/pull/12/files#diff-e178b687b10a71a3348107ae3154e44cR63 (maybe this line is stuck and you didn't achieve your line)

gyanesh-m · 2018-03-01T15:16:06Z

@menshikh-iv Oh ya, got it. It needs to be added in forge_test.sh. Thanks.

gyanesh-m · 2018-04-19T06:39:26Z

@menshikh-iv Hi, earlier I tried to profile the test cases with pytest-profiling but I was facing some difficulty in its installation as I couldn't find it in any existing conda-forge channel. So I did the profiling using cprofile for word2vec test file and what I found was the method which deals with acquiring of thread locks in the threading library executes 30 times more slower in osx than in linux. This leads to the completion of test file in linux in around 55s but the same test takes around 26 minutes in osx. Also, the test file in osx makes around 1300 more calls to the threading library as compared to the linux version. The graphs and the corresponding stat files are available here.

menshikh-iv · 2018-04-19T06:57:48Z

Thanks for the investigation @gyanesh-m, looks like optimized version doesn't work on OSX (although compiled successfully) =/, any ideas?

gyanesh-m · 2018-04-19T17:18:21Z

@menshikh-iv I looked up and found that multi threaded program in mac osx can be slower than single threaded for the BLAS library provided by Apple as compared to the one available from MacPorts [Source] . I am not sure if this is true now since that post is very old but it can be tried as a solution.

menshikh-iv · 2018-04-20T04:00:50Z

@gyanesh-m one important tip - we build OSX wheels for PyPI for a long time and this works as expected, check https://github.com/MacPython/gensim-wheels/.

Also, please check how works https://github.com/conda-forge/scikit-learn-feedstock (and numpy/scipy), I think we missed something critical here.

conda-forge-linter · 2018-04-20T04:56:15Z

Hi! This is the friendly automated conda-forge-linting service.

I was trying to look for recipes to lint for you, but it appears we have a merge conflict.
Please try to merge or rebase with the base branch to resolve this conflict.

Please ping the 'conda-forge/core' team (using the @ notation in a comment) if you believe this is a bug.

conda-forge-linter · 2018-04-20T04:58:01Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

conda-forge-linter · 2018-04-22T05:33:44Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

Selectors are suggested to take a <two spaces>#<one space>[<expression>] form. See lines [29]

conda-forge-linter · 2018-04-22T05:38:51Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

menshikh-iv · 2018-08-01T08:22:15Z

@gyanesh-m any updates? What's a current status?

gyanesh-m · 2018-08-01T13:38:06Z

@menshikh-iv Hey, I haven't looked into this for quite a while and I am busy for sometime now, so I won't be able to help further in this.

menshikh-iv · 2018-08-01T16:29:24Z

@gyanesh-m ok, but can you describe please, what are you tried, what you found, this will be really useful for future investigation, thanks!

gyanesh-m · 2018-08-02T17:01:49Z

@menshikh-iv Ok, I will do it over the weekend.

menshikh-iv · 2019-02-01T02:35:04Z

outdated, closing.

osx build with unittest

e30e71f

minor fix

3e19aa0

gyanesh-m added 2 commits February 28, 2018 11:57

Add travis file

66872b2

Merge branch 'osx-build-test' of https://github.com/gyanesh-m/gensim-…

ea24365

…feedstock into osx-build-test

menshikh-iv mentioned this pull request Feb 28, 2018

Add anaconda-cloud badge & setup build for MacOSX piskvorky/gensim#1901

Open

2 tasks

Make unittest verbose and increase timeout

6d73acf

menshikh-iv suggested changes Feb 28, 2018

View reviewed changes

Add fast version check, make doc2vec tests verbose & revert travis fi…

7c58269

…le changes

gyanesh-m added 3 commits March 1, 2018 20:47

Add tests in forge_test.sh

a86f216

fix test_fast check

17f2041

Fetch test_fast.py and then try

4648966

gyanesh-m force-pushed the osx-build-test branch from d42529a to 8c382de Compare March 31, 2018 19:48

Change profiler to cprofile and retry

62156e2

gyanesh-m closed this Apr 18, 2018

gyanesh-m reopened this Apr 18, 2018

gyanesh-m force-pushed the osx-build-test branch from e0514e8 to 62156e2 Compare April 20, 2018 04:57

gyanesh-m added 3 commits April 20, 2018 10:33

Add blas and openblas requirement

1e72ae1

Remove variant

55ce812

Set number of threads to 1 and retry

b1d757a

gyanesh-m closed this Apr 20, 2018

gyanesh-m reopened this Apr 20, 2018

Add path information

b7de836

minor fix

cdbac2d

gyanesh-m added 3 commits April 22, 2018 13:27

Add flags info

4e1b59a

Remove thread restriction

b9c9b7c

Add path info

cf68967

menshikh-iv mentioned this pull request Sep 22, 2018

[DNM] gensim v3.6.0 #14

Closed

menshikh-iv closed this Feb 1, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

osx build with unittest #12

osx build with unittest #12

gyanesh-m commented Feb 28, 2018

conda-forge-linter commented Feb 28, 2018

conda-forge-linter commented Feb 28, 2018

menshikh-iv commented Feb 28, 2018 •

edited

Loading

gyanesh-m commented Feb 28, 2018

menshikh-iv commented Feb 28, 2018

gyanesh-m commented Feb 28, 2018

menshikh-iv commented Feb 28, 2018

menshikh-iv Feb 28, 2018

souravsingh commented Feb 28, 2018

souravsingh commented Feb 28, 2018

menshikh-iv commented Mar 1, 2018

menshikh-iv commented Mar 1, 2018

gyanesh-m commented Mar 1, 2018

menshikh-iv commented Mar 1, 2018 •

edited

Loading

gyanesh-m commented Mar 1, 2018

menshikh-iv commented Mar 1, 2018 •

edited

Loading

gyanesh-m commented Mar 1, 2018

menshikh-iv commented Mar 1, 2018

menshikh-iv commented Mar 1, 2018

gyanesh-m commented Mar 1, 2018

gyanesh-m commented Apr 19, 2018

menshikh-iv commented Apr 19, 2018 •

edited

Loading

gyanesh-m commented Apr 19, 2018 •

edited

Loading

menshikh-iv commented Apr 20, 2018

conda-forge-linter commented Apr 20, 2018

conda-forge-linter commented Apr 20, 2018

conda-forge-linter commented Apr 22, 2018

conda-forge-linter commented Apr 22, 2018

menshikh-iv commented Aug 1, 2018

gyanesh-m commented Aug 1, 2018

menshikh-iv commented Aug 1, 2018

gyanesh-m commented Aug 2, 2018

menshikh-iv commented Feb 1, 2019

osx build with unittest #12

osx build with unittest #12

Conversation

gyanesh-m commented Feb 28, 2018

conda-forge-linter commented Feb 28, 2018

conda-forge-linter commented Feb 28, 2018

menshikh-iv commented Feb 28, 2018 • edited Loading

gyanesh-m commented Feb 28, 2018

menshikh-iv commented Feb 28, 2018

gyanesh-m commented Feb 28, 2018

menshikh-iv commented Feb 28, 2018

menshikh-iv Feb 28, 2018

Choose a reason for hiding this comment

souravsingh commented Feb 28, 2018

souravsingh commented Feb 28, 2018

menshikh-iv commented Mar 1, 2018

menshikh-iv commented Mar 1, 2018

gyanesh-m commented Mar 1, 2018

menshikh-iv commented Mar 1, 2018 • edited Loading

gyanesh-m commented Mar 1, 2018

menshikh-iv commented Mar 1, 2018 • edited Loading

gyanesh-m commented Mar 1, 2018

menshikh-iv commented Mar 1, 2018

menshikh-iv commented Mar 1, 2018

gyanesh-m commented Mar 1, 2018

gyanesh-m commented Apr 19, 2018

menshikh-iv commented Apr 19, 2018 • edited Loading

gyanesh-m commented Apr 19, 2018 • edited Loading

menshikh-iv commented Apr 20, 2018

conda-forge-linter commented Apr 20, 2018

conda-forge-linter commented Apr 20, 2018

conda-forge-linter commented Apr 22, 2018

conda-forge-linter commented Apr 22, 2018

menshikh-iv commented Aug 1, 2018

gyanesh-m commented Aug 1, 2018

menshikh-iv commented Aug 1, 2018

gyanesh-m commented Aug 2, 2018

menshikh-iv commented Feb 1, 2019

menshikh-iv commented Feb 28, 2018 •

edited

Loading

menshikh-iv commented Mar 1, 2018 •

edited

Loading

menshikh-iv commented Mar 1, 2018 •

edited

Loading

menshikh-iv commented Apr 19, 2018 •

edited

Loading

gyanesh-m commented Apr 19, 2018 •

edited

Loading