C matrices #158

JoostJM · 2016-11-16T02:46:46Z

Implement C for calculation of matrices
Currently supports:

GLCM
GLSZM

This greatly improves processing speed, while still keeping an all-python option available (currently enabled by setting radiomics.debugging to True.

Also included automatic linking and compiling to setup script. Needs additional package (cython) to work, but is now also included in setup script (if not installed, setup script installs it without error).

Lastly, added a new test, where the python generated matrices are compared to the C generated matrices for all test cases. No features are calculated here, but matrices are assessed element wise (maximum difference for any element is 1e-3, this is due to small machine precision errors possible in NGTDM).

JoostJM · 2016-11-16T03:04:22Z

Snapshot from profiling, C functions are highlighted:

JoostJM · 2016-11-22T03:26:49Z

Finished implementing time intensive features in C. Calculation time at profile now just over 6 seconds for 5 testcases. Still writing some docstrings.
Calculations now in C:

GLCM
GLDM (EXPERIMENTAL, REMOVED)
NGTDM (EXPERIMENTAL, REMOVED)
GLSZM
GLRLM
SurfaceArea

When GLDZM (EXPERIMENTAL, REMOVED) is merged into the master, I can also write a C function for this. This is very simple, as it would be identical to the one calculating the GLSZM, but instead of counting the processed voxels, it records the minimum distance in the zone.

fedorov · 2016-11-23T04:36:25Z

When I do python setup.py install on mac with this branch, I am getting the error below. Is there anything special/extra that I need to do?

@JoostJM did you set up a paid account on TravisCI to proceed with CI testing on Mac? @naucoin could help setting it up (see #126) once we have a paid account (or open source version of pyradiomics).

$ python setup.py install
/var/folders/yk/3nc16c9d241_ds1vfkdh1km80000gn/T/easy_install-64trmx/numpy-1.12.0b1/setup.py:366: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
  run_build = parse_setuppy_commands()
Traceback (most recent call last):
  File "setup.py", line 28, in <module>
    zip_safe=False)
  File "/Users/fedorov/anaconda/lib/python2.7/distutils/core.py", line 111, in setup
    _setup_distribution = dist = klass(attrs)
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/setuptools/dist.py", line 269, in __init__
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/setuptools/dist.py", line 313, in fetch_build_eggs
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/pkg_resources/__init__.py", line 827, in resolve
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/pkg_resources/__init__.py", line 1072, in best_match
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/pkg_resources/__init__.py", line 1084, in obtain
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/setuptools/dist.py", line 380, in fetch_build_egg
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/setuptools/command/easy_install.py", line 640, in easy_install
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/setuptools/command/easy_install.py", line 670, in install_item
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/setuptools/command/easy_install.py", line 853, in install_eggs
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/setuptools/command/easy_install.py", line 1081, in build_and_install
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/setuptools/command/easy_install.py", line 1067, in run_setup
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/setuptools/sandbox.py", line 246, in run_setup
  File "/Users/fedorov/anaconda/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/setuptools/sandbox.py", line 195, in setup_context
  File "/Users/fedorov/anaconda/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/setuptools/sandbox.py", line 166, in save_modules
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/setuptools/sandbox.py", line 141, in resume
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/setuptools/sandbox.py", line 154, in save_modules
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/setuptools/sandbox.py", line 195, in setup_context
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/setuptools/sandbox.py", line 243, in run_setup
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/setuptools/sandbox.py", line 273, in run
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/setuptools/sandbox.py", line 242, in runner
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg/setuptools/sandbox.py", line 46, in _execfile
  File "/var/folders/yk/3nc16c9d241_ds1vfkdh1km80000gn/T/easy_install-64trmx/numpy-1.12.0b1/setup.py", line 391, in <module>

  File "/var/folders/yk/3nc16c9d241_ds1vfkdh1km80000gn/T/easy_install-64trmx/numpy-1.12.0b1/setup.py", line 383, in setup_package

  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/numpy/distutils/core.py", line 135, in setup
    config = configuration()
  File "/var/folders/yk/3nc16c9d241_ds1vfkdh1km80000gn/T/easy_install-64trmx/numpy-1.12.0b1/setup.py", line 166, in configuration

  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/numpy/distutils/misc_util.py", line 1002, in add_subpackage
    caller_level = 2)
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/numpy/distutils/misc_util.py", line 971, in get_subpackage
    caller_level = caller_level + 1)
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/numpy/distutils/misc_util.py", line 908, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "numpy/setup.py", line 10, in configuration
    packages=['radiomics'],
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/numpy/distutils/misc_util.py", line 1002, in add_subpackage
    caller_level = 2)
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/numpy/distutils/misc_util.py", line 971, in get_subpackage
    caller_level = caller_level + 1)
  File "/Users/fedorov/anaconda/lib/python2.7/site-packages/numpy/distutils/misc_util.py", line 883, in _get_configuration_from_setup_py
    ('.py', 'U', 1))
  File "/private/var/folders/yk/3nc16c9d241_ds1vfkdh1km80000gn/T/easy_install-64trmx/numpy-1.12.0b1/numpy/core/setup.py", line 12, in <module>
    install_requires=['cython',
ImportError: No module named _build_utils.apple_accelerate

JoostJM · 2016-11-23T18:17:01Z

@jcfr, Can you resubmit your changes? I'm afraid I accidentally deleted them when I rebased this branch on the master, to incorporate the changes from the other PR's we just merged.

JoostJM · 2016-11-23T18:18:43Z

@fedorov as to your error, did you have numpy and cython installed prior to running this script?

jcfr · 2016-11-23T18:20:46Z

Sure. Will do after lunch.

On Nov 23, 2016 1:17 PM, "Joost van Griethuysen" [email protected]
wrote:

@jcfr https://github.com/jcfr, Can you resubmit your changes? I'm
afraid I accidentally deleted them when I rebased this branch on the
master, to incorporate the changes from the other PR's we just merged.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#158 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AANXo3BRVwPw9amLYyGaeBd4Ryb94NWFks5rBIMegaJpZM4KzSOX
.

jcfr · 2016-11-23T21:18:41Z

@JoostJM @pieper @fedorov Here it is. Let me know what you think.

As a side note, @isuruf is helping to add scikit-build into conda forge. See conda-forge/staged-recipes#1989

JoostJM · 2016-11-23T21:25:57Z

@jcfr My build is failing as I do not have CMake currently installed on my system. So this means your update needs the user to have CMake installed, as well as the extra dependencies. If we're okay with, it's fine with me.

fedorov · 2016-11-23T21:27:33Z

I am not worried about CMake as a dependency. I am worried about maintaining CMakeLists.txt files. But this might well be a necessary price to pay if you decide to use C.

jcfr · 2016-11-23T21:27:37Z

@JoostJM The following

pip install scikit-build cmake

should do it.

There is also more details in the commit message.

JoostJM · 2016-11-23T21:28:56Z

@jcfr No, I already did that, I ran your line again and it just says requirements satisfied

JoostJM · 2016-11-23T21:29:50Z

fedorov · 2016-11-23T21:31:48Z

I recall from somewhere that using MinGW is a bad idea ... welcome to the C world!

jcfr · 2016-11-23T21:31:52Z

@JoostJM

Do you have Visual Studio 2008 installed ? This is the official compiler for python 2.7

I have to head out and will look at it later. And also improve the documentation of scikit-build for alternative and how to use them. Example: https://www.microsoft.com/en-us/download/details.aspx?id=44266

((The issue with building from the Git shell is that there is a mingw compiler also installed ...))

JoostJM · 2016-11-23T21:38:04Z

I have Visual Studio 2015, but my environmentvariables are set so that VS90COMNTOOLS points to this newer compiler, so without using cmake, my python usually just finds my compiler

jcfr · 2016-11-23T21:57:01Z

python setup.py build -G "Visual Studio 14 2015 Win64"

or

python setup.py bdist -G "Visual Studio 14 2015 Win64"

or

python setup.py bdist_wheel -G "Visual Studio 14 2015 Win64"

More details here: http://scikit-build.readthedocs.io/en/latest/usage.html#command-line-options

I will look into python setup.py install -G "Visual Studio 14 2015 Win64"

Package is now built into `_skbuild`, update gitignore to ignore this folder.

Fix potential index out of range error in `fill_glszm` in tempData. This occurs when the entire tempData matrix is filled (Ns different regions. When the function fill_glszm then runs, it does not exit the while loop (as it encounters no -1 at the end of the matrix). Specify the size of the tempData matrix = (Ns + 1) * 2, so that the last element will always be -1 (max idx computed by function calculate_glszm is still Ns * 2). Additionally re-apply code style to C files

…Utils

See https://www.python.org/dev/peps/pep-0008/#imports

This approach allows to support use of radiomics package after installing pyradiomics as a regular binary distribution

…port

JoostJM · 2016-12-21T22:21:10Z

Rebased this branch onto master after merging #173. As there is code changed in the removed classes by this branch, I stored a copy of the not-rebased branch here.

Applied the update where experimental/unstable classes are removed from the C-implementation and tests.

jcfr · 2016-12-22T00:14:41Z

scikit-build has been released. Update of this topic will follow.

Remove C code for calculating the matrix in these classes. Remove the tests for these classes from test_cmatrices

Commit 7da5b76 caused the dynamic lookup of feature classes in `featureextractor` to break. Fix the function and move this lookup functionality to `__init__.py`, as it is used more extensively throughout the package.

This should be re-enabled after integrating #158

JoostJM · 2017-01-19T12:22:47Z

@jcfr I updated my scikit-build and cmake on my computer at work, but I am still getting errors. It's an unresolved external reference error in the load-function of numpy (only fails when compiling with cmake though). Here is a section the setup output, do you know whats going wrong?

This should be re-enabled after integrating #158

jcfr · 2017-02-14T16:56:58Z

@JoostJM Note that I started to rebase and re-organized commits. I will follow up with some update.

jcfr

This PR will be re-organized and rebased on top of the #196 set of changes

jcfr · 2017-02-15T19:01:01Z

@JoostJM Here is what I have so far: https://github.com/Radiomics/pyradiomics/tree/c-matrices-rebased-jcfr

I squashed together changes that made sense. Still need some work.

Would be nice to integrate separately all changes that are not specific to C implementation.

For example: f6b77ed

jcfr · 2017-02-15T22:59:42Z

Closing. This is superseded by #200

JoostJM added the enhancement label Nov 16, 2016

JoostJM force-pushed the c-matrices branch 3 times, most recently from 33628e6 to f06982c Compare November 16, 2016 02:52

JoostJM mentioned this pull request Nov 16, 2016

Computation time performance profiling of feature calculation #106

Closed

JoostJM force-pushed the c-matrices branch 6 times, most recently from cec483b to 005d4aa Compare November 22, 2016 03:07

JoostJM force-pushed the c-matrices branch from 244da2d to 33c52cb Compare November 22, 2016 15:31

jcfr force-pushed the c-matrices branch from 268d14d to eca3f2e Compare November 23, 2016 16:50

JoostJM force-pushed the c-matrices branch from eca3f2e to e7cecc9 Compare November 23, 2016 17:33

JoostJM and others added 9 commits December 21, 2016 23:02

STYL: Update .gitignore

2bd874f

Package is now built into `_skbuild`, update gitignore to ignore this folder.

ENH: Add C implementation for GLDZM calculation

d94dd00

ENH: Add C support to GLDZM featureclass

5cdaf7b

STYL: Convert "tests" into a package to allow relative import of test…

a8830c7

…Utils

STYL: tests: Fix order of package imports and remove unused ones

950094b

See https://www.python.org/dev/peps/pep-0008/#imports

BUG: Import "cMatrices" and "cShape" C extensions from radiomics package

68578be

This approach allows to support use of radiomics package after installing pyradiomics as a regular binary distribution

STYL: radiomics: More package imports order fixes, use of relative im…

7da5b76

…port

ENH: Add support for "python setup.py test"

f5a5e6d

JoostJM force-pushed the c-matrices branch from 5e4d0b9 to f5a5e6d Compare December 21, 2016 22:10

DEL: Remove experimental/unstable feature classes

df7c8d8

Remove C code for calculating the matrix in these classes. Remove the tests for these classes from test_cmatrices

JoostJM force-pushed the c-matrices branch from 822b6aa to df7c8d8 Compare December 22, 2016 08:47

BUG: Fix getFeatureClasses

0fc72a0

Commit 7da5b76 caused the dynamic lookup of feature classes in `featureextractor` to break. Fix the function and move this lookup functionality to `__init__.py`, as it is used more extensively throughout the package.

JoostJM force-pushed the c-matrices branch 2 times, most recently from 0fc72a0 to 81638df Compare December 23, 2016 09:19

jcfr added a commit that referenced this pull request Jan 18, 2017

ci: Disable auditwheel on circleci.

41caaeb

This should be re-enabled after integrating #158

JoostJM mentioned this pull request Feb 13, 2017

making it python 3 compatible #188

Closed

jcfr added a commit that referenced this pull request Feb 13, 2017

ci: Disable auditwheel on circleci.

15c182e

This should be re-enabled after integrating #158

jcfr mentioned this pull request Feb 14, 2017

Improve setup py and simplify ci #191

Merged

jcfr requested changes Feb 14, 2017

View reviewed changes

jcfr mentioned this pull request Feb 15, 2017

Support for faster C-based matrix and shape surface computation #200

Closed

2 tasks

jcfr closed this Feb 15, 2017

jcfr mentioned this pull request Feb 16, 2017

C matrices revised jcfr 20170216 #202

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

C matrices #158

C matrices #158

JoostJM commented Nov 16, 2016 •

edited

Loading

JoostJM commented Nov 16, 2016

JoostJM commented Nov 22, 2016 •

edited

Loading

fedorov commented Nov 23, 2016 •

edited

Loading

JoostJM commented Nov 23, 2016

JoostJM commented Nov 23, 2016

jcfr commented Nov 23, 2016

jcfr commented Nov 23, 2016 •

edited

Loading

JoostJM commented Nov 23, 2016

fedorov commented Nov 23, 2016

jcfr commented Nov 23, 2016

JoostJM commented Nov 23, 2016

JoostJM commented Nov 23, 2016

fedorov commented Nov 23, 2016

jcfr commented Nov 23, 2016 •

edited

Loading

JoostJM commented Nov 23, 2016 •

edited

Loading

jcfr commented Nov 23, 2016

JoostJM commented Dec 21, 2016

jcfr commented Dec 22, 2016

JoostJM commented Jan 19, 2017

jcfr commented Feb 14, 2017

jcfr left a comment

jcfr commented Feb 15, 2017

jcfr commented Feb 15, 2017

C matrices #158

C matrices #158

Conversation

JoostJM commented Nov 16, 2016 • edited Loading

JoostJM commented Nov 16, 2016

JoostJM commented Nov 22, 2016 • edited Loading

fedorov commented Nov 23, 2016 • edited Loading

JoostJM commented Nov 23, 2016

JoostJM commented Nov 23, 2016

jcfr commented Nov 23, 2016

jcfr commented Nov 23, 2016 • edited Loading

JoostJM commented Nov 23, 2016

fedorov commented Nov 23, 2016

jcfr commented Nov 23, 2016

JoostJM commented Nov 23, 2016

JoostJM commented Nov 23, 2016

fedorov commented Nov 23, 2016

jcfr commented Nov 23, 2016 • edited Loading

JoostJM commented Nov 23, 2016 • edited Loading

jcfr commented Nov 23, 2016

JoostJM commented Dec 21, 2016

jcfr commented Dec 22, 2016

JoostJM commented Jan 19, 2017

jcfr commented Feb 14, 2017

jcfr left a comment

Choose a reason for hiding this comment

jcfr commented Feb 15, 2017

jcfr commented Feb 15, 2017

JoostJM commented Nov 16, 2016 •

edited

Loading

JoostJM commented Nov 22, 2016 •

edited

Loading

fedorov commented Nov 23, 2016 •

edited

Loading

jcfr commented Nov 23, 2016 •

edited

Loading

jcfr commented Nov 23, 2016 •

edited

Loading

JoostJM commented Nov 23, 2016 •

edited

Loading