Add MXNet Backend (#59)

* Adding MXNet backend template. Adding all basic Variable and Tensor operations (#1)

* add activation functions

* add activation functions

* fix some legacy

* fix some legacy

* cross entropy

* cross entropy

* fix name scoping introduced in 2.0

* fix name scoping introduced in 2.0

* Add dropout, l2_normalization, random_normal/uniform/binomial (#2)

* remove the logic for hacking RNN

* remove the logic for hacking RNN

* add pooling with utils

* add pooling with utils

* minor

* lint and name scope fix

* fix access protected var

* fix add neighbor, removed __eq__ in KerasSymbol

* fix eval function, unittest for placeholder and variable

* add unittests

* fix bug

* fix bug

* fix

* add some temporary fixes in mxnet backend. undo change to the pytest.ini

* mxnet_backend graph fix, layer support  (#3)

* add activation functions

* fix some legacy

* cross entropy

* fix name scoping introduced in 2.0

* Add dropout, l2_normalization, random_normal/uniform/binomial (#2)

* remove the logic for hacking RNN

* add pooling with utils

* add activation functions

* fix some legacy

* cross entropy

* fix name scoping introduced in 2.0

* remove the logic for hacking RNN

* add pooling with utils

* minor

* lint and name scope fix

* fix access protected var

* fix add neighbor, removed __eq__ in KerasSymbol

* fix eval function, unittest for placeholder and variable

* add unittests

* fix bug

* fix bug

* fix

* add some temporary fixes in mxnet backend. undo change to the pytest.ini

* Keras function not working is a known issue, add skip in the test

* fix random_uniform/constant

* fix legacy randomize methods

* Fix MXNet backend operator bugs. Enabled Keras backend tests

* add bias

* Add Amazon copyrights to License (#6)

* fix

* fix

* fix backend for mlp

* fix context management, add optimizers

* minor change

* undo changes on example

* fix eval

* minor cleanup

* fix some property usage

* fixing AlphaDroupout, not finished yet

* add mx model instantiate

* modifies training model construct logic, fix some tests. fix reshape layer.

* minor fix

* fix bias_add

* more fix on Dense and bias_add

* In progress commit

* fix comment

* small fix

* remove pytest.skip in conv3d. But it failed with theano backend in my workspace though.

* Add conv2d and in_topk operator for mxnet backend (#11)

* Skip BatchDot tests for Theano backend. (#12)

* BatchDot, Basic Batchnorm, Fix BiasAdd, Fix Conv2D, CodeCleanup (#14)

* Fix Conv2d shape issues and enable Conv2D UTs

* Remove redundant mxnet only unit tests

* Adding batch_dot, remove deconv, code comments and cleanup

* Remove buggy conv1d implementation

* Fix CR comments. Fix lint check issues

* Move mxnet specific code from keras engine to mxnet_backend. (#15)

* Move MXNet optimizers from keras optimizers to mxnet backend (#16)

* Fix bug in reshape. Minor rename to avoid local conflicts

* Bug fixes and enable/skip all Keras tests for mxnet backend (#21)

* test results - 374 passed, 235 skipped in 114.44 seconds

* fix/skip keras tests - tests/integration_tests, tests/keras/applications

* fix/skip keras tests - tests/keras/engine/test_topology

* fix/skip keras tests - tests/keras/engine/test_training

* fix/skip keras tests - tests/keras/legacy/

* fix/skip keras tests - tests/keras/preprocessing

* fix/skip keras tests - tests/keras/utils/

* Fix CR comments

* Fix issues in zero_padding. Fix/Enable tests/layers/convolutional_test

* Add momentum to batchnorm. Enable/skip tests in layers/core, local, merge, noise, normalization

* Skip RNN tests in keras/tests/layers/recurrent_test, wrappers_test

* Fix bug in spatial padding, enable/skip tests in loss,optimizers,callback,loss_weighting, model_saving

* Fix mxnet backend multi-gpu training (#31)

Fixing bug for mxnet backend to use multiple gpus.

* Fix performance issue - Batchnormalization, Conv operator (#35)

* Fix default axis for batchnorm layer for channels_first data_format

* Performance improvement by avoiding kernel transpose in conv operation for channels_first format

* Fix model - architecture, weights and both, load and save. (#36)

* Prepare initial version of mxnet related documentation in keras (#38)

* Skip failing unit tests for unsupported functionality in mxnet backend

* Fix pep tests reported by CI

* Use pytest module skip, revert kernel_shape logic

* remove data_format param from bias_add API

* Allow Predict() without compile for mxnet backend and enable tests.

contributor - roywei@

* Fix bug - mxnet backend should not override keras config data_format to channels_first. Only warn of low performance

* Conv3d() operator implementation for Keras2.0 using MXNet backend (#40)

* conv3d implementation for keras2.0 as MXNet backend

* conv3d implementation/testing for keras2.0 using MXNet backend

* keeping -n option in pytest.ini file

* fixed comments given by Sandeep

* Add Conv1D support for MXNet backend (#44)

* Add Conv1D support for MXNet backend

* Fix CR comments

* Conv2d transpose (#47)

* add conv2d_transpose

* conv2d transpose for both channels, enabled test case

* add detailed comments and examples, fix style issue

* enable test case in topology

* Enable performance optimization for conv operators with MXNet backend. Make MXNet default backend with this branch (#48)

* Fix conv kernel shape bug for TF backend. (#50)

* Add support for keras multi_gpu_model() API with MXNet backend (#49)

* Add support for keras multi_gpu_model() API with MXNet backend. Autoset GPU0 context on GPU machine

* Fix typo

* Add SAME padding mode support for pooling operator. (#51)

* Add rnn() operator for MXNet backend with unrolling and masking feature (#46)

* Adding rnn() operator in Keras2.0 with MXNet as backend with unroll=True and Masking=True/False and enabled relevant testcases. Also, modified couple of operators.

* Modified comments

* Added comments to a method

* Enable categorical crossentropy testcases and made minor changes

* Modified message

* nit

* Added detail description of handling variable length input in RNN

* Skip conv2d_transpose and conv3d_transpose test-case for MXNet backend and minor changes in rnn()

* Adamax and NAdam optimizer for MXNet backend (#54)

* Add Adamax optimizer for MXNet backend

* Fix lr and adamax params

* Add Nadam optimizer for mxnet backend

* Add Conv3d transpose (#52)

* conv3d tranpose, enabled test case

* update kernel shape

* replace conv2d_transpse conv3d_transpose with convnd_transpose

* update value errors with MXNet Backend info, fix typo

* add check for conv3d transpose only supports gpu with cudnn

* update context check

* diable conv3d transpose test

* fix typo in comment

* Adding MXNet backend template. Adding all basic Variable and Tensor operations (#1)

* add activation functions

* add activation functions

* fix some legacy

* fix some legacy

* cross entropy

* cross entropy

* fix name scoping introduced in 2.0

* fix name scoping introduced in 2.0

* Add dropout, l2_normalization, random_normal/uniform/binomial (#2)

* remove the logic for hacking RNN

* remove the logic for hacking RNN

* add pooling with utils

* add pooling with utils

* minor

* lint and name scope fix

* fix access protected var

* fix add neighbor, removed __eq__ in KerasSymbol

* fix eval function, unittest for placeholder and variable

* add unittests

* fix bug

* fix bug

* fix

* add some temporary fixes in mxnet backend. undo change to the pytest.ini

* mxnet_backend graph fix, layer support  (#3)

* add activation functions

* fix some legacy

* cross entropy

* fix name scoping introduced in 2.0

* Add dropout, l2_normalization, random_normal/uniform/binomial (#2)

* remove the logic for hacking RNN

* add pooling with utils

* add activation functions

* fix some legacy

* cross entropy

* fix name scoping introduced in 2.0

* remove the logic for hacking RNN

* add pooling with utils

* minor

* lint and name scope fix

* fix access protected var

* fix add neighbor, removed __eq__ in KerasSymbol

* fix eval function, unittest for placeholder and variable

* add unittests

* fix bug

* fix bug

* fix

* add some temporary fixes in mxnet backend. undo change to the pytest.ini

* Keras function not working is a known issue, add skip in the test

* fix random_uniform/constant

* fix legacy randomize methods

* Fix MXNet backend operator bugs. Enabled Keras backend tests

* add bias

* Add Amazon copyrights to License (#6)

* fix

* fix

* fix backend for mlp

* fix context management, add optimizers

* minor change

* undo changes on example

* fix eval

* minor cleanup

* fix some property usage

* fixing AlphaDroupout, not finished yet

* add mx model instantiate

* modifies training model construct logic, fix some tests. fix reshape layer.

* minor fix

* fix bias_add

* more fix on Dense and bias_add

* In progress commit

* fix comment

* small fix

* remove pytest.skip in conv3d. But it failed with theano backend in my workspace though.

* Add conv2d and in_topk operator for mxnet backend (#11)

* Skip BatchDot tests for Theano backend. (#12)

* BatchDot, Basic Batchnorm, Fix BiasAdd, Fix Conv2D, CodeCleanup (#14)

* Fix Conv2d shape issues and enable Conv2D UTs

* Remove redundant mxnet only unit tests

* Adding batch_dot, remove deconv, code comments and cleanup

* Remove buggy conv1d implementation

* Fix CR comments. Fix lint check issues

* Move mxnet specific code from keras engine to mxnet_backend. (#15)

* Move MXNet optimizers from keras optimizers to mxnet backend (#16)

* Fix bug in reshape. Minor rename to avoid local conflicts

* Bug fixes and enable/skip all Keras tests for mxnet backend (#21)

* test results - 374 passed, 235 skipped in 114.44 seconds

* fix/skip keras tests - tests/integration_tests, tests/keras/applications

* fix/skip keras tests - tests/keras/engine/test_topology

* fix/skip keras tests - tests/keras/engine/test_training

* fix/skip keras tests - tests/keras/legacy/

* fix/skip keras tests - tests/keras/preprocessing

* fix/skip keras tests - tests/keras/utils/

* Fix CR comments

* Fix issues in zero_padding. Fix/Enable tests/layers/convolutional_test

* Add momentum to batchnorm. Enable/skip tests in layers/core, local, merge, noise, normalization

* Skip RNN tests in keras/tests/layers/recurrent_test, wrappers_test

* Fix bug in spatial padding, enable/skip tests in loss,optimizers,callback,loss_weighting, model_saving

* Fix mxnet backend multi-gpu training (#31)

Fixing bug for mxnet backend to use multiple gpus.

* Fix performance issue - Batchnormalization, Conv operator (#35)

* Fix default axis for batchnorm layer for channels_first data_format

* Performance improvement by avoiding kernel transpose in conv operation for channels_first format

* Fix model - architecture, weights and both, load and save. (#36)

* Prepare initial version of mxnet related documentation in keras (#38)

* Skip failing unit tests for unsupported functionality in mxnet backend

* Fix pep tests reported by CI

* Use pytest module skip, revert kernel_shape logic

* remove data_format param from bias_add API

* Allow Predict() without compile for mxnet backend and enable tests.

contributor - roywei@

* Fix bug - mxnet backend should not override keras config data_format to channels_first. Only warn of low performance

* Conv3d() operator implementation for Keras2.0 using MXNet backend (#40)

* conv3d implementation for keras2.0 as MXNet backend

* conv3d implementation/testing for keras2.0 using MXNet backend

* keeping -n option in pytest.ini file

* fixed comments given by Sandeep

* Add Conv1D support for MXNet backend (#44)

* Add Conv1D support for MXNet backend

* Fix CR comments

* Conv2d transpose (#47)

* add conv2d_transpose

* conv2d transpose for both channels, enabled test case

* add detailed comments and examples, fix style issue

* enable test case in topology

* Enable performance optimization for conv operators with MXNet backend. Make MXNet default backend with this branch (#48)

* Fix conv kernel shape bug for TF backend. (#50)

* Add support for keras multi_gpu_model() API with MXNet backend (#49)

* Add support for keras multi_gpu_model() API with MXNet backend. Autoset GPU0 context on GPU machine

* Fix typo

* Add SAME padding mode support for pooling operator. (#51)

* Add rnn() operator for MXNet backend with unrolling and masking feature (#46)

* Adding rnn() operator in Keras2.0 with MXNet as backend with unroll=True and Masking=True/False and enabled relevant testcases. Also, modified couple of operators.

* Modified comments

* Added comments to a method

* Enable categorical crossentropy testcases and made minor changes

* Modified message

* nit

* Added detail description of handling variable length input in RNN

* Skip conv2d_transpose and conv3d_transpose test-case for MXNet backend and minor changes in rnn()

* Adamax and NAdam optimizer for MXNet backend (#54)

* Add Adamax optimizer for MXNet backend

* Fix lr and adamax params

* Add Nadam optimizer for mxnet backend

* Add Conv3d transpose (#52)

* conv3d tranpose, enabled test case

* update kernel shape

* replace conv2d_transpse conv3d_transpose with convnd_transpose

* update value errors with MXNet Backend info, fix typo

* add check for conv3d transpose only supports gpu with cudnn

* update context check

* diable conv3d transpose test

* fix typo in comment

* Rebase to latest Keras - April 3, 2018

* Add build badges

* Fix multi_gpu API bug for CPU. Fix PEP. (#64)

* Fix multi_gpu API bug for CPU. Fix PEP.

* fix embedding layer bug (#61)

* fix embedding bug

* addressed comments, enabled more test cases

* add keras test

* reduce line length

* fix style, add blank lines

* Benchmark (#55)

* add conv2d_transpose

* conv2d transpose for both channels, enabled test case

* add detailed comments and examples, fix style issue

* add benchmark scripts for resnet and imagenet data

* combine scripts

* fix args

* fix num of gpus

* update log

* multi_gpu_model only support tf

* add benchamrk scripts for synthetic data

* update read me and scripts

* add mxnet traing result table

* update on readme

* add cifar10 dataset and enable various resnet layers

* fix compile for mxnet multiple gpu

* update callbacks

* update synthetic data script, add credits

* undo new line

* update readme, addressed pr comments

* update readme

* benchmark scripts style fix (#66)

* style fix

* remove unused import, fix line too long

* adrressed pr comments

* Added keras util API for conversion of data tensor from channels_last to channels_first using MXNet backend (#65)

* Added keras util API for conversion of data tensor from channels_last to channels_first using MXNet backend

* Modified comments

* Addressed review comments and made the API more generic accross backends

* Removed shape check

* Modified comments

* Added edge cases

* moved helper method as nested

* Added RNN benchmark scripts (#69)

* Added RNN benchmark scripts

* Fixed new line in bash script

* Removed different backend code and modified comments

* Removed spacing

* Automated the wikiText2 download script

* Added dataset_util functionality to have more flexible code

* Added minor comments

* modified minor comments

* Fixed the multi-gpu context (#68)

* Update benchmark result (#70)

* update benchmark result

* update result

* simplify folder structure

* add image result

* add note

* add note

keras/backend/__init__.py

@@ -38,6 +38,10 @@
     _epsilon = _config.get('epsilon', epsilon())
     assert isinstance(_epsilon, float)
     _backend = _config.get('backend', _BACKEND)
+<<<<<<< HEAD
+=======
+    assert _backend in {'theano', 'tensorflow', 'cntk', 'mxnet'}
+>>>>>>> Add MXNet Backend (#59)
     _image_data_format = _config.get('image_data_format',
                                      image_data_format())
     assert _image_data_format in {'channels_last', 'channels_first'}
@@ -73,6 +77,10 @@
 # Set backend based on KERAS_BACKEND flag, if applicable.
 if 'KERAS_BACKEND' in os.environ:
     _backend = os.environ['KERAS_BACKEND']
+<<<<<<< HEAD
+=======
+    assert _backend in {'theano', 'tensorflow', 'cntk', 'mxnet'}
+>>>>>>> Add MXNet Backend (#59)
     _BACKEND = _backend
 
 # Import backend functions.

tests/keras/backend/backend_test.py

@@ -1154,6 +1154,34 @@ def test_depthwise_conv_2d(self, k):
         # Test invalid use cases
         with pytest.raises(ValueError):
             k.depthwise_conv2d(k.variable(x_val), k.variable(kernel_val), data_format='channels_middle')
+<<<<<<< HEAD
+=======
+
+    @pytest.mark.skipif(K.backend() == 'theano', reason='Not supported.')
+    @pytest.mark.parametrize('op,input_shape,kernel_shape,depth_multiplier,padding,data_format', [
+        ('separable_conv2d', (2, 3, 4, 5), (3, 3), 1, 'same', 'channels_first'),
+        ('separable_conv2d', (2, 3, 5, 6), (4, 3), 2, 'valid', 'channels_first'),
+        ('separable_conv2d', (1, 6, 5, 3), (3, 4), 1, 'valid', 'channels_last'),
+        ('separable_conv2d', (1, 7, 6, 3), (3, 3), 2, 'same', 'channels_last'),
+    ])
+    def test_separable_conv2d(self, op, input_shape, kernel_shape, depth_multiplier, padding, data_format):
+        input_depth = input_shape[1] if data_format == 'channels_first' else input_shape[-1]
+        _, x = parse_shape_or_val(input_shape)
+        _, depthwise = parse_shape_or_val(kernel_shape + (input_depth, depth_multiplier))
+        _, pointwise = parse_shape_or_val((1, 1) + (input_depth * depth_multiplier, 7))
+        y1 = reference_operations.separable_conv(x, depthwise, pointwise, padding, data_format)
+        if K.backend() == 'cntk':
+            y2 = cntk_func_three_tensor(
+                op, input_shape,
+                depthwise, pointwise,
+                padding=padding, data_format=data_format)([x])[0]
+        else:
+            y2 = K.eval(getattr(K, op)(
+                K.variable(x),
+                K.variable(depthwise), K.variable(pointwise),
+                padding=padding, data_format=data_format))
+        assert_allclose(y1, y2, atol=1e-05)
+>>>>>>> Add MXNet Backend (#59)
 
     def legacy_test_pool2d(self):
         check_single_tensor_operation('pool2d', (5, 10, 12, 3),
@@ -1262,7 +1290,7 @@ def test_conv_invalid_use(self):
         with pytest.raises(ValueError):
             K.conv3d(dummy_x_3d, dummy_w_3d, data_format='channels_middle')
 
-        if K.backend() != 'theano':
+        if K.backend() != 'theano' and K.backend() != 'mxnet':
             with pytest.raises(ValueError):
                 K.separable_conv2d(dummy_x_2d, dummy_w_2d, dummy_w1x1_2d,
                                    data_format='channels_middle')

tests/keras/engine/test_training.py

@@ -517,8 +517,12 @@ def expected_shape(batch_size, n_batches):
         assert np.shape(out) == shape_0
 
 
+<<<<<<< HEAD
 @pytest.mark.skipif(sys.version_info < (3,),
                     reason='Cannot catch warnings in python 2')
+=======
+@pytest.mark.skipif(sys.version_info < (3,), reason='Cannot catch warnings in python 2')
+>>>>>>> Add MXNet Backend (#59)
 @keras_test
 def test_warnings():
     a = Input(shape=(3,), name='input_a')

tests/keras/layers/wrappers_test.py

@@ -12,6 +12,10 @@
 from keras.utils.generic_utils import object_list_uid, to_list
 
 
+pytestmark = pytest.mark.skipif(K.backend() == 'mxnet',
+                                reason='MXNet backend does not support TimeDistributed and RNN yet')
+
+
 pytestmark = pytest.mark.skipif(K.backend() == 'mxnet',
                                 reason='MXNet backend does not support TimeDistributed and RNN yet')