Question: does this work with TF Keras?

[Raukk]

I'm assuming that it doesn't work with tf.keras yet (or there are different bugs).
Is that correct or did I just do something stupid?

I ask this question because when I tried it with a simple model from a example I got some odd errors.

PS. Thank you; your article was really great, and I really like the visualizations, they are very helpful. Random question, are all the things you pointed out the same in Convolution layers as in your Dense examples, or do the visuals differ?

PSS. One thing I have trouble with is figuring out is which shapes/differences are problems and which are just harmless variations. If you had a cheat sheet explaining which things to look out for in each of the 4 graphs that would be great.

---

Note: This is not an bug/error report. I just thought I should provide more details.

With the simple model I got odd errors like this;

AttributeError                            Traceback (most recent call last)
<ipython-input-16-d1a685d6c63e> in <module>()
----> 1 replaydata = ReplayData(x_test, y_test, filename="filename", group_name="some group", model=model)
      2 
      3 # Now we feed the data to the actual Replay object so we can build the visualizations
      4 replay = Replay(replay_filename="filename", group_name="some group")
      5 

4 frames
/usr/local/lib/python3.6/dist-packages/h5py/_hl/base.py in _e(self, name, lcpl)
    130         else:
    131             try:
--> 132                 name = name.encode('ascii')
    133                 coding = h5t.CSET_ASCII
    134             except UnicodeEncodeError:

AttributeError: 'MeanMetricWrapper' object has no attribute 'encode'

Then, I changed the tf.keras to just Keras and it worked
(except the OOM exception on the GPU; ran in Google Colab).
Here is the whole Workbook: https://gist.github.com/Raukk/2b03bd8447433b83f862c7e7c437ec46

The model code (using tf.keras) is pasted below if you want to look into it.

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.BatchNormalization(input_shape=x_train.shape[1:]))
model.add(tf.keras.layers.Conv2D(64, (5, 5), padding='same', activation='elu'))

model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2), strides=(2,2)))
model.add(tf.keras.layers.Dropout(0.25))

model.add(tf.keras.layers.BatchNormalization(input_shape=x_train.shape[1:]))
model.add(tf.keras.layers.Conv2D(128, (5, 5), padding='same', activation='elu'))
model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))
model.add(tf.keras.layers.Dropout(0.25))

model.add(tf.keras.layers.BatchNormalization(input_shape=x_train.shape[1:]))
model.add(tf.keras.layers.Conv2D(256, (5, 5), padding='same', activation='elu'))
model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2), strides=(2,2)))
model.add(tf.keras.layers.Dropout(0.25))

model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(256))
model.add(tf.keras.layers.Activation('elu'))
model.add(tf.keras.layers.Dropout(0.5))
model.add(tf.keras.layers.Dense(10))
model.add(tf.keras.layers.Activation('softmax'))
model.summary()

[dvgodoy]

Hi @Raukk ,

Thanks for your report. I haven't tested it with TF.Keras - but from the error you pointed out with MeanMetricWrapper, I am assuming it is an old bug, one which I have just fixed and released on v0.1.2a1.
Could you please update DeepReplay and try again, if it solves this problem?

I don't think the plots make so much sense for CNNs... the idea was to demonstrate boundaries for binary classification problems. But if you use CNNs, even if you are trying to distinguish between cats and dogs, for instance, there is not a "boundary" that separates the samples from each other. Well, there actually is, but it lives in the highly dimensional space of the total number of pixels of each image.
You may try to add a second-to-last layer with 2 hidden units and plug DeepReplay to it (as I suggest in one of the examples), but I wouldn't expect much of this...

[Raukk]

I tried it again, with that version. The MeanMetricWrapper error is only if I use the TF.Keras version. It's Running in a Google Colab notebook using Python 3. Successfully installed deepreplay-0.1.2a1
I pasted my code at the bottom, if that helps.
Here's another try at linking the code in colab.

Quick clarification:
With the CNNs, I was only trying to use the Initializer Visualizations as you did in your Part II Article.
The graph that shows Z-Values, Weights, Activations, and Gradients.

From my understanding that CNNs internal mechanics are essentially a tiny Dense Network that is feed each window and then all the outputs get combined. I was assuming that the Weights, and Activations, and such were initialized similarly and could be displayed similarly. It's probably just a misunderstanding on my part.

!pip install deepreplay

import tensorflow as tf
import numpy as np

from deepreplay.callbacks import ReplayData
from deepreplay.datasets.ball import load_data
from deepreplay.plot import compose_plots
from deepreplay.replay import Replay

from matplotlib import pyplot as plt
import os

seed = 13

# get training data
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()

# add single lenght dimension
x_train = np.expand_dims(x_train, -1)
x_test = np.expand_dims(x_test, -1)

# adjust to between 0.0 and 1.0
x_train = x_train / 255.0
x_test = x_test / 255.0


# build a quick simple model
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Flatten(input_shape=x_train.shape[1:]))
model.add(tf.keras.layers.Dense(64))
model.add(tf.keras.layers.Dense(10))
model.add(tf.keras.layers.Activation('softmax'))
model.summary()
model.compile(loss='sparse_categorical_crossentropy',
             optimizer='adam',
             metrics=['accuracy'])

#Code I copied:       
i='7'

# Since we only need initial weights, we don't even need to train the model! 
# We still use the ReplayData callback, but we can pass the model as argument instead
replaydata = ReplayData(x_test, y_test, filename="filename"+i, group_name="group_name"+i, model=model)

# Now we feed the data to the actual Replay object
# so we can build the visualizations
replay = Replay(replay_filename="filename"+i, group_name="group_name"+i)

# Using subplot2grid to assemble a complex figure...
fig = plt.figure(figsize=(12, 6))
ax_zvalues = plt.subplot2grid((2, 2), (0, 0))
ax_weights = plt.subplot2grid((2, 2), (0, 1))
ax_activations = plt.subplot2grid((2, 2), (1, 0))
ax_gradients = plt.subplot2grid((2, 2), (1, 1))

wv = replay.build_weights(ax_weights)
gv = replay.build_gradients(ax_gradients)
# Z-values
zv = replay.build_outputs(ax_zvalues, before_activation=True, 
                          exclude_outputs=True, include_inputs=False)
# Activations
av = replay.build_outputs(ax_activations, exclude_outputs=True, include_inputs=False)

# Finally, we use compose_plots to update all
# visualizations at once
fig = compose_plots([zv, wv, av, gv], 
                    epoch=0, 
                    title='Activation: sigmoid - Initializer: Normal $\sigma = 0.01$')

/usr/local/lib/python3.6/dist-packages/deepreplay/callbacks.py in on_train_begin(self, logs)
    109             metric_name = metric
    110             if callable(metric):
--> 111                 metric_name = metric.__name__
    112             self.group.create_dataset(metric_name, shape=(self.n_epochs,), dtype='f')
    113 

AttributeError: 'MeanMetricWrapper' object has no attribute '__name__'

[dvgodoy]

@Raukk thanks for the info
I think you discovered a bug in the violin plots whenever activation functions are added as an element in the sequential model by itself (as opposed to a parameter in the actual layer).
I am still investigating and figuring out how to fix it - I will let you know when it's done.
There are also some minor adjustments to make it work with TF.keras, but this shouldn't be a problem.

[Raukk]

Glad I could help Identify this.

I generally set the Activation as a parameter as well, but I grabbed a Google Example to start from, and they did it that way so I left it.

I'll let you know if I see anything else as I experiment.