Matrix multiplication

[vandenBergArthur]

Hi all,

I am working with a custom ML model that's originally written using the subclassing method. I am in the process of recreating this model using the Functional API, so that I can use hls4ml to deploy it on an FPGA.

However, I've come across a big issue; the original model performed a tf.matmul(t1,t2) operation where tensor 1 has a shape of (1, 9, 3, 128, 25) and tensor 2 of (1, 3, 25, 25) where the 1st dimension is just the batch size. To convert this into a Keras layer that's supported by hls4ml, I tried to use the Dot layer. Unfortunately, this is unsuccessful because I ran into the following issue:

ERROR: Dot of tensors with rank > 1 is not yet supported.

Then, I tried to create a custom Keras layer and use the hls4ml.converters.tf_to_hls(yamlConfig) function instead of hls4ml.converters.keras_to_hls(cfg) because in the source code of tf_to_hls I found that matmul should be supported.

My model looks like this:

class MatMul(tf.keras.layers.Layer):
    def __init__(self, **kwargs):
        super(MatMul, self).__init__(**kwargs)

    def call(self, inputs):
        return tf.matmul(inputs[0], inputs[1])
    
    def get_config(self):
        config = super(MatMul, self).get_config()
        return config
    
    def from_config(cls, config):
        return cls(**config)

# Define input layers
x = Input(shape=(9, 3, 128, 25), name='input_X')
A = Input(shape=(3,25,25), name='input_A')
inputs=[x,A]

# Define custom layer
matmul_layer = MatMul()

# Define output layer
outputs = matmul_layer(inputs)

# Define model
model = keras.Model(inputs=inputs, outputs=outputs, name='matmul_model')

# Save model
#model.save('/home/arthur/small_testingModels/TEST_TEST', save_traces=True)

# Print summary of model
model.summary()
keras.utils.plot_model(model,show_shapes=True)

Next, I tried to create the hls model with the following code:

yamlConfig = hls4ml.converters.parse_yaml_config('/home/arthur/Downloads/hls4ml_config.yml')

hls_model = hls4ml.converters.tf_to_hls(yamlConfig)

The configuration file that's parsed is created manually and looks like this:

TensorFlowModel: /home/arthur/small_testingModels/TEST_TEST/saved_model.pb
Backend: Vivado
ClockPeriod: 10
HLSConfig:
  Model:
    Precision: ap_fixed<16,6>
    ReuseFactor: 2
    Strategy: Resource
IOType: io_parallel

OutputDir: /home/arthur/configs/TEST_CUSTOMMATMUL/
ProjectName: customMatmulLayerTester
XilinxPart: xc7z020clg400-1

But this results in the following error:

---------------------------------------------------------------------------
DecodeError                               Traceback (most recent call last)
File ~/anaconda3/envs/hls4ml-tutorial/lib/python3.8/site-packages/hls4ml/converters/tf_to_hls.py:136, in tf_to_hls(yamlConfig)
    135         graph_def = tf.compat.v1.GraphDef()
--> 136         graph_def.ParseFromString(f.read())
    137 except BaseException as e:

DecodeError: Error parsing message with type 'tensorflow.GraphDef'

During handling of the above exception, another exception occurred:

Exception                                 Traceback (most recent call last)
Cell In[35], line 1
----> 1 hls_model = hls4ml.converters.tf_to_hls(yamlConfig)

File ~/anaconda3/envs/hls4ml-tutorial/lib/python3.8/site-packages/hls4ml/converters/tf_to_hls.py:138, in tf_to_hls(yamlConfig)
    136         graph_def.ParseFromString(f.read())
    137 except BaseException as e:
--> 138     raise Exception('Error loading the graph definition: {}'.format(str(e)))
    140 try:
    141     assert graph_def is not None

Exception: Error loading the graph definition: Error parsing message with type 'tensorflow.GraphDef'

So, my question is; is it possible to use a custom Keras layer inside a Keras model that uses tf.matmul? If so, how do I save the model correctly ? Do I use the SavedModel or the h5 format? And, how do I create a correct configuration file?

Ps: Am I correct that sublcassed Keras models won't work with hls4ml?

I look forward to your feedback!

Thanks in advance!

Arthur

[vloncar]

We don't have direct implementation of matmul. We use matrix-vector multiplication in most places (like Dense layer or convolutional layers). Also, tf_to_hls hasn't seen development for a long time, so it probably doesn't work well anymore. But the biggest issue I see here that I mentioned in the other issues you opened is that those tensors are simply too large for io_parallel

[vandenBergArthur]

Thanks for the reply @vloncar !

We are trying to make a (ST-)GCN Model that is compatible with hls4ml so that we can deploy it (or blocks of the algorithm) on a PYNQ-Z2 board (with Vivado). But we are stuck since the matrix multiplication with the adjacency matrix is a key part of the algorithm.

When digging into the source code, we found some template header files like nnet_conv2d.h. Since it seems like we will not be able to use the Dot layer, we were thinking of customizing/creating a header file that will suit our needs (to perform matmul). Is this idea feasible from your point of view? (Unfortunately we are pretty inexperienced in C/C++)
Or do you have other suggestions?

[vloncar]

You can use the extension API to introduce your custom layer and HLS implementation to hls4ml. See the example of KL loss. Getting that HLS to behave well will be another matter. Naive approaches with unrolling the matrix multiplication loops will probably not work. But then there's also a matter of what are your latency constraints and do you even need to unroll that much.

Also, we're working on enhancing GNN support in hls4ml, that will be the major focus for this year, so stay tuned for that.