Automatic precision inference #855

vloncar · 2023-08-20T21:01:01Z

Description

This introduces the ability to specify auto as a precision string, that implies hls4ml should infer the precision in some way. This is not exposed by default via the config_from... functions for now. The goal is to have the framework for inferring types in some ways within hls4ml (e.g., QONNX parser) before fully exposing it to users. An initial inference of precision has been added via the infer_precision_types optimizer, based on previous attempts by various people. It's not advanced in any way. During testing, I encountered some issues with SeparableConv1D templates which I fixed.

Type of change

Bug fix (non-breaking change that fixes an issue) - Only related to the SeparableConv1D issue
New feature (non-breaking change which adds functionality)

Tests

There are new tests in test_auto_precision.py that cover the few use cases.

Checklist

I have read the guidelines for contributing.
I have commented my code, particularly in hard-to-understand areas.
I have made corresponding changes to the documentation.
My changes generate no new warnings.
I have installed and run pre-commit on the files I edited or added.
I have added tests that prove my fix is effective or that my feature works.

jmitrevs · 2023-10-11T00:32:58Z

Clang doesn't seem to like this

(fastml39) Jovans-Mac:hls4mlprj_auto_conv2d_Quartus_io_stream jmitrevs$ bash build_lib.sh 
In file included from firmware/myproject.cpp:2:
In file included from firmware/parameters.h:11:
firmware/nnet_utils/nnet_conv2d_stream.h:135:5: error: no matching function for call to 'shift_line_buffer_2d'
    nnet::shift_line_buffer_2d<data_T, CONFIG_T>(in_elem, line_buffer, shift_buffer);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
firmware/nnet_utils/nnet_conv2d_stream.h:199:13: note: in instantiation of function template specialization 'nnet::compute_output_buffer_2d<nnet::array<ac_fixed<16, 6>, 4>, nnet::array<ac_fixed<35, 15>, 4>, config8>' requested here
            compute_output_buffer_2d<data_T, res_T, CONFIG_T>(padds, res, line_buffer, kernel_window, weights, biases);
            ^
firmware/myproject.cpp:93:11: note: in instantiation of function template specialization 'nnet::conv_2d_cl<nnet::array<ac_fixed<16, 6>, 4>, nnet::array<ac_fixed<35, 15>, 4>, config8>' requested here
    nnet::conv_2d_cl<layer7_t, last_layer_result_t, config8>(layer7_out, layer8_out, w8, b8);
          ^
firmware/nnet_utils/nnet_conv2d_stream.h:69:6: note: candidate template ignored: substitution failure [with data_T = nnet::array<ac_fixed<16, 6>, 4>, CONFIG_T = config8]: zero-length arrays are not permitted in C++
void shift_line_buffer_2d(
     ^
1 error generated.
clang: error: no such file or directory: 'myproject.o'

while g++ compiles it just fine.

jmitrevs · 2023-10-11T03:12:10Z

The issue seems to be that this:

    model.add(Conv2D(4, kernel_size=(1, 1), activation='relu', name='last_layer'))  # Will become PointwiseConv2D

doesn't actually become PointwiseConv2D for Quartus. The bug it uncovered seems tangential to this PR; nevertheless, we need to fix it, either as part of this PR or separately.

vloncar · 2023-10-11T18:37:15Z

is this the same issue that was observed in #878?

jmitrevs · 2023-10-11T18:41:51Z

is this the same issue that was observed in #878?

I believe so. If you have a filter of size 1, then things like line_buffer[CONFIG_T::filt_height - 1][CONFIG_T::n_chan] wind up with zero-size arrays.

hls4ml/backends/quartus/quartus_backend.py

hls4ml/backends/vivado/vivado_backend.py

test/pytest/test_auto_precision.py

jmitrevs · 2023-11-20T18:45:43Z

hls4ml/model/optimizer/passes/infer_precision.py

+
+    def _infer_precision(self, node, types_to_infer):
+        node_class = node.class_name
+        if node_class in ['Dense']:


I wonder if it's better to use something like isinstance(node, Dense) instead of matching to a class name. Matching to a class name doesn't deal with inheritance. For example, I can see the BatchNormalization matching failing for ApplyAlpha matching.

LOL, I did it specifically to avoid this since it is usually not what we want. The ApplyAlpha is an example of this.

With QONNX ApplyAlpha does need the precision propagated. It worries me if derived classes by default have different behavior. That violates the "is a" principle. I think if you want different behavior you explicitly should code it. What happens in ApplyAlpha if you don't forbid it? Should QONNX not use ApplyAlpha in this case?

You derive a class to have different behavior, not the same one. By principle, ApplyAlpha "is NOT" a BatchNormalization conceptually, it just happens to share the implementation (and honestly it shouldn't, both should have a same parent class, for example ScaleShift, then your logic would be good). Another example of this is DepthwiseConv2D vs Conv2D. It's true they are convolutions, but they have different behavior. I think it would be better for new layers that potentially inherit from other layers instead of the base Layer class to be unsupported by this rather then for them to be silently supported in a wrong way. I'm not gonna die on this hill though, if you feel strongly about this we could revisit, but I'd like stronger arguments 😄.

For ApplyAlpha, you can use it in QONNX, I can add it to have the same behavior as BN if that is what is needed.

hls4ml/model/optimizer/passes/infer_precision.py

jmitrevs · 2023-11-20T22:20:23Z

hls4ml/model/optimizer/passes/infer_precision.py

+        return inferred_types
+
+    def _infer_dense_precision(self, node, types_to_infer):
+        n_ops = node.get_attr('n_in') * node.get_attr('n_out')


I don't think the total n_ops is the important value for the accumulator precision. Ignoring bias for now, each output value is a certain number of multiplies, resulting in the input_width + weight_width part of the equation, plus the accumulation, math.ceil(np.log2(num_acc)). But num_acc != num_ops. In particular, it's node.get_attr('n_in') (-1?), at least for the standard 1D Dense layer. Bias modifies things a bit, but the general trends are the same. This is the result we had from the CMS hackathon with Sioni: https://github.com/jmitrevs/hls4ml/blob/bit-correct/hls4ml/model/optimizer/passes/propagate_dense_precision.py

Ah, true, I'll fix this.

hls4ml/model/optimizer/passes/infer_precision.py

jmitrevs · 2023-11-21T01:46:47Z

hls4ml/model/optimizer/passes/infer_precision.py

+
+        return ['result_t']
+
+    def _infer_common_precision(self, node, types_to_infer, n_ops):


I think this assumes integer or fixed precision types. Do we need to handle, e.g., xnor precision types?

I should test this PR with the binary model that has Xnor types. I wouldn't expect that you need to infer anything in that case, but perhaps it breaks the optimizer.

jmitrevs · 2024-01-27T00:26:29Z

I made vloncar #53 into your branch with changes for dense and standard convolution. Let me know what you think. If you like the way I made this, I can also add signed/unsigned support to the other precision propagations (like merge, bn, sepconv) either in that PR or a different one.

jmitrevs · 2024-02-05T23:25:08Z

hls4ml/model/optimizer/passes/infer_precision.py

+    def _infer_bn_precision(self, node, types_to_infer):
+        inferred_types = []
+
+        if 'scale_t' in types_to_infer:


Wouldn't the input quantization in this case be the input mean and variance (+ other things), not the scale and bias?

(actually I have to see how this is handled by the qkeras parser)

hls4ml/model/optimizer/passes/infer_precision.py

…rallel conv

jmitrevs · 2024-03-02T20:53:42Z

hls4ml/model/optimizer/__init__.py

@@ -33,6 +33,7 @@
 register_flow(
    'convert',
    [
+        'infer_precision_types',


I would prefer having this towards the end of convert, or just leaving the call out of convert, since we call it in the optimize flow anyway. What is the purpose of doing it here? Here it messes up some of the conversion steps I have for qonnx, which only happen if the type is not set, since it doesn't want to override set types, and this inferring here sets the types.

"optimize" is sort-of optional, it belongs to "convert" because after that stage the other optimizers don't expect "auto" to exist. If onnx has its own optimizers that run, why not group them into a flow and run before convert? the idea is that after "convert" it shouldn't matter where the model came from

But we can move it later in concert, right?

If you have onnx-specific optimizers that must go before, place them before this one.

latency pooling overhaul vivado latency pooling overhaul vitis latency pooling overhaul, fix comment fix boundry cond fix syn issues latency pooling overhaul Fix pooling accum_t autoset & avoid global override [pre-commit.ci] auto fixes from pre-commit hooks better way to get inp layer name fix for vitis / input_t fetch torch padding fix avoid name dup in torch api test rm pooling precision override in favor of fastmachinelearning#855

vloncar added 4 commits August 20, 2023 22:46

Add UnspecifiedPrecisionType

e8317c4

Rudimentary optimizer to infer 'auto' precision

7f6ff3c

Auto precision test

a8cf7e8

Sepconv fixes

8d10f24

vloncar requested a review from jmitrevs August 20, 2023 21:01

vloncar added the please test Trigger testing by creating local PR branch label Aug 20, 2023

jmitrevs added this to the v0.8.0 milestone Sep 8, 2023

Merge branch 'main' into auto_precision

2bf8c4a

jmitrevs added please test Trigger testing by creating local PR branch and removed please test Trigger testing by creating local PR branch labels Oct 6, 2023

jmitrevs mentioned this pull request Oct 11, 2023

Add support for filt_height==1 for streaming quartus conv2d #886

Merged

7 tasks

jmitrevs modified the milestones: v0.8.0, v1.0.0 Oct 20, 2023

This was referenced Nov 9, 2023

[DRAFT] Fix pooling accum_t autoset & avoid global override #917

Closed

Add support for HGQ proxy model #914

Merged

jmitrevs reviewed Nov 17, 2023

View reviewed changes

hls4ml/backends/quartus/quartus_backend.py Show resolved Hide resolved

jmitrevs reviewed Nov 17, 2023

View reviewed changes

hls4ml/backends/vivado/vivado_backend.py Show resolved Hide resolved

Merge branch 'main' into auto_precision

ef66b9b

jmitrevs added please test Trigger testing by creating local PR branch and removed please test Trigger testing by creating local PR branch labels Nov 17, 2023

jmitrevs reviewed Nov 18, 2023

View reviewed changes

test/pytest/test_auto_precision.py Show resolved Hide resolved

jmitrevs reviewed Nov 20, 2023

View reviewed changes

hls4ml/model/optimizer/passes/infer_precision.py Show resolved Hide resolved

jmitrevs reviewed Nov 20, 2023

View reviewed changes

jmitrevs reviewed Nov 21, 2023

View reviewed changes

hls4ml/model/optimizer/passes/infer_precision.py Outdated Show resolved Hide resolved

jmitrevs reviewed Nov 21, 2023

View reviewed changes

jmitrevs reviewed Feb 5, 2024

View reviewed changes

jmitrevs reviewed Feb 6, 2024

View reviewed changes

hls4ml/model/optimizer/passes/infer_precision.py Outdated Show resolved Hide resolved

jmitrevs and others added 3 commits February 6, 2024 21:22

update precision propagation for signed, select im2col for quartus pa…

62b9cc8

…rallel conv

Merge branch 'main' into auto_precision

f46f7d5

Make inferring no_bias a configurable option of the optimizer

bf0c778

jmitrevs mentioned this pull request Feb 21, 2024

Latency Pooling Header Updates #973

Merged

7 tasks

Merge branch 'main' into auto_precision

ae4e288

jmitrevs reviewed Mar 2, 2024

View reviewed changes

jmitrevs mentioned this pull request Mar 12, 2024

Update QONNX parsing for 1.0 #979

Merged

8 tasks

jmitrevs and others added 4 commits April 17, 2024 17:31

updates to infering precision from qonnx branch

66225c9

remove count, become more selective on when True is returned

023cfa2

fix pooling precision

6014dc8

remove typing

7019f19

calad0i added a commit to calad0i/hls4ml that referenced this pull request Apr 18, 2024

rm pooling precision override in favor of fastmachinelearning#855

dcf805e

Merge branch 'main' into auto_precision

aa457f1

jmitrevs added please test Trigger testing by creating local PR branch and removed please test Trigger testing by creating local PR branch labels Apr 18, 2024

Fix avg pooling op check

6c26f1a

vloncar force-pushed the auto_precision branch from 9f0bb0e to 6c26f1a Compare April 18, 2024 19:13

jmitrevs added please test Trigger testing by creating local PR branch and removed please test Trigger testing by creating local PR branch labels Apr 18, 2024

jmitrevs approved these changes Apr 19, 2024

View reviewed changes

jmitrevs merged commit 1616caf into fastmachinelearning:main Apr 19, 2024
9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automatic precision inference #855

Automatic precision inference #855

vloncar commented Aug 20, 2023

jmitrevs commented Oct 11, 2023

jmitrevs commented Oct 11, 2023

vloncar commented Oct 11, 2023

jmitrevs commented Oct 11, 2023

jmitrevs Nov 20, 2023 •

edited

Loading

vloncar Nov 20, 2023

jmitrevs Nov 20, 2023 •

edited

Loading

vloncar Nov 21, 2023

jmitrevs Nov 20, 2023

vloncar Nov 21, 2023

jmitrevs Nov 21, 2023

vloncar Nov 21, 2023

jmitrevs commented Jan 27, 2024

jmitrevs Feb 5, 2024

jmitrevs Feb 5, 2024

jmitrevs Mar 2, 2024

vloncar Mar 2, 2024

jmitrevs Mar 2, 2024

vloncar Mar 2, 2024


		return ['result_t']

		def _infer_common_precision(self, node, types_to_infer, n_ops):

Automatic precision inference #855

Automatic precision inference #855

Conversation

vloncar commented Aug 20, 2023

Description

Type of change

Tests

Checklist

jmitrevs commented Oct 11, 2023

jmitrevs commented Oct 11, 2023

vloncar commented Oct 11, 2023

jmitrevs commented Oct 11, 2023

jmitrevs Nov 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmitrevs Nov 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmitrevs commented Jan 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmitrevs Nov 20, 2023 •

edited

Loading

jmitrevs Nov 20, 2023 •

edited

Loading