SLEAP network error

[bruno-f-cruz]

The following question was asked in the Bonsai Users - Google Groups
Link to original post

---

Original post

Hi bonsai users,

I've been trying to get a SLEAP network in bonsai but I get the following error (below)
Anyone have any idea why this might be happening?

Attached are the configuration files

Thanks,
Diogo

sleap_diogo_network.zip

[bruno-f-cruz]

Hey Diogo,

Can you share the code you used to export the network? There seems to be a problem with the deserialization of your .pb file.

Cheers,
B

[diogoflduarte]

Hi Bruno,

I used the following code to export the .pb file
sleap-export --model "my_model_folder as per documentation here lookup sleap-export.

Caveat: I had to install SLEAP from source since the current version has a bugged parser to tensorflow.
More details in this SLEAP GitHub issue.

Best,
Diogo

[bruno-f-cruz]

Heya,

Out of curiosity, can you try to export using one of the following examples of this notebook?

https://github.com/bruno-f-cruz/bonsai.sleap_export/blob/main/sleap_scripts/export_networks.ipynb
Should be the cell with the comment "Single instance model"

[diogoflduarte]

centroid = CentroidCrop(keras_model=centroid_model, crop_size=16, input_scale = 1.0)

what should the crop_size be? I wrote 16 as per config file.
I get the following error stack:
`---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_41640\2794522486.py in
9 #Make sure you set the crop size to the expected input of the Id layer
10 centroid = CentroidCrop(
---> 11 keras_model=centroid_model, crop_size=16, input_scale = 1.0)
12
13 instance_peaks = TopDownMultiClassFindPeaks(keras_model=id_model, return_class_vectors = True)

c:\users\diogo\careylab\code\sleap\sleap\nn\inference.py in init(self, keras_model, crop_size, input_scale, pad_to_stride, output_stride, peak_threshold, refinement, integral_patch_size, return_confmaps, confmaps_ind, offsets_ind, return_crops, **kwargs)
1546 if self.confmaps_ind is None:
1547 raise ValueError(
-> 1548 "Index of the confidence maps output tensor must be specified if not "
1549 "named 'CentroidConfmapsHead'."
1550 )

ValueError: Index of the confidence maps output tensor must be specified if not named 'CentroidConfmapsHead'.`

what are the dimensions here?
preds = model.predict(np.zeros((4, 1088, 1440, 1), dtype="uint8"))

[bruno-f-cruz]

Run the "single instance" not "centroid"

#Single instance model (using full picture for now...)

runs_folder = r"C:\Users\neurogears\Desktop\Sleap_demo\models"

single_instance_model_path = runs_folder + "\\" + r"221027_092218.single_instance.n=88\best_model.h5"
single_instance_model = tf.keras.models.load_model(single_instance_model_path, compile = False)

model = SingleInstanceInferenceModel(
    SingleInstanceInferenceLayer(keras_model=single_instance_model)
)

preds = model.predict(np.zeros((4, 1088, 1440, 1), dtype="uint8"))

export_frozen_graph(model, preds, Path(runs_folder) / "BonsaiModels" /  "SingleInstance" )

The only parameter you should need to change is the resolution of your image in:

preds = model.predict(np.zeros((4, 1088, 1440, 1), dtype="uint8"))

[diogoflduarte]

Yup, your modifications did result in a compile.

I pointed the bonsai node to the newly exported pb file, it still complained about a missing zlibwapi.dll.
Copied that from the NVIDIA folder to the bonsai "Extensions" folder.

Now I get the following error:

[bruno-f-cruz]

Looking at your network, it seems the input should be:

preds = model.predict(np.zeros((4, 352, 848, 1), dtype="uint8"))

I can run it in Bonsai without errors using:

[diogoflduarte]

Hi Bruno,

1. the compiled network via your code or the one in the sleap repo yields similar networks (same input)
2. The expected image size is larger than the training image. I trained with a 626 x 278 image and the network in the .pb file expects 848 x 352.
3. Your fix for resizing works, however the network performs poorly due to the scaled image. Instead I used ResizeCanvas (bonsai program below) and the output is as expected (for other users, resizecanvas will pad the image with zeros around).

Number 3) is a bit of a hack because the network should expect the 626 x 278 image in any case. I'll point an issue report to this thread.

Thank you for all the help!!!

[bruno-f-cruz]

ResizeCanvas is indeed the identical operation to the preprocessing done in SLEAP. On top of that, it is extremely cheap to run :).
We are going to track this as an issue in the Bonsai.Sleap repo and hopefully try to come up with a way to do this automatically in the future.

bonsai-rx/sleap#10