Problem with Multi-species models

[grand0lf]

Hi
I have been trying to train a model with multiple tree species but no matter how I do it I end up with what seems to be nothing. Training takes time and seems to go well but predict_image (with an image from the training dataset) returns None and evaluate as well.
I have tried the example in the documentation for Multi-species models with the same results. Am I doing something wrong? What shall I do after m.trainer.fit(m) to test, save and use my trained model?
I have been trying both locally in a Jupyter notebook and in Google colab with the same results.

Eg:

import os
from deepforest import main
from deepforest import get_data

m = main.deepforest(config_args={"num_classes":2}, label_dict={"Alive":0,"Dead":1})

deepforest_release_model = main.deepforest()
deepforest_release_model.use_release() # or use_bird_release

m.model.backbone.load_state_dict(deepforest_release_model.model.backbone.state_dict())

m.model.head.regression_head.load_state_dict(deepforest_release_model.model.head.regression_head.state_dict())

csv_file = get_data("testfile_multi.csv")
m.config["train"]["csv_file"] = csv_file
m.config["train"]["root_dir"] = os.path.dirname(csv_file)
m.config["train"]["fast_dev_run"] = True
m.config["train"]['epochs'] = 5
    
m.config["validation"]["csv_file"] = csv_file
m.config["validation"]["root_dir"] = os.path.dirname(csv_file)

m.create_trainer()
m.trainer.fit(m)

predictions = m.predict_file(csv_file=csv_file, root_dir=os.path.dirname(csv_file))
predictions.head()

image_path = get_data("SOAP_061.png")
boxes = m.predict_image(path=image_path, return_plot = True, color=(0, 165, 255), thickness=3)
print(boxes)

save_dir = os.path.join(os.getcwd(),'pred_result')
try:
  os.mkdir(save_dir)
except FileExistsError:
  pass
results = m.evaluate(csv_file, os.path.dirname(csv_file), iou_threshold = 0.0, savedir= save_dir)
results['box_precision']

Cheers

[ethanwhite]

Hi @grand0lf - I don't immediately see anything wrong with your approach and I can confirm that the code from the docs currently produces no predictions. I have a question in with @bw4sz to see if he remembers if it ever produced predictions or if it is closer to pseudo code.

Here's a tighter version of the reprex for use as we look into this:

import os
from deepforest import main
from deepforest import get_data

m = main.deepforest(config_args={"num_classes":2}, label_dict={"Alive":0,"Dead":1})
deepforest_release_model = main.deepforest()
deepforest_release_model.use_release()
m.model.backbone.load_state_dict(deepforest_release_model.model.backbone.state_dict())
m.model.head.regression_head.load_state_dict(deepforest_release_model.model.head.regression_head.state_dict())

csv_file = get_data("testfile_multi.csv")
m.config["train"]["csv_file"] = csv_file
m.config["train"]["root_dir"] = os.path.dirname(csv_file)
m.config["train"]['epochs'] = 5
m.config["batch_size"] = 1

m.create_trainer()
m.trainer.fit(m)

predictions = m.predict_image(path = get_data("SOAP_061.png"))
predictions

Since it's only one image I ran it for 200 epochs just to confirm that it had a chance to learn and the result is the same - zero predictions for the two class model. So, there's at least a confusing doc here and possibly a bug. 500 epochs and it produced predictions. So the lack of predictions is insufficient training, which makes sense since we're only training on one image.

@grand0lf - while we look into this you can check out our multi-class training on birds, which we know works:
https://github.com/weecology/EvergladesSpeciesModel/blob/main/everglades_species.py

[ethanwhite]

Actually, it is a training length issue. I still hadn't run it long enough. At 500 epochs we get reasonable results back.

So @grand0lf, this takes me back to where I should have started, which is to ask about how many training images you have and how many epochs you're training for. I suspect that like the demo you haven't trained enough for the model to learn to detect trees in the multi-class context.

[grand0lf]

Actually, it is a training length issue. I still hadn't run it long enough. At 500 epochs we get reasonable results back.

So @grand0lf, this takes me back to where I should have started, which is to ask about how many training images you have and how many epochs you're training for. I suspect that like the demo you haven't trained enough for the model to learn to detect trees in the multi-class context.

I trained on 120 images with a total of 3000 trees and 5 classes, 10 epochs. I have more images to train on if thats beneficial.
I initially thought that my training was put on top of the original model and that the trees would be found but maybe not classed correctly but it makes sense that it returns nothing when the training isn't enough. I will train on more images and more epochs and come back with the result.

[ethanwhite]

I initially thought that my training was put on top of the original model and that the trees would be found but maybe not classed correctly but it makes sense that it returns nothing when the training isn't enough.

You're right in general, having the trained detection head definitely helps speed things up, and I am a little surprised that 120 images and 10 epochs didn't give you any predictions, but our main multi-class work (similar number of classes, but in birds) is trained on thousands of images so this is the equivalent of <1 epoch for our training in this context. If you're still not seeing any predictions with 10x more training (images x epochs) let us know and we can look at what's going on in more detail.

[grand0lf]

I initially thought that my training was put on top of the original model and that the trees would be found but maybe not classed correctly but it makes sense that it returns nothing when the training isn't enough.

You're right in general, having the trained detection head definitely helps speed things up, and I am a little surprised that 120 images and 10 epochs didn't give you any predictions, but our main multi-class work (similar number of classes, but in birds) is trained on thousands of images so this is the equivalent of <1 epoch for our training in this context. If you're still not seeing any predictions with 10x more training (images x epochs) let us know and we can look at what's going on in more detail.

I reran the training with the 120 images and 50 epochs and got a result with some (~50%) predictions for an image in the training data set - let be the species wasn't correct. I will continue with another 1000 images and hopefully the predictions will get better.

[ethanwhite]

Glad to hear it. Definitely a step in the right direction. I look forward to hearing what happens with 1000 images.