Error during training

[oguzhannysr]

@bw4sz ,Hello, I wanted to run the train example given in colab with my own data, but I get an error before training. Can you help me? Also, how can I train using all 15 of my differently labeled images (400*400px)? I saw that the example given in colab is an example made for a single image.

#initial the model and change the corresponding config file
m = main.deepforest()
m.config['gpus'] = '-1' #move to GPU and use all the GPU resources
m.config["train"]["csv_file"] = annotations_file
m.config["train"]["root_dir"] = os.path.dirname(annotations_file)
m.config["score_thresh"] = 0.4
m.config["train"]['epochs'] = 2
m.config["validation"]["csv_file"] = validation_file
m.config["validation"]["root_dir"] = os.path.dirname(validation_file)
#create a pytorch lighting trainer used to training
m.create_trainer()
#load the lastest release model
m.use_release()
start_time = time.time()
m.trainer.fit(m)
print(f"--- Training on GPU: {(time.time() - start_time):.2f} seconds ---")

INFO:pytorch_lightning.accelerators.cuda:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
/usr/local/lib/python3.10/dist-packages/torch/optim/lr_scheduler.py:28: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate.
  warnings.warn("The verbose parameter is deprecated. Please use get_last_lr() "
INFO:pytorch_lightning.callbacks.model_summary:
  | Name       | Type                  | Params | Mode 
-------------------------------------------------------------
0 | model      | RetinaNet             | 32.1 M | train
1 | iou_metric | IntersectionOverUnion | 0      | train
2 | mAP_metric | MeanAveragePrecision  | 0      | train
-------------------------------------------------------------
31.9 M    Trainable params
222 K     Non-trainable params
32.1 M    Total params
128.592   Total estimated model params size (MB)
204       Modules in train mode
0         Modules in eval mode
/usr/local/lib/python3.10/dist-packages/albumentations/core/composition.py:161: UserWarning: Got processor for bboxes, but no transform to process it.
  self._set_keys()
/usr/local/lib/python3.10/dist-packages/pytorch_lightning/utilities/data.py:105: Total length of `DataLoader` across ranks is zero. Please make sure this was your intention.
/usr/lib/python3.10/multiprocessing/popen_fork.py:66: RuntimeWarning: os.fork() was called. os.fork() is incompatible with multithreaded code, and JAX is multithreaded, so this will likely lead to a deadlock.
  self.pid = os.fork()
/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/fit_loop.py:298: The number of training batches (1) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
Epoch 0: 100%
 1/1 [00:00<00:00,  4.70it/s, v_num=3]
/usr/lib/python3.10/multiprocessing/popen_fork.py:66: RuntimeWarning: os.fork() was called. os.fork() is incompatible with multithreaded code, and JAX is multithreaded, so this will likely lead to a deadlock.
  self.pid = os.fork()
---------------------------------------------------------------------------
MisconfigurationException                 Traceback (most recent call last)
[<ipython-input-25-e367a6df930f>](https://localhost:8080/#) in <cell line: 2>()
      1 start_time = time.time()
----> 2 m.trainer.fit(m)
      3 print(f"--- Training on GPU: {(time.time() - start_time):.2f} seconds ---")

8 frames
[/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/training_epoch_loop.py](https://localhost:8080/#) in _update_learning_rates(self, interval, update_plateau_schedulers)
    382                         if config.strict:
    383                             avail_metrics = list(trainer.callback_metrics)
--> 384                             raise MisconfigurationException(
    385                                 f"ReduceLROnPlateau conditioned on metric {monitor_key}"
    386                                 f" which is not available. Available metrics are: {avail_metrics}."

MisconfigurationException: ReduceLROnPlateau conditioned on metric val_classification which is not available. Available metrics are: ['lr-SGD']. Condition can be set using `monitor` key in lr scheduler dict

https://colab.research.google.com/drive/1gKUiocwfCvcvVfiKzAaf6voiUVL2KK_r?usp=sharing

[ethanwhite]

Hi @oguzhannysr - can you share one of your images and associated label files? Since the code runs successfully on the built-in datasets it means there is something about your data that doesn't match our expectations and its hard to help without being above to see the data.

[oguzhannysr]

labeled.zip
@ethanwhite ,Of course, I share the examples in zip.

[bw4sz]

Can you print the config file so I can understand why the val_classification metric does not exist. I sense this is due to poorly formatted bounding box in the validation dataset. Show me the contents of the validation and annotation files, in the .zip you sent there is only one csv "output_labels_colab.csv".