Actionness Regression not working

[makecent]

The claimed improvement from actionness regression does not seem to materialize based on my implementation using this code repository. The results with and without actionness regression are very similar.

Upon inspecting the implementation, I noticed a potential issue:

TadTR/models/tadtr.py

Lines 314 to 325 in 983ae14

	src_segments = outputs['pred_segments'].view((-1, 2))
	target_segments = torch.cat([t['segments'] for t in targets], dim=0)
	losses = {}
	iou_mat = segment_ops.segment_iou(
	segment_ops.segment_cw_to_t1t2(src_segments),
	segment_ops.segment_cw_to_t1t2(target_segments))
	gt_iou = iou_mat.max(dim=1)[0]
	pred_actionness = outputs['pred_actionness']
	loss_actionness = F.l1_loss(pred_actionness.view(-1), gt_iou.view(-1).detach())

On line 315, all target segments in the batch are concatenated, and on line 323 the maximum IoU between a predicted segment and all target segments is taken as the actionness ground truth. However, the IoUs are computed across videos, likely producing a maximum IoU between a predicted segment in video A and a ground truth segment in video B.

Even after correcting this issue, there was still no performance improvement from the actionness regression in my runs (the performance drops a lot actually). Upon my inspection, that because the actionness regression suffers from a serious label imbalance problem as most target IoUs are zero.