Video Datamodules

[djdameln]

Description

• Adds video datamodules.
• Introduces VideoAnomalibDataset class, which serves as the base class for all video datasets.
• Adds UCSDped and Avenue datasets as first video anomaly datasets.
• The VideoAnomalibDataset contains a samples dataframe, similar to the AnomalibDataset, but it lists the video files instead of the image files. This helps ensure that frames from the same video file always stay together in the same subset after splitting or subsampling.
• The central component of the VideoAnomalibDataset is a new entity called the ClipsIndexer. This class is based on torchvision's VideoClips, and creates a list of consecutive sub-videos (referred to as clips). When the indexer object is instantiated, it indexes all clips that can be extracted from the videos in the dataset, so that the ith clip in the dataset can be retrieved by calling clips_indexer.get_item(i). When the dataset is a validation or test set, the output of the ClipsIndexer also contains a ground truth mask for each frame in the clip.
• The number of frames in each clip and the number of frames between consecutive clips can be configured using the constructor arguments clip_length_in_frames and frames_between_clips, which are also exposed to the config.yaml. In the special case where clip_length_in_frames is set to 1, the video dataset behaves like an image dataset, which allows us to use the dataset with any of the current Anomalib models.
• Since different datasets use different file structures for storing the masks and videos, a subclass which inherits from ClipsIndexer will need to be implemented for every new dataset that gets added to the library. The concrete subclass should implement at least the get_masks method, which defines how a mask is retrieved from the file system given the index of the video and the indices of the frames within the video. If the videos of the dataset are not stored in standard video files (e.g. the videos are folders of images where each image is a single video frame), the subclass should also override get_clips and _compute_frame_pts (see UCSDpedClipsIndexer for an example).

Known issues

• Visualization only supported for single-frame, and only as separate image files (as opposed to re-composing the video).
• No inference support.

Example configs

dataset:
  name: ucsd
  format: ucsdped
  path: ./datasets/ucsdped
  category: UCSDped2
  task: segmentation
  clip_length_in_frames: 1
  frames_between_clips: 1
  image_size: 256
  train_batch_size: 32
  eval_batch_size: 32
  num_workers: 8
  transform_config:
    train: null
    eval: null
  val_split_mode: same_as_test # options: [same_as_test, from_test]
  tiling:
    apply: false
    tile_size: null
    stride: null
    remove_border_count: 0
    use_random_tiling: False
    random_tile_count: 16

dataset:
  name: avenue
  format: avenue
  path: ./datasets/avenue
  gt_dir: ./datasets/avenue/ground_truth_demo
  task: segmentation
  clip_length_in_frames: 1
  frames_between_clips: 10
  image_size: 256
  train_batch_size: 32
  eval_batch_size: 32
  num_workers: 8
  transform_config:
    train: null
    eval: null
  val_split_mode: same_as_test # options: [same_as_test, from_test]
  tiling:
    apply: false
    tile_size: null
    stride: null
    remove_border_count: 0
    use_random_tiling: False
    random_tile_count: 16

Changes

• Bug fix (non-breaking change which fixes an issue)
• Refactor (non-breaking change which refactors the code base)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist

• My code follows the pre-commit style and check guidelines of this project.
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing tests pass locally with my changes

[ashwinvaidya17]

I have the same comment but not sure if github is updating on my browser. I cannot see the new comments you have added on the web ui but I can see them in the notification email.

[ashwinvaidya17]

Thanks!