LoadMultiViewImageFromFiles fails in data pipeline

[SimonDoll]

Shape error in mmdet3d/datasets/pipelines/formating.py when loading multi view images.
Line 50: img = np.ascontiguousarray(results['img'].transpose(2, 0, 1))
For multiview images results['img'] is of shape (height, width, 3 (rgb), 6 (multi view))
The transpose call fails with ValueError: axes don't match array due to the last axis.

Reproduction

1. What command or script did you run?

python tools/test.py config checkpoint --eval mAp

I added loading multi view images to the default nuscenes config (train and test pipeline):

dict(
        type='LoadMultiViewImageFromFiles'
    ),

Environment
sys.platform: linux
Python: 3.7.7 (default, May 7 2020, 21:25:33) [GCC 7.3.0]
CUDA available: True
GPU 0: GeForce RTX 2070 SUPER
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 10.1, V10.1.243
GCC: gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
PyTorch: 1.6.0
PyTorch compiling details: PyTorch built with:

• GCC 7.3
• C++ Version: 201402
• Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v1.5.0 (Git Hash e2ac1fac44c5078ca927cb9b90e1b3066a0b2ed0)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 10.1
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
• CuDNN 7.6.3
• Magma 2.5.2
• Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

TorchVision: 0.7.0
OpenCV: 4.4.0
MMCV: 1.2.1
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 10.1
MMDetection: 2.7.0
MMDetection3D: 0.8.0+e5a155e

Error traceback

Original Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 185, in _worker_loop
    data = fetcher.fetch(index)
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/tmp/mmdetection3d_module/mmdet3d/datasets/custom_3d.py", line 292, in __getitem__
    return self.prepare_test_data(idx)
  File "/tmp/mmdetection3d_module/mmdet3d/datasets/custom_3d.py", line 168, in prepare_test_data
    example = self.pipeline(input_dict)
  File "/opt/conda/lib/python3.7/site-packages/mmdet/datasets/pipelines/compose.py", line 40, in __call__
    data = t(data)
  File "/tmp/mmdetection3d_module/mmdet3d/datasets/pipelines/test_time_aug.py", line 103, in __call__
    data = self.transforms(_results)
  File "/opt/conda/lib/python3.7/site-packages/mmdet/datasets/pipelines/compose.py", line 40, in __call__
    data = t(data)
  File "/tmp/mmdetection3d_module/mmdet3d/datasets/pipelines/formating.py", line 254, in __call__
    results = super(DefaultFormatBundle3D, self).__call__(results)
  File "/tmp/mmdetection3d_module/mmdet3d/datasets/pipelines/formating.py", line 51, in __call__
    img = np.ascontiguousarray(results['img'].transpose(2, 0, 1))
ValueError: axes don't match array

Bug fix
The transpose call fails with ValueError: axes don't match array due to the last axis. I'm new to the framework and do not know wether fixing this by simply leaving the last dimension for the multi view images is enough:

Line 50: img = np.ascontiguousarray(results['img'].transpose(2, 0, 1, 3))

As a hotfix this definitively works out for me...
Is this function called for 3D arrays as well?

I'm happy to do a pull request if You tell me wether we need different cases for single view (3 dimensions) vs multi view (4 dimensions) arrays and if using transpose is enough or if I should do the same with swapaxes to be invariant to the array shape..
Thanks for any help 👍

[ZwwWayne]

Hi @SimonDoll ,
I am a little bit confused with your bug. If the current frame contains one image (e.g., KITTI dataset), then this line of code should be fine. If the current frame contains more than one image (e.g., 6 images in nuScenes dataset), then results['img'] is a list and will not go through line 50. But it seems that multiple images are concatenated in results['img'] in your code.

[SimonDoll]

Hi @ZwwWayne, thanks for Your fast response 👍
Sorry for the confusion.. if You want to debug, this can be reproduced by using the default nuscenes config as a base and only adding use_camera=True and the image loading step to test and train pipelines

dict(
        type='LoadMultiViewImageFromFiles'
)

It seems as if results['img'] in formatting.py is of type numpy.ndarray therefore the list case does not match, shape for nuscenes is (900, 1600, 3, 6).
Am I doing something wrong here regarding the image loading?

[alexa19lab]

Hi @SimonDoll ,
I am a little bit confused with your bug. If the current frame contains one image (e.g., KITTI dataset), then this line of code should be fine. If the current frame contains more than one image (e.g., 6 images in nuScenes dataset), then results['img'] is a list and will not go through line 50. But it seems that multiple images are concatenated in results['img'] in your code.

Hi @ZwwWayne, I think when use LoadMultiViewImageFromFiles in config, the images are stacked in mmdetection3d_deeproute_extention/mmdet3d/datasets/pipelines/loading.py, line 47,

img = np.stack(
            [mmcv.imread(name, self.color_type) for name in filename], axis=-1)

and is assigned to results['img'] in line 52.

So apparently results['img'] would not be a list after this operation

[Wuziyi616]

This issue is fixed by #611