Poor performance on custom dataset

[blackmrb]

Thanks for the great work~
I use 3 camera(front_far, front_left front_right), 50 frames, panda 128 lidar on my own dataset, the other config is same as default cofig.

the train and test psnr

{"split": "train", "iteration": 30000, "psnr": 31.93791029746072, "ssim": 0.9148447503123367, "lpips": 0.2528826928975289}
{"split": "test", "iteration": 30000, "psnr": 23.408180978563095, "ssim": 0.7495270156198077, "lpips": 0.3991368371579382}

the render result of test data

The reder image seem poor, and I found that the first few frames were rendered okay, but the later ones were much worse. I changed fix_radius to 10.0 and 15.0, but it seems not work. Could you please give me some advise? What parameters need to be adjusted?

[CuriousCat-7]

do you have any lidar pointcloud in your side view?

[Fumore]

Hi, it seems that the images are not captured by a pinhole camera. Our method may fail to reconstruct the scene if your images are not well rectified.

[blackmrb]

Hi, it seems that the images are not captured by a pinhole camera. Our method may fail to reconstruct the scene if your images are not well rectified.

Thanks, maybe is camera model reason. I just use front camera which is pinhole, the result seems normal.

I test another static scene only using front camera(50 frames), the test data set render is OK, but when add a 0.5m offset on the raw pose, the rendered image is very poor.

test render image (test_30000_render)

007

027

047

nvs render image, 横向y +0.5m

007

027

047

My impletion is below, is something wrong?

Step 1: Save the transform, cam_infos, and c2ws.

cam_infos_copy = copy.deepcopy(cam_infos)
c2ws_copy = copy.deepcopy(c2ws)
c2ws, transform, scale_factor = transform_poses_pca(c2ws, fix_radius=args.fix_radius)

Step 2: When generate new pose, the cam_infos is c2ws_copy , c2ws is c2ws_copy , I add 0.5m translate to raw pose, then do the same transform as transform_poses_pca()

def cameraList_from_camInfos(cam_infos, resolution_scale, args, transform=None, c2ws=None):
    camera_list = []
    for id, c in enumerate(tqdm(cam_infos)):
        if transform is not None and c2ws is not None:
            c2w_raw = c2ws[id]
            c2w_raw[1, -1] += 0.5  # y axis
            c2w = transform @ c2w_raw # same transform
            w2c = np.linalg.inv(c2w)
            c.R[:] = np.transpose(w2c[:3, :3])  # R is stored transposed due to 'glm' in CUDA code
            c.T[:] = w2c[:3, 3]
        camera_list.append(loadCam(args, id, c, resolution_scale))

    return camera_list

[blackmrb]

do you have any lidar pointcloud in your side view?

no lidar in side view in the first scene, there is a front m1 lidar. @CuriousCat-7

[CuriousCat-7]

do you have any lidar pointcloud in your side view?

no lidar in side view in the first scene, there is a front m1 lidar. @CuriousCat-7

Yeah, as far as I know, their method should perform unwell on dynamic objects when there is no lidar supervision. This is kind of a general conclusion with gaussian splatting based method. Try mono-depth's pesudo-pointcloud may alleviate this problem.