4D semantic segmentation on Synthia Dataset model code

[sandeepnmenon]

Is it possible to release the 4D semantic segmentation on Synthia Dataset model code.

Thank you

[sandeepnmenon]

Hi @hehefan
Continuing the conversation for 4D semantic segmentation from the PSTNet repo.

As mentioned in this paper,

"After the transformer,
because the point 4D convolution layers subsample points,
we add feature propagation layers to interpolate point features. Inspired by [43], we use inverse distance weighted
average based on k nearest neighbors:"

Is

P4Transformer/modules-pytorch-1.4.0/point_4d_convolution.py

Line 182 in 0d813eb

class P4DTransConv(nn.Module):

used for that purpose ?

Also, how many P4DConv layers were used to subsample the points before feeding into the transfomer?

It will be really helpful if you could share the model code used in the paper.
Thank you

[hehefan]

Hi @sandeepnmenon,

Sorry for my late reply. I have uploaded the segmentation model under the models folder.

Best regards.

[sandeepnmenon]

Thank you @hehefan
If its not too much trouble, can you upload the dataloader and training code for synthia as well.

[hehefan]

Hi @sandeepnmenon,

I have uploaded the codes. Hopefully, they are useful.

Best regards.

[sandeepnmenon]

Thank you @hehefan . These are of great help.
Wanted to understand the logic behind the data pre-processing steps.

1. Why are the frames in a sequence divided by the center z into two halves before passing to the model and for the loss propagation ?
2. Why is the point order shuffled?

[sandeepnmenon]

Hi @hehefan

I had a few more doubts as to the data processing part for this.
I see for one particular sequence, all the point clouds are loaded(stacked) and passed to the model at one go. What if you have really long sequences of dense point clouds.
For example, I am dealing with point cloud sequences with a total of approx. 170M points spread over 300 frames. Can this pipeline and model handle that kind of data?

My gpu is 24 GB Tesla m40

Thank you

[hehefan]

Hi @sandeepnmenon,

This model cannot handle so many points. I have a few suggestions.

1. Point-wise segmentation is different from video-level classification. There is no need to process 300 frames one time. Therefore, you can split 300 frames into several video clips.
2. For each frame, you can split the space into several subspaces. Moreover, for each subspace, you can further randomly decompose the points into several parts or groups. In this way, you can handle the massive points.

Actually, handling so many points is challenging, even for a single frame. The model can process about 8x16384x3 points one time with 24GB memory.

Best regards.

[sandeepnmenon]

Thank you @hehefan for all your insights.
Would really appreciate if I could reach out to you for further queries.

Closing the issue now.