Are camera angles used by the model?

[SanBingYouYong]

Hi,

I wonder if the angle information is used in the process, either passed to the network directly or in some other ways.

I notice there's performance loss when the model gets sketches of unseen angles, and I see that the custom DatasetSketch class returns the camera angle of current sketch file, but the wrapper around it are from standard library (DataLoader and lightning trainer). Not sure if this angle information is maintained along the way or dropped after checking that there are enough files of correct angles as specified by the recipe.yml file.

Thanks!

[ofekp]

Hello and thank you for your interest,

I am not sure I fully understood the question, I will describe what I think you need to know. However, please direct me to any spot you want to better understand.
I would also like to understand if the performance degradation you speak of is in accuracy (in which case it is expected) or a computational one (i.e. something takes longer).

As you mentioned, the view angles are specified in the recipe files
https://github.com/threedle/GeoCode/blob/main/dataset_generator/recipe_files/recipe_chair.yml#L311-L318
they are split between angles that we will train on and angles that will only be used for testing.
During the dataset generation, we generate the sketches for all the angles.
For training, we only use the sketches from the angles that are meant to be trained on, and we do not train on test-only sketch angles.
https://github.com/threedle/GeoCode/blob/main/geocode/geocode_train.py#L30
During testing, we provide the dataset class with all the angles
https://github.com/threedle/GeoCode/blob/main/geocode/geocode_test.py#L91

I hope this helps, let me know if you have any further questions.

[SanBingYouYong]

Thanks for the response!

The performance issue, also as it was mentioned in the supplementary materials, refers to accuracy drops, as in the shape it reproduces match less to the original sketch.

Also regarding this issue, I'll assume it's hard for human users to draw by hands a sketch of a given angle, thus the reproduced shape will often not be ideal? In my own experiments, I don't see a significant difference between angles seen and those unseen in training tho.

Also I keep getting results that oscillate between two shapes on one sketch, and it happens in many settings (angles, sketches, etc.).

I modified the code (mainly the test process) to work on a single-image dataset where I put in my test sketches, and I wrote a Blender plugin to capture the annotation to sketches and run the inference, then it auto-fills the shape parameters. I wonder if this oscillation in results are due to minor differences on how I set up my scene or something else.

[SanBingYouYong]

This is basically how I set up my scene. I had to tweak things a bit to make the produced image similar to your dataset images, so the angle (especially the elevation -30degree down) might not be exactly the same.

[SanBingYouYong]

Here's one of my test images if you are interested in carrying out experiment on it too: -30degrees down and 15 degrees horizontally. It's generated using Freestyle engine in Blender with only simple postproc, so it might be a bit more noisy.

[ofekp]

Yes, the oscillation you are seeing is the direct result of the Random Horizontal Flip that is not disabled during the test phase:
https://github.com/threedle/GeoCode/blob/main/data/dataset_sketch.py#L37C1-L37C47
A simple if statement or removing that line will solve that issue for you.

The user will not be able to draw with the same angle exactly. Ideally, 10 or even more views should be used during training. That will make things much more resilient to unseen angles. Interestingly, using a different encoder instead of VGG did not improve this result. This further supports the need for augmentation of the camera angles during the training phase.

[SanBingYouYong]

Thanks for the explanation.

I noticed a small issue of shape not updating properly after loading params to geometry nodes, specificially in other two domains (vase and table), and it turns out the original method (toggling obj.show_bounds) might not be sufficient, and the issue can be fixed by adding one more line of obj.data.update() after calling load_shape_from_yml. Strangely enough, replacing the original update method in

GeoCode/common/bpy_util.py

Lines 134 to 135 in 01afd9f

	obj.show_bounds = not obj.show_bounds
	obj.show_bounds = not obj.show_bounds

to the latter did not solve the problem, and one would still have to call it one more time after loading params.

reference: https://blender.stackexchange.com/questions/163300/how-to-update-an-object-after-changing-its-uv-coordinates

[ofekp]

That is correct, the viewport should be updated using
bpy.context.view_layer.update()

Is this what you did - #19? Let me know and I will merge.

Thank you very much 🙏

[SanBingYouYong]

Very strange that replacing the existing method with obj.data.update() did not work, but adding one extra line of that works, in the end of the whole parameter loading process.
I'll try bpy.context.view_layer.update() later.

[SanBingYouYong]

Update:
Replacing the original code with either method did not work for me. This might actually have something to do with my extracted pipeline.
Fixing the issue with adding one more line in the end, the obj.data.update() method worked but bpy.context.view_layer.update() did not.

[ofekp]

Thank you for this @SanBingYouYong, I merged the fix with obj.data.update().

[SanBingYouYong]

Hi, now I'm trying to run a few epochs of training to see what a similar system should behave during training, but 30+ epochs passed the loss is not decreasing from 22+-1. Is it normal? Usually how many epochs would it take until we observe improvement?

[SanBingYouYong]

600 epochs passed and the loss decreased by 2, from 22 to 19. Is this expected or am I doing something wrong?

[cg-py]

The loss is exponentially decreasing, you should see a substantial decrease even after 1 epoch.
Are you running only with sketch input? what is your training command? have you verified your dataset?

[SanBingYouYong]

Dataset download link was removed, I used the dataset downloaded months ago when I was trying to do the integration. I'll manually inspect it and see what could go wrong once I get back to it.

[cg-py]

@SanBingYouYong sorry for removing the link, we will restore it eventually but there is no option to do it now.
I may consider generating the data yourself as a workaround.
Let me know if you need help with that.