Blender

[ireneb612]

@youngwoo-yoon I have used your previous repository to extrapolate the json and wav file (Gesture generation from trimodal context). Now I followed the instructions you wrote here to get the animation on blender, but I get an Error of the Python scripr, May you give me more informations? or Help??

[youngwoo-yoon]

Could you share the error message from the Blender script? It might be on the console.

[ireneb612]

@youngwoo-yoon Hi so I solved that issue by installing a module!
Now I run the script with by putting in the temp folder the wav file and a pkl file with the generated gestures. (I obtained then with the older repository!)
And it saved me the images but they are all in the same position!
This is the output of the console:

Saved: 'C:\temp_blender\render\zkQL7UJYDIY_774_9.pkl\0252.png'
Time: 00:00.69 (Saving: 00:00.16)

*** Rendering completed ***
./temp_blender\render\zkQL7UJYDIY_774_9.pkl
./temp_blender\zkQL7UJYDIY_774_9_000_0.wav ./temp_blender\render\zkQL7UJYDIY_774_9.mov
['ffmpeg', '-r', '30', '-y', '-i', './temp_blender\render\zkQL7UJYDIY_774_9.pkl/%04d.png', '-i', './temp_blender\zkQL7UJYDIY_774_9_000_0.wav', '-vcodec', 'prores_ks', '-strict', '-2', './temp_blender\render\zkQL7UJYDIY_774_9.mov']
ffmpeg version 2021-09-22-git-447cf53774-full_build-www.gyan.dev Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 10.3.0 (Rev5, Built by MSYS2 project)
configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libdav1d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libglslang --enable-vulkan --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
libavutil 57. 6.100 / 57. 6.100
libavcodec 59. 9.100 / 59. 9.100
libavformat 59. 5.100 / 59. 5.100
libavdevice 59. 0.101 / 59. 0.101
libavfilter 8. 9.100 / 8. 9.100
libswscale 6. 1.100 / 6. 1.100
libswresample 4. 0.100 / 4. 0.100
libpostproc 56. 0.100 / 56. 0.100
[image2 @ 0000025931a7eb40] Could find no file with path './temp_blender\render\zkQL7UJYDIY_774_9.pkl/%04d.png' and index in the range 0-4
./temp_blender\render\zkQL7UJYDIY_774_9.pkl/%04d.png: No such file or directory
*** Merging audio and video completed ***
rendering time: 192.7 s, 1.3 FPS

Idk if the data that i'm giving to the script is not correct but as you have seen i cannot manage to make the SGTOOLKIT work.

Also I noticed it saved the images under C:\ directrory and then when tries to save the mov file in the current directory it does not find it, but I changed the directory and now it works.

[ireneb612]

@youngwoo-yoon
This is the first part of the output in the console

Error: File written by newer Blender binary (300.42), expect loss of data!
[]
['./temp_blender\zkQL7UJYDIY_774_9.pkl']
./temp_blender\zkQL7UJYDIY_774_9.pkl
[[ 0.06607835 -0.99775163 0.01119507]
[ 0.03074777 -0.89272014 0.44956126]
[ 0.03360709 -0.95198691 -0.3042885 ]
[-0.95043667 0.29594596 0.09532118]
[-0.5003058 0.85713853 0.12250566]
[-0.07895095 0.86926516 0.48800085]
[ 0.88876734 0.4271306 -0.16628911]
[ 0.41356524 0.90894871 -0.0526881 ]
[ 0.1484493 0.87212503 0.46621963]]
mixamorig:Spine
<Vector (0.0657, 0.0118, 0.9978)>
<Vector (-0.0657, 0.9891, 0.1320)>
mixamorig:Head
<Vector (0.0149, 0.2251, 1.4462)>
<Vector (-0.0149, 1.4462, 0.2251)>
mixamorig:HeadTop_End
<Vector (0.0337, -0.3040, 0.9521)>
<Vector (-0.0337, 0.9521, -0.3040)>
mixamorig:LeftShoulder
<Vector (-0.9504, 0.0949, -0.2962)>
<Vector (-0.0942, 0.9944, 0.0473)>
mixamorig:LeftArm
<Vector (-0.5004, 0.1228, -0.8571)>
<Vector (-0.0587, 0.5539, 0.8306)>
mixamorig:LeftForeArm
<Vector (-0.0760, 0.4863, -0.8725)>
<Vector (-0.3740, 0.0634, 0.9272)>
mixamorig:RightShoulder
<Vector (0.8890, -0.1662, -0.4267)>
<Vector (-0.1911, 0.9693, 0.1546)>
mixamorig:RightArm
<Vector (0.4134, -0.0529, -0.9090)>
<Vector (-0.1206, 0.4816, 0.8680)>
mixamorig:RightForeArm
<Vector (0.1478, 0.4672, -0.8718)>
<Vector (0.3560, 0.1331, 0.9250)>
Saved: 'C:\temp_blender\render\zkQL7UJYDIY_774_9.pkl\0000.png'
Time: 00:03.23 (Saving: 00:00.50)

Saved: 'C:\temp_blender\render\zkQL7UJYDIY_774_9.pkl\0001.png'
Time: 00:00.64 (Saving: 00:00.15)

[youngwoo-yoon]

Good to know that it's running at least. If you can share the generated pkl file, I would run Blender script on my side. So we would know which one of pkl and Blender script is wrong.

[ireneb612]

@youngwoo-yoon Hi ! thank you so much for your help. Here at this link you will find afolder named BLENDER where you can fin the temp_blender folder within which you can find the .wav , .pkl files with the addition of a .json file that i made custom from the pickle one!!

https://drive.google.com/drive/folders/1B8fy2z2TbK6WfCVrXhAFVwhfN2gSgw-w?usp=sharing

Also inside the render folder you can find some images that i get as output. I have not uploaded all of them but thay all look the same!

Pay attention that I have changed the path as you specified but i've also changed one key name of the json file since I noticed it was different, hope i did not mistake it.

Let me now!

[youngwoo-yoon]

The JSON file has wrong structure. For instance, there is "out_dir_vec" key instead of "output-data". I guess you're using the previous project (https://github.com/ai4r/Gesture-Generation-from-Trimodal-Context). The Blender script in this repo is not compatible to the previous project. Please use this repo to synthesis gestures or you might use the Blender script in the previous repo.

[youngwoo-yoon]

The JSON file has wrong structure. For instance, there is "out_dir_vec" key instead of "output-data". I guess you're using the previous project (https://github.com/ai4r/Gesture-Generation-from-Trimodal-Context). The Blender script in this repo is not compatible to the previous project. Please use this repo to synthesis gestures or you might use the Blender script in the previous repo.

[ireneb612]

@youngwoo-yoon yes I have changed that key to the correct one, yes I'm using the previous repository as specified! And thank you, I'll try one of these 2 options!

[ireneb612]

@youngwoo-yoon So I run this toolkit and I exported the json file and the wav, I run your script with my directories... It runs correctly but still all the rendered images are in the same in the pose position!

https://drive.google.com/drive/folders/1B8fy2z2TbK6WfCVrXhAFVwhfN2gSgw-w?usp=sharing

Does it work for you?

[youngwoo-yoon]

Yes, it works. Could you try with the blender file in the repository? I noticed the blender file you shared has a problem of character rigging--the character object is not attached to the bones (see the screenshot below).

[youngwoo-yoon]

This is the first frame rendered by the blender file in the repository. You can see a pose not the T pose.

[ireneb612]

@youngwoo-yoon yes! Thank you! I'm not an expert of blender but I was in rest position and I had too many activity and the bones were not attached to the armature, not it works ✈️

[ireneb612]

@youngwoo-yoon Hi! So I managed to run your blender file but I noticed it is in the reverted position! How can I make the sholders not collapse when I rotate the Armature?

[youngwoo-yoon]

Hello. Good to hear that now you can run the blender script. Do you mean that you want to render the character facing other directions? If it is, moving/rotating the view camera is the simplest solution, in my opinion.