Live-Portrait ONNX model

[aihacker111]

This is the link to HuggingFace model id and Github code demo for running project with onnx model which I convert completed 5 original models

I fixed the Grid Sample 5D error and successfully converted the Warping model via ONNX. Besides, other models all work fine and fast

Live Portrait ONNX: https://huggingface.co/myn0908/Live-Portrait-ONNX
Github repo for running my ONNX: https://github.com/aihacker111/Efficient-Live-Portrait

Thank you @cleardusk and your team for great project

[juntaosun]

Well done, awesome job! TERRIFIC! 👍👍👍

[zzzweakman]

Thank you very much for your contribution! Could you please provide the test results for the inference speed? @aihacker111

[Sunitha-Thomas]

@aihacker111
Thanks for the onnx models it worked great. But when i ran your onnx export script but not working.
torch.onnx.errors.OnnxExporterError: Unsupported: ONNX export of operator GridSample with 5D volumetric input. Please feel free to request support or submit a pull request on PyTorch GitHub: https://github.com/pytorch/pytorch/issues this error again, how to fix?

[aihacker111]

@Sunitha-Thomas It's very easy than you though, just reduce your torch version and make sure it < version 2.0.0 -> it'll work

[aihacker111]

@zzzweakman Yeah, sure bro
This is the process time when run full original pytorch models

This run on 354 frames with more than 1 hour
And This is the process time when run full onnx models

At the same number of frames and inputs, onnx model is boosting x2 faster

Noted: I'm tested all of this on Apple Silicon M1 Pro CPU and not using CUDA , not change anything

[aihacker111]

Oh for the explain of the question: Why the latest pytorch and onnx version is raising Grid Sample 5D model,
I'm checking the function convert in ONNX and seem like it's a bug of Microsoft, it's block when the Convolution pass to the Converting Function -> this is a reason that cannot converted, but in the lower version Pytorch (example: Torch 1.13.0)
Torch and ONNX lib is matching everything , seem like we have to back door

[aihacker111]

@Sunitha-Thomas already fixed Grid Example 5D in my source code , you can pull or git clone again

[juntaosun]

After testing, change to CUDA mode.
Only one “live_portraiet_onnx/onnx/warping.onnx” model failed.

# Enable CUDA mode
providers = ['CUDAExecutionProvider','CPUExecutionProvider'] 
self.w_session = ort.InferenceSession(self.warping, providers=providers)

It reported an error and the program aborted.
[ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running GridSample node. Name:'/dense_motion_network/GridSample' Status Message: Only 4-D tensor is supported

[aihacker111]

@juntaosun please show me the full scripts

[juntaosun]

@aihacker111

Only CUDA mode is enabled, other codes are the same. like this：

    def initialized_models(self):
        providers = ['CUDAExecutionProvider','CPUExecutionProvider']  
        self.m_session = ort.InferenceSession(self.motion_extractor, providers=providers)
        self.f_session = ort.InferenceSession(self.features_extractor, providers=providers)
        self.w_session = ort.InferenceSession(self.warping, providers=providers) # failed
        self.g_session = ort.InferenceSession(self.spade, providers=providers)

        self.s_session = ort.InferenceSession(self.stitch_path, providers=providers)
        self.s_l_session = ort.InferenceSession(self.lip_path, providers=providers)
        self.s_e_session = ort.InferenceSession(self.eye_path, providers=providers)

[aihacker111]

@juntaosun thank you , I'll check it later

[aihacker111]

@juntaosun only enable cuda , the warping model doesn't run ? , how about CPU

[aihacker111]

@juntaosun Thank you, I know where is the bug

[Echolink50]

Is this working properly. How do we run it? Just swap the models out or is there more to it? Thanks

[aihacker111]

@Echolink50 , git my repo , u can run it

[aihacker111]

@juntaosun hey bro , can you check and help me fix the result video is reversed with original video , something mistake in the code

[noname0121]

@aihacker111 , git my repo , u can run it
Can you give more detailed instructions with a video?
I just downloaded git clone with the address you gave and don't know what to do anymore

[aihacker111]

@noname0121 I will update readme later , please wait we need to clean all the code for easy to use

[noname0121]

@aihacker111 I will update readme later , please wait we need to clean all the code for easy to use

Thank you very much, have a nice day

[henryruhs]

@aihacker111 Author of FaceFusion here, thanks for making this public. We need more people like you in the OS scene.

[aihacker111]

@henryruhs Thank you so much, I'm really like your FaceFusion repo , it help me much during my AI engineer Journey ^_^ . I'm plan to convert to TensorRT for optimizing all of this model that help everyone can run it. Thank you

[aihacker111]

@juntaosun I'm checked relate to it

[aihacker111]

output.mp4

this output is from the option real-time webcam not list sequence frames, so clearly it's related to warping.onnx , maybe I'm setting key with wrong output, let me check

[juntaosun]

@aihacker111

The problem of animation reversal has been solved.

warping_network.py

def forward(self, feature_3d, kp_driving, kp_source):  # Parameter order

inference_portrait_onnx.py

def warp_decode(self, feature_3d, kp_source, kp_driving  

      # Please note that the position of the parameters is swapped to match the correct input
        ort_inputs = {
            self.w_input_names[0]: np.array(feature_3d),
            self.w_input_names[1]: np.array(kp_driving) ,  # Old: kp_source
            self.w_input_names[2]: np.array(kp_source)   # Old: kp_driving
        }

       outputs = self.w_session.run(self.w_output_names, ort_inputs)

Now the animation works! 😀😀😀

[aihacker111]

@juntaosun 😂😂😂😂😂 onnx's input never ceases to surprise us as long as the arrays are shuffled

[Echolink50]

IS this working on WIndows? Are you all able to use the gradio app.py from Liveportrait?

[aihacker111]

Hello @Echolink50
I and @juntaosun is still cleaning the code and will update as pull request for this project , but the inference code is working, you can try it on window, macos and linux

[Echolink50]

Ok. I am trying to install now. I tried to install just like Liveportrait but its saying things can't install in python 3.9 venv? Do I need to use python 3.10 venv?

[aihacker111]

Yeah , i’ll group code and installation as easy for everyone can use

[kitckso]

@aihacker111 Can you share you idea about your video summarization plan? Do you mean detect key frames from driving video?

[aihacker111]

Video summarization will power if we control pefectly the moment detection in this, as between 2 squence frame we can group 1 frame
We have 500 frame and then through video sum we can have 250 frames or < 500 frames
Just stop as the idea, I’ll try later
Or the plan B is use Optical flow to present the movement in video
@kitckso

[aihacker111]

As you can imagine more when we connecting Motion Director (pan left , right, zoom in, out) to get result as a marketing video with more affect as well as animal meme , Deep fake in films when the main have a drama🤣,
Rightnow I need to enhance it better , all idea i said will have someone do that

[kitckso]

@aihacker111 It seems interesting and I'm thinking how it works.

Add video summerization to detect key frames
Only process key frames
This should cause some frames not moving and maybe produce strange result, or may only works if video have many idle frames

Add video summerization to detect key frames
Detect key points on key frames
Interpolate key points on other frames
Run spade and warp on all frames as usual
This seems won't improve performance since the most time consuming parts are spade and warp, but we can't skip it on non-keyframes.

Similar to 1, but use frame interpolation like https://github.com/google-research/frame-interpolation to interpolate other frames, but I forgot the performance of frame interpolation, it could possible slower than spade and warp.

[aihacker111]

@kitckso yeah you think right , but we have many open source and idea, so we can create a own idea , just testing and know extractly what we want

[kitckso]

@aihacker111 You're right, there are millions of possibilities by combining 2 or more projects, it is interesting to explore different combinations.
Thanks for your contribution on ONNX and ideas.

[aihacker111]

@kitckso
This is my personal project: https://huggingface.co/spaces/myn0908/S2I-Artwork-Sketch-to-Image-Diffusion
My model is train on large datasets with Lora, so the result is good , but I find that the image generate related more with quality of prompt , so I’m connect Super Generate prompt by T5-XXL to my project and it work perfectly,
Overall, Nothing is impossible my friend. Thank you

[aihacker111]

#133 Everyone please follow this to get new repo for running

[warmshao]

一切正常，感谢您的贡献！❤️❤️❤️ 它现在非常快，至少比 CUDA 上的原始代码快 3 倍！👍👍👍
s7--s7_concat.mp4

我测试了下，3060显卡，CUDA11.8
ONNX+CUDA 测试275帧 154.56s
pytorch+CUDA 测试275帧 144.68s
我的怎么更慢了呢....

I'm using 3060 too. I haven't try ONNX, but I tested 2xx frames with pytorch only takes 18s, seems your speed is abnormal.

really, ~70ms/frame, how to do ?

[kitckso]

@warmshao Sorry that it's not accurate, it is the time for "animating" process, i.e. spade and warp. My video is 9s, fps is roughly 25, so frame number should be 2xx, I remember the whole process is around 30s. Roughly 3s to process 1s video.

I tested in Linux docker container.
Maybe CPU is also a factor, I found one CPU core is 100% loading.

[Echolink50]

@warmshao Sorry that it's not accurate, it is the time for "animating" process, i.e. spade and warp. My video is 9s, fps is roughly 25, so frame number should be 2xx, I remember the whole process is around 30s. Roughly 3s to process 1s video.

I tested in Linux docker container. Maybe CPU is also a factor, I found one CPU core is 100% loading.

I also get a about 3 seconds to process each 1 second of video with the main git. I still have yet to get the ONNX branch to use GPU.

[MarsEverythingTech]

Hello,

Is the ONNX feature integrated into GPU and uses CUDA?

Thanks

[aihacker111]

@MarsEverythingTech not yet, in fixing progress, wait some day Thank you

[aihacker111]

ONNX feature updated and next plan in the next week right here: #142

[warmshao]

Both TensorRT and ONNXRUNTIME-GPU have been implemented in my repository https://github.com/warmshao/FasterLivePortrait. Both have improved speed, especially TensorRT which can achieve real-time rendering a frame at 30+ FPS. Feel free to enjoy it!

[chrisbward]

thanks @aihacker111 and @warmshao - any details on how we can replicate the onnx models, converting from the original files?

[warmshao]

thanks @aihacker111 and @warmshao - any details on how we can replicate the onnx models, converting from the original files?

my script to export onnx: https://github.com/warmshao/LivePortrait/blob/multi_faces/export_onnx.py

[JimLee4530]

thanks @aihacker111 and @warmshao - any details on how we can replicate the onnx models, converting from the original files?

my script to export onnx: https://github.com/warmshao/LivePortrait/blob/multi_faces/export_onnx.py

i use your script to export onnx, but raise error
"""
torch.onnx.errors.OnnxExporterError: Unsupported: ONNX export of operator GridSample with 5D volumetric input. Please feel free to request support or submit a pull request on PyTorch GitHub: https://github.com/pytorch/pytorch/issues
"""
Can you give me the environment configuration ?

[aihacker111]

@JimLee4530 Upgrade your pytorch version with nightly preview
pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121

Then you change the opset version from 16 to 20

[warmshao]

thanks @aihacker111 and @warmshao - any details on how we can replicate the onnx models, converting from the original files?

my script to export onnx: https://github.com/warmshao/LivePortrait/blob/multi_faces/export_onnx.py

i use your script to export onnx, but raise error """ torch.onnx.errors.OnnxExporterError: Unsupported: ONNX export of operator GridSample with 5D volumetric input. Please feel free to request support or submit a pull request on PyTorch GitHub: https://github.com/pytorch/pytorch/issues """ Can you give me the environment configuration ?

pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121

[JimLee4530]

thanks @aihacker111 and @warmshao - any details on how we can replicate the onnx models, converting from the original files?

my script to export onnx: https://github.com/warmshao/LivePortrait/blob/multi_faces/export_onnx.py

i use your script to export onnx, but raise error """ torch.onnx.errors.OnnxExporterError: Unsupported: ONNX export of operator GridSample with 5D volumetric input. Please feel free to request support or submit a pull request on PyTorch GitHub: https://github.com/pytorch/pytorch/issues """ Can you give me the environment configuration ?

pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121

thank you very much @warmshao @aihacker111
pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121

And change the opset version from 16 to 20 is work!

[warmshao]

Install-free, extract-and-play Windows package with TensorRT support now available! please refer FasterLivePortrait change logs, really convenient and very fast!!!

[ChunChenLin]

This is the link to HuggingFace model id and Github code demo for running project with onnx model which I convert completed 5 original models

I fixed the Grid Sample 5D error and successfully converted the Warping model via ONNX. Besides, other models all work fine and fast

Live Portrait ONNX: https://huggingface.co/myn0908/Live-Portrait-ONNX Github repo for running my ONNX: https://github.com/aihacker111/Efficient-Live-Portrait

Thank you @cleardusk and your team for great project

@aihacker111 May I ask how did you fix GridSample issue in warping module? Currently, I encounter the below error during the inference with ONNX model. Many thanks. :))

onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from models/warping_module.onnx failed:Node (/dense_motion_network/GridSample) Op (GridSample) [ShapeInferenceError] Input 0 expected to have rank 4 but has rank 5