kwaivgi / liveportrait Goto Github PK
View Code? Open in Web Editor NEWBring portraits to life!
Home Page: https://liveportrait.github.io
License: Other
Bring portraits to life!
Home Page: https://liveportrait.github.io
License: Other
It really worked. No post processing required. Loved it.
Thanks to all
Jianzhu Guo 1† Dingyun Zhang 1,2 Xiaoqiang Liu 1 Zhizhou Zhong 1,3 Yuan Zhang 1
Pengfei Wan 1 Di Zhang 1
Would love to see the audio from reference video is stitched in output video
Is there a plan to include the sound in the driving video to be included in the Generated Video?
It would be nice if we can have talking avatars
Great work! Is it possible to inference on videos? It seems possible based on the demonstration page I viewed.
I am getting the following when performing an animate. I have done a 'git pull' to make sure I have latest files and re-ran 'pip install -r requirements.txt' after doing so.
I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\venv\lib\site-packages\torch\nn
\modules\conv.py:605: UserWarning: Plan failed with a cudnnException:
CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed
cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at
..\aten\src\ATen\native\cudnn\Conv_v8.cpp:919.)
return F.conv3d(
I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\venv\lib\site-packages\torch\nn
\modules\conv.py:456: UserWarning: Plan failed with a cudnnException:
CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed
cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at
..\aten\src\ATen\native\cudnn\Conv_v8.cpp:919.)
return F.conv2d(input, weight, bias, self.stride,
Animating... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:29
Concatenating result... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Dump to animations/s10--d5_concat.mp4
writing ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:--[swscaler @ 000001adabad0000] Warning: data is not aligned! This can lead to a speed loss
Dump to animations/s10--d5.mp4
(LivePortrait) PS J:\githubs\LivePortrait> python .\inference.py
ERROR:albumentations.check_version:Error fetching version info
Traceback (most recent call last):
File "I:\anaconda3\envs\LivePortrait\lib\site-packages\albumentations\check_version.py", line 32, in fetch_version_info
data = response.read()
File "I:\anaconda3\envs\LivePortrait\lib\http\client.py", line 476, in read
s = self.safe_read(self.length)
File "I:\anaconda3\envs\LivePortrait\lib\http\client.py", line 626, in safe_read
chunk = self.fp.read(min(amt, MAXAMOUNT))
File "I:\anaconda3\envs\LivePortrait\lib\socket.py", line 704, in readinto
return self.sock.recv_into(b)
File "I:\anaconda3\envs\LivePortrait\lib\ssl.py", line 1275, in recv_into
return self.read(nbytes, buffer)
File "I:\anaconda3\envs\LivePortrait\lib\ssl.py", line 1133, in read
return self.sslobj.read(len, buffer)
socket.timeout: The read operation timed out
Traceback (most recent call last):
File "J:\githubs\LivePortrait\inference.py", line 7, in
from src.live_portrait_pipeline import LivePortraitPipeline
File "J:\githubs\LivePortrait\src\live_portrait_pipeline.py", line 20, in
from .utils.cropper import Cropper
File "J:\githubs\LivePortrait\src\utils\cropper.py", line 10, in
from .face_analysis_diy import FaceAnalysisDIY
File "J:\githubs\LivePortrait\src\utils\face_analysis_diy.py", line 9, in
from .dependencies.insightface.app import FaceAnalysis
File "J:\githubs\LivePortrait\src\utils\dependencies\insightface_init.py", line 18, in
from . import app
File "J:\githubs\LivePortrait\src\utils\dependencies\insightface\app_init.py", line 2, in
from .mask_renderer import *
File "J:\githubs\LivePortrait\src\utils\dependencies\insightface\app\mask_renderer.py", line 8, in
from ..thirdparty import face3d
File "J:\githubs\LivePortrait\src\utils\dependencies\insightface\thirdparty\face3d_init.py", line 3, in
from . import mesh
File "J:\githubs\LivePortrait\src\utils\dependencies\insightface\thirdparty\face3d\mesh_init.py", line 9, in
from .cython import mesh_core_cython
ImportError: cannot import name 'mesh_core_cython' from 'src.utils.dependencies.insightface.thirdparty.face3d.mesh.cython' (unknown location)
anyone has this error ? how to fixed ?
hi,gays,
(D:\work\Git\LivePortrait\LivePortrait) D:\work\Git\LivePortrait>python app.py
[09:06:26] Load appearance_feature_extractor done. live_portrait_wrapper.py:29
Load motion_extractor done. live_portrait_wrapper.py:32
Load warping_module done. live_portrait_wrapper.py:35
[09:06:27] Load spade_generator done. live_portrait_wrapper.py:38
Load stitching_retargeting_module done. live_portrait_wrapper.py:42
[09:06:31] LandmarkRunner warmup time: 1.211s landmark_runner.py:89
Traceback (most recent call last):
File "D:\work\Git\LivePortrait\app.py", line 28, in
gradio_pipeline = GradioPipeline(
File "D:\work\Git\LivePortrait\src\gradio_pipeline.py", line 26, in init
super().init(inference_cfg, crop_cfg)
File "D:\work\Git\LivePortrait\src\live_portrait_pipeline.py", line 39, in init
self.cropper = Cropper(crop_cfg=crop_cfg)
File "D:\work\Git\LivePortrait\src\utils\cropper.py", line 44, in init
self.face_analysis_wrapper = FaceAnalysisDIY(
File "D:\work\Git\LivePortrait\src\utils\face_analysis_diy.py", line 37, in init
super().init(name=name, root=root, allowed_modules=allowed_modules, **kwargs)
File "D:\work\Git\LivePortrait\src\utils\dependencies\insightface\app\face_analysis.py", line 33, in init
model = model_zoo.get_model(onnx_file, **kwargs)
File "D:\work\Git\LivePortrait\src\utils\dependencies\insightface\model_zoo\model_zoo.py", line 96, in get_model
model = router.get_model(providers=providers, provider_options=provider_options)
File "D:\work\Git\LivePortrait\src\utils\dependencies\insightface\model_zoo\model_zoo.py", line 40, in get_model
session = PickableInferenceSession(self.onnx_file, **kwargs)
File "D:\work\Git\LivePortrait\src\utils\dependencies\insightface\model_zoo\model_zoo.py", line 25, in init
super().init(model_path, **kwargs)
File "D:\work\Git\LivePortrait\LivePortrait\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 419, in init
self._create_inference_session(providers, provider_options, disabled_optimizers)
File "D:\work\Git\LivePortrait\LivePortrait\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 472, in _create_inference_session
sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidProtobuf: [ONNXRuntimeError] : 7 : INVALID_PROTOBUF : Load model from D:\work\Git\LivePortrait\src\utils../../pretrained_weights/insightface\models\buffalo_l\det_10g.onnx failed:Protobuf parsing failed.
that input image + input video (both auto cropped and processed) and generates output video
ty
Missing inference code of "Portrait video editing with stitching". Can anyone provide it?
To the developers; GOOD JOB applause emoji
I followed the setup stages in the README (pip installations and downloading models from the Drive folder), and then tried to run inference.py but got this error message :
[07:53:17] Load appearance_feature_extractor done.
live_portrait_wrapper.py:29
Traceback (most recent call last):
File "/efs/user_folders/yossib/Dev/LivePortrait/inference.py", line 33, in <module>
main()
File "/efs/user_folders/yossib/Dev/LivePortrait/inference.py", line 23, in main
live_portrait_pipeline = LivePortraitPipeline(
File "/efs/user_folders/yossib/Dev/LivePortrait/src/live_portrait_pipeline.py", line 38, in __init__
self.live_portrait_wrapper: LivePortraitWrapper = LivePortraitWrapper(cfg=inference_cfg)
File "/efs/user_folders/yossib/Dev/LivePortrait/src/live_portrait_wrapper.py", line 31, in __init__
self.motion_extractor = load_model(cfg.checkpoint_M, model_config, cfg.device_id, 'motion_extractor')
File "/efs/user_folders/yossib/Dev/LivePortrait/src/utils/helper.py", line 132, in load_model
model.load_state_dict(torch.load(ckpt_path, map_location=lambda storage, loc: storage))
File "/opt/conda/envs/LivePortrait/lib/python3.9/site-packages/torch/serialization.py", line 1004, in load
with _open_zipfile_reader(opened_file) as opened_zipfile:
File "/opt/conda/envs/LivePortrait/lib/python3.9/site-packages/torch/serialization.py", line 456, in __init__
super().__init__(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory
This is how my models folder looks like :
(LivePortrait) ~/Dev/LivePortrait$ find pretrained_weights/
pretrained_weights/
pretrained_weights/insightface
pretrained_weights/insightface/models
pretrained_weights/insightface/models/buffalo_l
pretrained_weights/insightface/models/buffalo_l/det_10g.onnx
pretrained_weights/insightface/models/buffalo_l/2d106det.onnx
pretrained_weights/liveportrait
pretrained_weights/liveportrait/base_models
pretrained_weights/liveportrait/base_models/appearance_feature_extractor.pth
pretrained_weights/liveportrait/base_models/spade_generator.pth
pretrained_weights/liveportrait/base_models/motion_extractor.pth
pretrained_weights/liveportrait/base_models/warping_module.pth
pretrained_weights/liveportrait/landmark.onnx
pretrained_weights/liveportrait/retargeting_models
pretrained_weights/liveportrait/retargeting_models/stitching_retargeting_module.pth
pretrained_weights/.gitkeep
Great Work!!! Could you give more instructions on how to reproduce the demos in the project page.
Thank you for the interesting work.
I found that you used KVQ (CVPR 2023) to filter out the low-qaulity frames, and you are the author of KVQ.
Any chance to access KVQ ? or any other options such as HyperIQA ?
I've gone through the steps on the main page to install. Setup environment, install requirements and gradio as well as the models. I get the same error whether I use a conda env or python one.
When running python inference.py or python app.py I get the following:
(venv) I:\zzzz_stablediffusion-other\zz_LP\LivePortrait>python app.py
INFO:albumentations.check_version:A new version of Albumentations is available: 1.4.11 (you have 1.4.10). Upgrade using: pip install --upgrade albumentations
Traceback (most recent call last):
File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\app.py", line 11, in <module>
from src.gradio_pipeline import GradioPipeline
File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\src\gradio_pipeline.py", line 8, in <module>
from .live_portrait_pipeline import LivePortraitPipeline
File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\src\live_portrait_pipeline.py", line 20, in <module>
from .utils.cropper import Cropper
File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\src\utils\cropper.py", line 11, in <module>
from .face_analysis_diy import FaceAnalysisDIY
File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\src\utils\face_analysis_diy.py", line 9, in <module>
from .dependencies.insightface.app import FaceAnalysis
File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\src\utils\dependencies\insightface\__init__.py", line 18, in <module>
from . import app
File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\src\utils\dependencies\insightface\app\__init__.py", line 2, in <module>
from .mask_renderer import *
File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\src\utils\dependencies\insightface\app\mask_renderer.py", line 8, in <module>
from ..thirdparty import face3d
File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\src\utils\dependencies\insightface\thirdparty\face3d\__init__.py", line 3, in <module>
from . import mesh
File "I:\zzzz_stablediffusion-other\zz_LP\LivePortrait\src\utils\dependencies\insightface\thirdparty\face3d\mesh\__init__.py", line 9, in <module>
from .cython import mesh_core_cython
ImportError: cannot import name 'mesh_core_cython' from 'src.utils.dependencies.insightface.thirdparty.face3d.mesh.cython' (unknown location)
Everything was working fine before the update.
D:\AI\LivePortrait\venv\lib\site-packages\torch\nn\modules\conv.py:605: UserWarning: Plan failed with a cudnnException:
CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED
(Triggered internally at ..\aten\src\ATen\native\cudnn\Conv_v8.cpp:919.)
return F.conv3d(
Animating... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:02
Concatenating result... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Dump to animations/1111111111111111111--Gradio_3_concat.mp4
Dump to animations/1111111111111111111--Gradio_3.mp4
Traceback (most recent call last):
File "D:\AI\LivePortrait\venv\lib\site-packages\gradio\queueing.py", line 532, in process_events
response = await route_utils.call_process_api(
File "D:\AI\LivePortrait\venv\lib\site-packages\gradio\route_utils.py", line 276, in call_process_api
output = await app.get_blocks().process_api(
File "D:\AI\LivePortrait\venv\lib\site-packages\gradio\blocks.py", line 1928, in process_api
result = await self.call_function(
File "D:\AI\LivePortrait\venv\lib\site-packages\gradio\blocks.py", line 1514, in call_function
prediction = await anyio.to_thread.run_sync(
File "D:\AI\LivePortrait\venv\lib\site-packages\anyio\to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "D:\AI\LivePortrait\venv\lib\site-packages\anyio_backends_asyncio.py", line 2177, in run_sync_in_worker_thread
return await future
File "D:\AI\LivePortrait\venv\lib\site-packages\anyio_backends_asyncio.py", line 859, in run
result = context.run(func, *args)
File "D:\AI\LivePortrait\venv\lib\site-packages\gradio\utils.py", line 832, in wrapper
response = f(*args, **kwargs)
File "D:\AI\LivePortrait\src\gradio_pipeline.py", line 64, in execute_video
gr.Info("Run successfully!", duration=2)
TypeError: Info() got an unexpected keyword argument 'duration'
Just wondering if there is any hope of having this project be used to create talking avatar that is audio driven. Im having fun with this proect but it would be nice to have talking heads
face detector not detecting animals such as dogs and cats although project pages mentions it might be possible
very nice work but need something simple but big
when human move head the body make move little bit that change will make the model number one
bravo keep going....
An error was caused by the missing file: src\utils\resources\mask_template.png
步骤 1:安装Flask
首先,你需要安装Flask。可以通过pip来安装:
pip install Flask
步骤 2:创建Flask应用
创建一个新的Python脚本,例如app1.py,并添加以下代码:
`from flask import Flask, request, jsonify
import subprocess
import os
app = Flask(name)
@app.route('/run-inference', methods=['POST'])
def run_inference():
# 获取JSON请求中的数据
data = request.json
source_image = data.get('source_image')
driving_video = data.get('driving_video')
if not source_image or not driving_video:
return jsonify({"error": "Missing source_image or driving_video parameter"}), 400
# 构建命令
command = f'python inference.py -s {source_image} -d {driving_video}'
# 运行命令
try:
subprocess.run(command, check=True, shell=True, cwd='C:/ALLWEBUI/liveportarit/LivePortrait')
return jsonify({"message": "Inference completed successfully"}), 200
except subprocess.CalledProcessError as e:
return jsonify({"error": str(e)}), 500
if name == 'main':
app.run(host='0.0.0.0', port=5000)
步骤 3:运行Flask应用 在终端中运行Flask应用:
python app1.py
`
步骤 4:通过API调用
现在,你可以通过API来调用这个命令。例如,使用curl命令:
`import requests
url = 'http://localhost:5000/run-inference'
data = {
'source_image': 'Your image file here',
'driving_video': 'assets/examples/driving/d13(1).mp4'
}
response = requests.post(url, json=data)
print(response.json())
`
Hello,
I have noticed in the latest update in argument_configs.py, the gradio section, the Share is set to True. Shouldn't it be False?
Thanks!
How can I use it locally with CPU?
I have not been able to find parameter to run it from CPU and I receive the message: RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx
Thanks
我承认可灵刚出的时候说话大声了,现在又来个这个!!!!马上测试!!!!!!!
Hey, you guys are just incredibly awesome! 🌟 This is hands down the best project I've come across recently. The test results are flawlessly perfect! 📊💯 I checked out the sections on your homepage about 'Portrait video editing with stitching' and 'Generalization to Animals'. 🐶🎨 Are there any release plans for these features
Excellent work, congratulations!
I wonder if there is an option to remove the head motion completely. In my test I was driving it with a series of still
images and the head started shaking and changing its size.
Thanks in advance
Hello! The installation went well, but an error is displayed at startup:
ImportError: cannot import name 'mesh_core_cython' from 'src.utils.dependencies.insightface.thirdparty.face3d.mesh.cython' (unknown location)
Thanks for the project ❤️ I made a jupyter notebook 🥳 I hope you like it.
pip install -r requirements.txt
ERROR: Ignored the following versions that require a different python version: 1.14.0 Requires-Python >=3.10; 1.14.0rc1 Requires-Python >=3.10; 1.14.0rc2 Requires-Python >=3.10
ERROR: Could not find a version that satisfies the requirement onnxruntime-gpu==1.18.0 (from versions: none)
ERROR: No matching distribution found for onnxruntime-gpu==1.18.0
Feature request to select multiple people or a specific person within a picture would be cool
re-targeting in the gradio seems to show correct results in the example image it produces but does not change the video
I'm trying to create a colab notebook for the gradio version of the project. I have all the pieces together but for some reason when I run the final cell "python app.py" , the cell seem to get stuck and doesn't execute the rest of the script. No error pops up but a Gradio link never generates. Would anyone happen to have an idea what might cause this. Any help would be appreciated. Link to the colab - https://colab.research.google.com/drive/15pX7d8FfCMTty1v0DpgeApIR4VCduyDR?usp=sharing
as the title describes, can u explain how to make a driving video like the samples in ./assets/examples/driving
I would like to use the pet effect feature. Do you have plans to make it available?
Great work! Not really an issue but was curious if there was a way to get the torso to move a little more naturally within a portrait shot.
伟大的工作,效果很好。
https://github.com/zhaoyun0071/LivePortrait-windows-GUI
V1.2 版本:2024-7-7
1、现在驱动视频支持自动裁剪人脸区域了,默认勾选自动裁剪视频人脸区域即可;
2、修复16代显卡黑图的问题,不勾选半精度即可;
3、生成的视频和驱动视频帧率抱持一致,且带有驱动视频的声音;
4、支持pkl格式的驱动视频导入,pkl格式就是勾选自动裁剪视频人脸区域后自动生成的数据文件,和驱动视频同名同目录。
V1.1 版本:2024-7-5
1、支持批量选择图片和驱动视频。批量生成。
V1.0 版本:2024-7-4
1、基于LivePortrait项目,原汁原味,支持图片模拟驱动视频的表情、五官动作等。
I am working on a 1-click installers (all ready for Windows, RunPod, Massed Compute & Kaggle (works amazing on free Kaggle account)) and a tutorial video and flag_eye_retargeting is not working
I have made a lot of changes and implemented the logic but here comparison
Default settings as below
When target eyes-open ratio is 30% more of original and target lip-open ratio is 80% more as below
Excellent work guys! Loved reading the paper. Are there any plans to release the training code and/or the dataset?
Hello,
How can I use video2template to create a good driving video?
Thanks in advance.
How to fine tune, refer to the demonstration of fine-tuning animal faces in the paper
First of all, thank you for this great work. When I tried it in my pixel art-based images, it produced very successful results. However, I have one question: Is it possible to make it not apply filtering during resizing and perform a nearest neighbor style resize? I changed the interpolation in the cv2 resize parts, but the result did not change. I think the model is doing this smoothing internally during resize. Do you have any suggestions for this?
Im using Kijai implementation in Comfyui it would be good to mask are of the image to transfer head movement this will be good for hair, hat, helmet, etc.. here is a video example wen the tech only moves the face area and it supposed to move the helmet too.
I managed to remove wobble of shoulders by masking driving vid so only the head is visible this had a great effect.
The head on results from certain driving vids can have bad wobble caused by the head enlarging, I imagine this is due to a bad driving vid affecting the alignment crop radius, on another project I was able to set a persistent crop radius based on the most common value that worked rather well what are your thoughts on resolving this?
I don't have the driving vid for this or the settings so if anybody has all the settings checked and the concanated file and a noticable head wobble then please upload
Good work!
How can I crop a video to prepare it for inference tasks?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.