sdbds / magic-animate-for-windows

MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model

MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model

Zhongcong Xu · Jianfeng Zhang · Jun Hao Liew · Hanshu Yan · Jia-Wei Liu · Chenxu Zhang · Jiashi Feng · Mike Zheng Shou


National University of Singapore   |   ByteDance

📢 News

• [2023.12.8] Add loading local safetensors or ckpt,you can change config/prompts/animation.yaml about pretrained_model_path for your local SD1.5 model. such as "D:\\stablediffusion-webui\\models\Stable-diffusion\\v1-5-pruned.ckpt"
• [2023.12.4] Release inference code and gradio demo. We are working to improve MagicAnimate, stay tuned!
• [2023.11.23] Release MagicAnimate paper and project page.

⚒️ Installation

prerequisites: python>=3.8, CUDA>=11.3, ffmpeg and git.

Python and Git:

• Python 3.10.11: https://www.python.org/ftp/python/3.10.11/python-3.10.11-amd64.exe

• git: https://git-scm.com/download/win

• Install ffmpeg for your operating system (https://www.geeksforgeeks.org/how-to-install-ffmpeg-on-windows/)

  notice:step 4 use windows system Set Enviroment Path.

Give unrestricted script access to powershell so venv can work:

• Open an administrator powershell window
• Type Set-ExecutionPolicy Unrestricted and answer A
• Close admin powershell window

git clone --recurse-submodules https://github.com/sdbds/magic-animate-for-windows/

Install with Powershell run install.ps1 or install-cn.ps1(for Chinese)

Use local model

Add loading local safetensors or ckpt,you can change config/prompts/animation.yaml about pretrained_model_path for your local SD1.5 model. such as "D:\\stablediffusion-webui\\models\Stable-diffusion\\v1-5-pruned.ckpt"

🎨 Gradio Demo

Online Gradio Demo:

Try our online gradio demo quickly.

Local Gradio Demo:

Launch local gradio demo on single GPU:

Powershell run with run_gui.ps1

Launch local gradio demo if you have multiple GPUs:

Edit run_gui.ps1 set $mutil_gpu=1 then run.

Then open gradio demo in local browser.

🙏 Acknowledgements

We would like to thank AK(@_akhaliq) and huggingface team for the help of setting up oneline gradio demo.

🎓 Citation

If you find this codebase useful for your research, please use the following entry.

@inproceedings{xu2023magicanimate,
    author    = {Xu, Zhongcong and Zhang, Jianfeng and Liew, Jun Hao and Yan, Hanshu and Liu, Jia-Wei and Zhang, Chenxu and Feng, Jiashi and Shou, Mike Zheng},
    title     = {MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model},
    booktitle = {arXiv},
    year      = {2023}
}

magic-animate-for-windows's Issues

运行后添加任何视频都显示“错误”

安装运行成功，打开生成的网页后，无论添加什么视频都显示“错误”

运行显示的代码如下（有点长）
————————————————————————————————————————
安全警告
请只运行你信任的脚本。虽然来自 Internet 的脚本会有一定的用处，但此脚本可能会损坏你的计算机。如果你信任此脚本，请使用
Unblock-File cmdlet 允许运行该脚本，而不显示此警告消息。是否要运行 E:\magic-animate-for-windows-2.1\run_gui.ps1?
[D] 不运行(D) [R] 运行一次(R) [S] 暂停(S) [?] 帮助 (默认值为“D”): r
E:\magic-animate-for-windows-2.1\magicanimate\pipelines\pipeline_animation.py:43: FutureWarning: Importing DiffusionPipeline or ImagePipelineOutput from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead.
from diffusers.pipeline_utils import DiffusionPipeline
Initializing MagicAnimate Pipeline...

missing keys: 1246;

unexpected keys: 51;

The config attributes {'addition_embed_type': None, 'addition_embed_type_num_heads': 64, 'addition_time_embed_dim': None, 'conditioning_channels': 3, 'encoder_hid_dim': None, 'encoder_hid_dim_type': None, 'global_pool_conditions': False, 'num_attention_heads': None, 'transformer_layers_per_block': 1} were passed to ControlNetModel, but are not expected and will be ignored. Please verify your config.json configuration file.
It is recommended to provide attention_head_dim when calling get_down_block. Defaulting attention_head_dim to 8.
It is recommended to provide attention_head_dim when calling get_down_block. Defaulting attention_head_dim to 8.
It is recommended to provide attention_head_dim when calling get_down_block. Defaulting attention_head_dim to 8.
It is recommended to provide attention_head_dim when calling get_down_block. Defaulting attention_head_dim to 8.
Using Densepose ControlNet
E:\magic-animate-for-windows-2.1\magicanimate\pipelines\pipeline_animation.py:103: FutureWarning: The configuration file of this scheduler: DDIMScheduler {
"_class_name": "DDIMScheduler",
"_diffusers_version": "0.21.4",
"beta_end": 0.012,
"beta_schedule": "linear",
"beta_start": 0.00085,
"clip_sample": true,
"clip_sample_range": 1.0,
"dynamic_thresholding_ratio": 0.995,
"num_train_timesteps": 1000,
"prediction_type": "epsilon",
"rescale_betas_zero_snr": false,
"sample_max_value": 1.0,
"set_alpha_to_one": true,
"steps_offset": 0,
"thresholding": false,
"timestep_spacing": "leading",
"trained_betas": null
}
is outdated. steps_offset should be set to 1 instead of 0. Please make sure to update the config accordingly as leaving steps_offset might led to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the scheduler/scheduler_config.json file
deprecate("steps_offset!=1", "1.0.0", deprecation_message, standard_warn=False)
E:\magic-animate-for-windows-2.1\magicanimate\pipelines\pipeline_animation.py:116: FutureWarning: The configuration file of this scheduler: DDIMScheduler {
"_class_name": "DDIMScheduler",
"_diffusers_version": "0.21.4",
"beta_end": 0.012,
"beta_schedule": "linear",
"beta_start": 0.00085,
"clip_sample": true,
"clip_sample_range": 1.0,
"dynamic_thresholding_ratio": 0.995,
"num_train_timesteps": 1000,
"prediction_type": "epsilon",
"rescale_betas_zero_snr": false,
"sample_max_value": 1.0,
"set_alpha_to_one": true,
"steps_offset": 1,
"thresholding": false,
"timestep_spacing": "leading",
"trained_betas": null
}
has not set the configuration clip_sample. clip_sample should be set to False in the configuration file. Please make sure to update the config accordingly as not setting clip_sample in the config might lead to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the scheduler/scheduler_config.json file
deprecate("clip_sample not set", "1.0.0", deprecation_message, standard_warn=False)
Initialization Done!
Running on local URL: http://127.0.0.1:7860
Running on public URL: https://aa63f8e33c20cd50dd.gradio.live

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run gradio deploy from Terminal to deploy to Spaces (https://huggingface.co/spaces)
Exception in callback _ProactorBasePipeTransport._call_connection_lost(None)
handle: <Handle _ProactorBasePipeTransport._call_connection_lost(None)>
Traceback (most recent call last):
File "C:\Users\CJZ\AppData\Local\Programs\Python\Python310\lib\asyncio\events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "C:\Users\CJZ\AppData\Local\Programs\Python\Python310\lib\asyncio\proactor_events.py", line 162, in _call_connection_lost
self._sock.shutdown(socket.SHUT_RDWR)
ConnectionResetError: [WinError 10054] 远程主机强迫关闭了一个现有的连接。
Traceback (most recent call last):
File "E:\magic-animate-for-windows-2.1\venv\lib\site-packages\gradio\queueing.py", line 456, in call_prediction
output = await route_utils.call_process_api(
File "E:\magic-animate-for-windows-2.1\venv\lib\site-packages\gradio\route_utils.py", line 232, in call_process_api
output = await app.get_blocks().process_api(
File "E:\magic-animate-for-windows-2.1\venv\lib\site-packages\gradio\blocks.py", line 1520, in process_api
inputs = self.preprocess_data(fn_index, inputs, state)
File "E:\magic-animate-for-windows-2.1\venv\lib\site-packages\gradio\blocks.py", line 1303, in preprocess_data
processed_input.append(block.preprocess(inputs_cached))
File "E:\magic-animate-for-windows-2.1\venv\lib\site-packages\gradio\components\video.py", line 168, in preprocess
duration = processing_utils.get_video_length(file_name)
File "E:\magic-animate-for-windows-2.1\venv\lib\site-packages\gradio\processing_utils.py", line 725, in get_video_length
duration = subprocess.check_output(
File "C:\Users\CJZ\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 420, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "C:\Users\CJZ\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 501, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\CJZ\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 966, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\CJZ\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 1435, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] 系统找不到指定的文件。
Traceback (most recent call last):
File "E:\magic-animate-for-windows-2.1\venv\lib\site-packages\gradio\queueing.py", line 456, in call_prediction
output = await route_utils.call_process_api(
File "E:\magic-animate-for-windows-2.1\venv\lib\site-packages\gradio\route_utils.py", line 232, in call_process_api
output = await app.get_blocks().process_api(
File "E:\magic-animate-for-windows-2.1\venv\lib\site-packages\gradio\blocks.py", line 1520, in process_api
inputs = self.preprocess_data(fn_index, inputs, state)
File "E:\magic-animate-for-windows-2.1\venv\lib\site-packages\gradio\blocks.py", line 1303, in preprocess_data
processed_input.append(block.preprocess(inputs_cached))
File "E:\magic-animate-for-windows-2.1\venv\lib\site-packages\gradio\components\video.py", line 168, in preprocess
duration = processing_utils.get_video_length(file_name)
File "E:\magic-animate-for-windows-2.1\venv\lib\site-packages\gradio\processing_utils.py", line 725, in get_video_length
duration = subprocess.check_output(
File "C:\Users\CJZ\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 420, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "C:\Users\CJZ\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 501, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\CJZ\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 966, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\CJZ\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 1435, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] 系统找不到指定的文件。

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "E:\magic-animate-for-windows-2.1\venv\lib\site-packages\gradio\queueing.py", line 501, in process_events
response = await self.call_prediction(awake_events, batch)
File "E:\magic-animate-for-windows-2.1\venv\lib\site-packages\gradio\queueing.py", line 465, in call_prediction
raise Exception(str(error) if show_error else None) from error
Exception: None

生成的速度太慢了

显卡是Gtx3060, 12GB显存，使用提供的demo。
如下图所示。

速度如下：

请问是我显卡不行吗？不至于这么慢吧。

Nice job thanks! Add an option to have just the final rendered video?

Title explains it all

pls advise, when i run install_cn.ps1, get this error message

Installing collected packages: tqdm, requests, protobuf
Attempting uninstall: tqdm
Found existing installation: tqdm 4.65.0
Uninstalling tqdm-4.65.0:
Successfully uninstalled tqdm-4.65.0
Attempting uninstall: requests
Found existing installation: requests 2.28.2
Uninstalling requests-2.28.2:
Successfully uninstalled requests-2.28.2
Attempting uninstall: protobuf
Found existing installation: protobuf 3.20.3
Uninstalling protobuf-3.20.3:
Successfully uninstalled protobuf-3.20.3
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
openxlab 0.0.32 requires requests~=2.28.2, but you have requests 2.31.0 which is incompatible.
openxlab 0.0.32 requires tqdm~=4.65.0, but you have tqdm 4.66.1 which is incompatible.
mediapipe 0.10.9 requires protobuf<4,>=3.11, but you have protobuf 4.24.2 which is incompatible.

should i pip install requests==2.28.2 tqdm==4.65.0 protobuf<4,>=3.11 as the error message promted?

Failed to load tokenizer

Hi there,

I have set pretrained_model_path in animation.yaml to the full directory to v1-5-pruned.ckpt. When I run ./run_gui.ps1, I get this error:
ValueError: Calling CLIPTokenizer.from_pretrained() with the path to a single file or url is not supported for this tokenizer. Use a model identifier or the path to a directory instead.

Any help would be appreciated.

Thank you,
Jonah

how to fix after python -m demo.gradio_animate

NEXT ISSUE FROM magic-research#50

CUDA SDK 11.8 + ENV latest already installed
CUDA NCNN 8.9+ ENV latest already installed
FFMPEG 6 + ENV already installed

unet/config.json from https://huggingface.co/runwayml/stable-diffusion-v1-5/blob/main/unet/config.json already exist

create pretrained_models
git lfs clone https://huggingface.co/zcxu-eric/MagicAnimate
git lfs clone https://huggingface.co/stabilityai/sd-vae-ft-mse
git lfs clone https://huggingface.co/runwayml/stable-diffusion-v1-5

conda create -n m_animate_for_win python=3.10 -y
conda activate m_animate_for_win
pip install -r requirements-windows.txt
python -m demo.gradio_animate  <<<< has issue

(m_animate_for_win) V:\_ANIMATION\MAGIC_ANIMATE\magic-animate-for-windows>python -m demo.gradio_animate
V:\_ANIMATION\MAGIC_ANIMATE\magic-animate-for-windows\magicanimate\pipelines\pipeline_animation.py:43: FutureWarning: Importing `DiffusionPipeline` or `ImagePipelineOutput` from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead.
  from diffusers.pipeline_utils import DiffusionPipeline
Initializing MagicAnimate Pipeline...
loaded temporal unet's pretrained weights from pretrained_models/stable-diffusion-v1-5\unet ...
### missing keys: 560;
### unexpected keys: 0;
### Temporal Module Parameters: 417.1376 M
Traceback (most recent call last):
  File "C:\Users\NGXCRYPT-2ND\.conda\envs\m_animate_for_win\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\NGXCRYPT-2ND\.conda\envs\m_animate_for_win\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "V:\_ANIMATION\MAGIC_ANIMATE\magic-animate-for-windows\demo\gradio_animate.py", line 19, in <module>
    animator = MagicAnimate()
  File "V:\_ANIMATION\MAGIC_ANIMATE\magic-animate-for-windows\demo\animate.py", line 70, in __init__
    vae = AutoencoderKL.from_pretrained(config.pretrained_model_path, subfolder="vae")
  File "C:\Users\NGXCRYPT-2ND\.conda\envs\m_animate_for_win\lib\site-packages\diffusers\models\modeling_utils.py", line 553, in from_pretrained
    config, unused_kwargs, commit_hash = cls.load_config(
  File "C:\Users\NGXCRYPT-2ND\.conda\envs\m_animate_for_win\lib\site-packages\diffusers\configuration_utils.py", line 364, in load_config
    raise EnvironmentError(
OSError: Error no file named config.json found in directory pretrained_models/stable-diffusion-v1-5.

thanks

torch.cuda.OutOfMemoryError: CUDA out of memory

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 160.00 MiB (GPU 0; 8.00 GiB total capacity; 14.25 GiB already allocated; 0 bytes free; 14.50 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

安装超时

fatal: unable to access 'https://huggingface.co/zcxu-eric/MagicAnimate/': Failed to connect to huggingface.co port 443 after 21313 ms: Couldn't connect to server
Error(s) during clone:
git clone failed: exit status 128

后面那些模型都是这个情况

Can the optimize the vram requirements and increase the generation speed?

Can the optimize the vram requirements and increase the generation speed?

run报错，请问怎么解决

Traceback (most recent call last):
File "demo/gradio_animate.py", line 17, in
from demo.animate import MagicAnimate
File "F:\GitHub_very_good\magic-animate-for-windows\demo\animate.py", line 32, in
from magicanimate.models.model_util import load_models, torch_gc
File "F:\GitHub_very_good\magic-animate-for-windows\magicanimate\models\model_util.py", line 37, in
) -> tuple[CLIPTokenizer, CLIPTextModel, UNet2DConditionModel,]:
TypeError: 'type' object is not subscriptable

[Feature Request] Debug checkbox!

How about adding a debug checkbox like in this pull request to save only the resulting video, excluding the reference image and densepose footage?

magic-research#28

[FileNotFoundError] I would like to try running it locally. Could you please help.

(venv) PS C:\magic-animate-for-windows-main> .\run_gui.ps1
Running on local URL:  http://127.0.0.1:7860
Running on public URL: https://d18cabb55d0e6cea6e.gradio.live

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)
Traceback (most recent call last):
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio\queueing.py", line 456, in call_prediction
    output = await route_utils.call_process_api(
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio\route_utils.py", line 232, in call_process_api        
    output = await app.get_blocks().process_api(
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio\blocks.py", line 1531, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio\blocks.py", line 1421, in postprocess_data
    outputs_cached = processing_utils.move_files_to_cache(prediction_value, block, postprocess=True)  # type: ignore
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio\processing_utils.py", line 279, in move_files_to_cache
    return client_utils.traverse(data, _move_to_cache, client_utils.is_file_obj)
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio_client\utils.py", line 791, in traverse
    new_obj[key] = traverse(value, func, is_root)
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio_client\utils.py", line 787, in traverse
    return func(json_obj)
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio\processing_utils.py", line 271, in _move_to_cache
    temp_file_path = move_resource_to_block_cache(payload.path, block)
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio\processing_utils.py", line 248, in move_resource_to_block_cache
    return block.move_resource_to_block_cache(url_or_file_path)
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio\blocks.py", line 257, in move_resource_to_block_cache
    temp_file_path = processing_utils.save_file_to_cache(
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio\processing_utils.py", line 185, in save_file_to_cache
    temp_dir = hash_file(file_path)
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio\processing_utils.py", line 117, in hash_file
    with open(file_path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\magic-animate-for-windows-main\\demo\\outputs\\2023-12-13T12-57-36.mp4'
Traceback (most recent call last):
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio\queueing.py", line 456, in call_prediction
    output = await route_utils.call_process_api(
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio\route_utils.py", line 232, in call_process_api
    output = await app.get_blocks().process_api(
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio\blocks.py", line 1531, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio\blocks.py", line 1421, in postprocess_data
    outputs_cached = processing_utils.move_files_to_cache(prediction_value, block, postprocess=True)  # type: ignore
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio\processing_utils.py", line 279, in move_files_to_cache
    return client_utils.traverse(data, _move_to_cache, client_utils.is_file_obj)
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio_client\utils.py", line 791, in traverse
    new_obj[key] = traverse(value, func, is_root)
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio_client\utils.py", line 787, in traverse
    return func(json_obj)
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio\processing_utils.py", line 271, in _move_to_cache
exception:

Traceback (most recent call last):
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio\queueing.py", line 501, in process_events
    response = await self.call_prediction(awake_events, batch)
  File "C:\magic-animate-for-windows-main\venv\lib\site-packages\gradio\queueing.py", line 465, in call_prediction
    raise Exception(str(error) if show_error else None) from error
Exception: None

Did I miss any code that I need to install or modify? It cannot be run locally. When I create a video, I get an error saying the file cannot be found.

没有gpu的window机器能跑起来吗

你好，我的电脑是普通的windows机器，是否可以跑起来这个项目，如果可以的话，要注意哪些事项？

instalation error ( ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'

at the end of installation, get this error messages

ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements-video.txt'

尝试使用run_VidControlnetAux_gui和本体的时候都出现了同样的报错，秒错。无法使用

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run gradio deploy from Terminal to deploy to Spaces (https://huggingface.co/spaces)
Traceback (most recent call last):
File "E:\Stable-Diffusion\animation\magic-animate-for-windows\venv\Lib\site-packages\gradio\queueing.py", line 456, in call_prediction
output = await route_utils.call_process_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Stable-Diffusion\animation\magic-animate-for-windows\venv\Lib\site-packages\gradio\route_utils.py", line 232, in call_process_api
output = await app.get_blocks().process_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Stable-Diffusion\animation\magic-animate-for-windows\venv\Lib\site-packages\gradio\blocks.py", line 1520, in process_api
inputs = self.preprocess_data(fn_index, inputs, state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Stable-Diffusion\animation\magic-animate-for-windows\venv\Lib\site-packages\gradio\blocks.py", line 1303, in preprocess_data
processed_input.append(block.preprocess(inputs_cached))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Stable-Diffusion\animation\magic-animate-for-windows\venv\Lib\site-packages\gradio\components\video.py", line 168, in preprocess
duration = processing_utils.get_video_length(file_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Stable-Diffusion\animation\magic-animate-for-windows\venv\Lib\site-packages\gradio\processing_utils.py", line 725, in get_video_length
duration = subprocess.check_output(
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\subprocess.py", line 466, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\subprocess.py", line 548, in run
with Popen(*popenargs, **kwargs) as process:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\subprocess.py", line 1026, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Python311\Lib\subprocess.py", line 1538, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] 系统找不到指定的文件。
Traceback (most recent call last):
File "E:\Stable-Diffusion\animation\magic-animate-for-windows\venv\Lib\site-packages\gradio\queueing.py", line 456, in call_prediction
output = await route_utils.call_process_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Stable-Diffusion\animation\magic-animate-for-windows\venv\Lib\site-packages\gradio\route_utils.py", line 232, in call_process_api
output = await app.get_blocks().process_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Stable-Diffusion\animation\magic-animate-for-windows\venv\Lib\site-packages\gradio\blocks.py", line 1520, in process_api
inputs = self.preprocess_data(fn_index, inputs, state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Stable-Diffusion\animation\magic-animate-for-windows\venv\Lib\site-packages\gradio\blocks.py", line 1303, in preprocess_data
processed_input.append(block.preprocess(inputs_cached))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Stable-Diffusion\animation\magic-animate-for-windows\venv\Lib\site-packages\gradio\components\video.py", line 168, in preprocess
duration = processing_utils.get_video_length(file_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Stable-Diffusion\animation\magic-animate-for-windows\venv\Lib\site-packages\gradio\processing_utils.py", line 725, in get_video_length
duration = subprocess.check_output(
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\subprocess.py", line 466, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\subprocess.py", line 548, in run
with Popen(*popenargs, **kwargs) as process:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\subprocess.py", line 1026, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Python311\Lib\subprocess.py", line 1538, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] 系统找不到指定的文件。

Getting same result when using original 1.5 model and other 1.5 based model

I got same result when i tried to use other model from civitai. i modified the pretrained model path to downloaded .safetensor file. did i miss anything?
Also is there's anything i can help with?

Is M1 able to run this project?

my computer can run stable diffusion but I don't know whether it can run this project.

ModuleNotFoundError: No module named 'diffusers.pipeline_utils'

(animate) PS F:\code\magic-animate-for-windows> python -m demo.gradio_animate Traceback (most recent call last): File "D:\ProgramData\Anaconda3\envs\animate\lib\runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "D:\ProgramData\Anaconda3\envs\animate\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "F:\code\magic-animate-for-windows\demo\gradio_animate.py", line 17, in <module> from demo.animate import MagicAnimate File "F:\code\magic-animate-for-windows\demo\animate.py", line 33, in <module> from magicanimate.pipelines.pipeline_animation import AnimationPipeline File "F:\code\magic-animate-for-windows\magicanimate\pipelines\pipeline_animation.py", line 43, in <module> from diffusers.pipeline_utils import DiffusionPipeline ModuleNotFoundError: No module named 'diffusers.pipeline_utils'

Error no file named config.json found in directory pretrained_models/stable-diffusion-v1-5.

Error no file named config.json found in directory pretrained_models/stable-diffusion-v1-5.

我增加了分辨率匹配，帧率匹配，自定义prompt

https://github.com/nicehero/magic-animate-for-colab

请问这个问题怎么解决

安装Video_controlnet_aux...
fatal: not a git repository (or any of the parent directories): .git
ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements-video.txt'

OSError: Error no file named diffusion_pytorch_model.bin found in directory pretrained_models/MagicAnimate/appearance_encoder

My pretrained_models/MagicAnimate/appearance_encoder is empty after running install.ps1

Running ./run_gui.ps1 will give the below error

(venv) PS C:\Users\leona\dev\magic-animate-for-windows> ./run_gui.ps1
C:\Users\leona\dev\magic-animate-for-windows\magicanimate\pipelines\pipeline_animation.py:43: FutureWarning: Importing `DiffusionPipeline` or `ImagePipelineOutput` from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead.
  from diffusers.pipeline_utils import DiffusionPipeline
Initializing MagicAnimate Pipeline...
### missing keys: 1246; 
### unexpected keys: 51;
Traceback (most recent call last):
  File "C:\Users\leona\dev\magic-animate-for-windows\demo\gradio_animate.py", line 19, in <module>
    animator = MagicAnimate()
  File "C:\Users\leona\dev\magic-animate-for-windows\demo\animate.py", line 90, in __init__
    self.appearance_encoder = AppearanceEncoderModel.from_pretrained(
  File "C:\Users\leona\dev\magic-animate-for-windows\venv\lib\site-packages\diffusers\models\modeling_utils.py", line 618, in from_pretrained
    model_file = _get_model_file(
  File "C:\Users\leona\dev\magic-animate-for-windows\venv\lib\site-packages\diffusers\utils\hub_utils.py", line 284, in _get_model_file
    raise EnvironmentError(
OSError: Error no file named diffusion_pytorch_model.bin found in directory pretrained_models/MagicAnimate/appearance_encoder.

I am following this guide to install
https://www.youtube.com/watch?v=jHfxCD0W5es&t=320s

getting "For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions."

I can run the program flawlessly upon Running on local URL: http://127.0.0.1:7860, but once it start animate the error occure.

Traceback (most recent call last):
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\gradio\queueing.py", line 456, in call_prediction
    output = await route_utils.call_process_api(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\gradio\route_utils.py", line 232, in call_process_api
    output = await app.get_blocks().process_api(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\gradio\blocks.py", line 1522, in process_api
    result = await self.call_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\gradio\blocks.py", line 1144, in call_function
    prediction = await anyio.to_thread.run_sync(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\gradio\utils.py", line 674, in wrapper
    response = f(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\demo\gradio_animate.py", line 30, in animate
    return animator(
           ^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\demo\animate.py", line 236, in __call__
    sample = self.pipeline(
             ^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\magicanimate\pipelines\pipeline_animation.py", line 654, in __call__
    ref_image_latents = self.images2latents(source_image[None, :], latents_dtype).cuda()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\magicanimate\pipelines\pipeline_animation.py", line 394, in images2latents
    latents.append(self.vae.encode(images[frame_idx:frame_idx+1])['latent_dist'].mean * 0.18215)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\diffusers\utils\accelerate_utils.py", line 46, in wrapper
    return method(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\diffusers\models\autoencoder_kl.py", line 260, in encode
    h = self.encoder(x)
        ^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\diffusers\models\vae.py", line 144, in forward
    sample = self.mid_block(sample)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\diffusers\models\unet_2d_blocks.py", line 562, in forward
    hidden_states = attn(hidden_states, temb=temb)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\diffusers\models\attention_processor.py", line 417, in forward
    return self.processor(
           ^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\diffusers\models\attention_processor.py", line 1036, in __call__
    hidden_states = F.scaled_dot_product_attention(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: the launch timed out and was terminated
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\gradio\queueing.py", line 501, in process_events
    response = await self.call_prediction(awake_events, batch)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\gradio\queueing.py", line 465, in call_prediction
    raise Exception(str(error) if show_error else None) from error
Exception: None

I am runing gtx980 with Python 3.11.7 torch-2.0.1cu118-cp311-cp311-win_amd64

C:\Users\PatZ>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Wed_Feb__8_05:53:42_Coordinated_Universal_Time_2023
Cuda compilation tools, release 12.1, V12.1.66
Build cuda_12.1.r12.1/compiler.32415258_0

Virus Detected

I keep getting below error while trying to run magic-animate. Can anybody confirm if this is just a false positive.

Running on local URL: http://127.0.0.1:7860
Traceback (most recent call last):
File "G:\GITHUB\magic-animate-for-windows\venv\lib\site-packages\gradio\networking.py", line 232, in setup_tunnel
address = tunnel.start_tunnel()
File "G:\GITHUB\magic-animate-for-windows\venv\lib\site-packages\gradio\tunneling.py", line 67, in start_tunnel
self.url = self._start_tunnel(BINARY_PATH)
File "G:\GITHUB\magic-animate-for-windows\venv\lib\site-packages\gradio\tunneling.py", line 95, in _start_tunnel
self.proc = subprocess.Popen(
File "C:\Users\micha\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 971, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\micha\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 1440, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
OSError: [WinError 225] Operation did not complete successfully because the file contains a virus or potentially unwanted software

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "G:\GITHUB\magic-animate-for-windows\venv\lib\site-packages\gradio\blocks.py", line 1978, in launch
share_url = networking.setup_tunnel(
File "G:\GITHUB\magic-animate-for-windows\venv\lib\site-packages\gradio\networking.py", line 235, in setup_tunnel
raise RuntimeError(str(e)) from e
RuntimeError: [WinError 225] Operation did not complete successfully because the file contains a virus or potentially unwanted software

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "G:\GITHUB\magic-animate-for-windows\demo\gradio_animate.py", line 141, in
demo.launch(share=True)
File "G:\GITHUB\magic-animate-for-windows\venv\lib\site-packages\gradio\blocks.py", line 1996, in launch
if Path(BINARY_PATH).exists():
File "C:\Users\micha\AppData\Local\Programs\Python\Python310\lib\pathlib.py", line 1290, in exists
self.stat()
File "C:\Users\micha\AppData\Local\Programs\Python\Python310\lib\pathlib.py", line 1097, in stat
return self._accessor.stat(self, follow_symlinks=follow_symlinks)
OSError: [WinError 225] Operation did not complete successfully because the file contains a virus or potentially unwanted software: 'G:\GITHUB\magic-animate-for-windows\venv\lib\site-packages\gradio\frpc_windows_amd64_v0.2'

安装报错，请问怎么解决？

安装Video_controlnet_aux...
fatal: not a git repository (or any of the parent directories): .git
Defaulting to user installation because normal site-packages is not writeable
ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
Defaulting to user installation because normal site-packages is not writeable
ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements-video.txt'
安装完毕

请问怎么解决？