Coder Social home page Coder Social logo

deforum-art / sd-webui-deforum Goto Github PK

View Code? Open in Web Editor NEW
2.6K 39.0 371.0 129.4 MB

Deforum extension for AUTOMATIC1111's Stable Diffusion webui

Home Page: https://deforum.github.io

License: Other

Python 95.55% JavaScript 1.97% Jupyter Notebook 2.30% CSS 0.19%
animations automatic1111 plugin stable-diffusion webui deforum extension

sd-webui-deforum's Introduction

Deforum Stable Diffusion — official extension for AUTOMATIC1111's webui

Last Commit GitHub issues GitHub stars GitHub forks

Need help? See our FAQ

Getting Started

  1. Install AUTOMATIC1111's webui.

  2. Now two ways: either clone the repo into the extensions directory via git commandline launched within in the stable-diffusion-webui folder

git clone https://github.com/deforum-art/sd-webui-deforum extensions/deforum

Or download this repository, locate the extensions folder within your WebUI installation, create a folder named deforum and put the contents of the downloaded directory inside of it. Then restart WebUI.

Or launch A1111, navigate to the Extensions tab, choose Available, find deforum in the list of available extensions and install it. Restart A1111 once the extension has been installed. 3. Open the webui, find the Deforum tab at the top of the page.

  1. Enter the animation settings. Refer to this general guide and this guide to math keyframing functions in Deforum. However, in this version prompt weights less than zero don't just like in original Deforum! Split the positive and the negative prompt in the json section using --neg argument like this "apple:`where(cos(t)>=0, cos(t), 0)`, snow --neg strawberry:`where(cos(t)<0, -cos(t), 0)`"

  2. To view animation frames as they're being made, without waiting for the completion of an animation, go to the 'Settings' tab and set the value of this toolbar above zero. Warning: it may slow down the generation process.

adsdasunknown

  1. Run the script and see if you got it working or even got something. In 3D mode a large delay is expected at first as the script loads the depth models. In the end, using the default settings the whole thing should consume 6.4 GBs of VRAM at 3D mode peaks and no more than 3.8 GB VRAM in 3D mode if you launch the webui with the '--lowvram' command line argument.

  2. After the generation process is completed, click the button with the self-describing name to show the video or gif result right in the GUI!

  3. Join our Discord where you can post generated stuff, ask questions and more: https://discord.gg/deforum.

  • There's also the 'Issues' tab in the repo, for well... reporting issues ;)
  1. Profit!

Known issues

  • This port is not fully backward-compatible with the notebook and the local version both due to the changes in how AUTOMATIC1111's webui handles Stable Diffusion models and the changes in this script to get it to work in the new environment. Expect that you may not get exactly the same result or that the thing may break down because of the older settings.

Screenshots

Amazing raw Deforum animation by Pxl.Pshr:

  • Turn Audio ON!

(Audio credits: SKRILLEX, FRED AGAIN & FLOWDAN - RUMBLE (PHACE'S DNB FLIP))

RumbleByPxlPshr.mp4

Setting file of that video: here.


Main extension tab:

image

Keyframes tab:

image

License

This program is distributed under the terms of the GNU Affero Public License v3.0, copyright (c) 2023 Deforum LLC.

Some of its sublicensed integrated 3rd party components may have other licenses, see LICENSE for usage terms.

sd-webui-deforum's People

Contributors

andyxr avatar aredden avatar blackneoo avatar deforum avatar doggettx avatar enzymezoo-code avatar funofabot avatar hithereai avatar jarrodpan avatar kabachuha avatar kitchenn3 avatar matissesprojects avatar nicolai256 avatar nousr avatar owenvincent avatar patrickvonplaten avatar pesser avatar pharmapsychotic avatar phi-line avatar phobos97 avatar reallybigname avatar rewbs avatar rromb avatar rwscarb avatar thorsthunder04 avatar viking1304 avatar viztopia avatar xmyx avatar yyahav avatar zarxrax avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sd-webui-deforum's Issues

video init mode not saving the final version of the frame

When using video init mode, the preview window shows expected output, however the actual output images that are saved look significantly worse. I believe it might be saving the images at an early step count instead of waiting until all the steps are completed.

Attached image shows the discrepancy. Tested on multiple videos, multiple strength and other settings, by multiple people.
videoinit

Apple Silicon — Running into 2 Errors trying to use 3D Settings

Minutes into many frames and it seemingly going well, I usually get one of two errors

"TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead."

or

"TypeError: Concatenation operation is not implemented for NumPy arrays, use np.concatenate() instead. Please do not rely on this error; it may not be given on all Python implementations."

Can be with the exact same settings, even. Thanks!

DEFORUM script 3d animation wont come out as it does on google collab - using SAME PARAMETERS as online

Hello, i am trying to do the same animation as the ones i do on Google collab, USING THE SCRIPT. Using the SAME PARAMETERS as i use online, even I am copy and paste the arguments like camera movements and all good. BUT, the result is NOTHING as the deforum on Google collab.

WHAT DO I MEAN? The animation comes out as a simple interpolation effect, even if I used 3D animation and have installed the ckpt required and all.... animations using the SAME parameters as online, Won't SHOW same results....

AM I MISSING SOMETHING?

resume from timestring

resume from time string not wok in 2d or 3d mode -

File "D:\sd1111a\stable-diffusion-webui\modules\ui.py", line 221, in f
res = list(func(*args, **kwargs))
File "d:\sd1111a\stable-diffusion-webui\webui.py", line 63, in f
res = func(*args, **kwargs)
File "D:\sd1111a\stable-diffusion-webui\extensions\deforum\scripts\deforum.py", line 219, in run_deforum
File "D:\sd1111a\stable-diffusion-webui\extensions\deforum\scripts\deforum.py", line 66, in run
File "D:\sd1111a\stable-diffusion-webui\extensions\deforum\scripts\deforum\render.py", line 94, in render_animation
prev_sample = sample_from_cv2(img)
File "D:\sd1111a\stable-diffusion-webui\extensions\deforum\scripts\deforum\animation.py", line 18, in sample_from_cv2
sample = ((sample.astype(float) / 255.0) * 2) - 1
AttributeError: 'NoneType' object has no attribute 'astype'

Inverted color channels

If the diffusion cadence is set to any value higher than 1, generated .pngs (except the initial picture) seem to have red and blue color channels swapped. Although they look nomal in the generated preview in UI, those written to disk have this issue.

Example - init generated from default script settings vs frame 0:
2022101722221200006-1052078896-apple_2 0, strawberry_0 0, snow, detailed painting by greg rutkowski
20221017222331_00000

No module named 'py3d_tools'

Have looked at previous issues on this same topic and solutions aren't clear / don't work. #28 and #34

Running this on a google colab notebook and have correctly named the folder deforum in the extensions folder of stable-diffusion-webui

Deforum script for 2D, pseudo-2D and 3D animations
v0.5-webui-beta
Additional models path: /content/gdrive/MyDrive/sd/stable-diffusion-webui/models/Deforum
['/content/gdrive/MyDrive/sd/stable-diffusion/src/k-diffusion', '/content/gdrive/MyDrive/sd/stable-diffusion-webui', '/content/gdrive/MyDrive/sd/stable-diffusion-webui', '/env/python', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/dist-packages', '/usr/lib/python3/dist-packages', '/content/gdrive/MyDrive/sd/stable-diffusion', '/content/gdrive/MyDrive/sd/stable-diffusion/src/taming-transformers', '/content/gdrive/MyDrive/sd/stable-diffusion/src/codeformer', '/content/gdrive/MyDrive/sd/stable-diffusion/src/blip', '/usr/local/lib/python3.7/dist-packages/IPython/extensions', '/tmp/tmp2_ivv8oz', '/content/gdrive/MyDrive/sd/stable-diffusion/scripts/deforum/src', '/content/gdrive/MyDrive/sd/stable-diffusion/extensions/deforum/scripts/deforum/src']
/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/deforum/scripts/deforum/generate.py:104: DeprecationWarning: invalid escape sequence \S
  """, re.VERBOSE)
/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/deforum/scripts/deforum/prompt.py:91: DeprecationWarning: invalid escape sequence \:
  """, re.VERBOSE)
Error completing request
Arguments: (False, '', '2D', 120, 'replicate', '0:(0)', '0:(1.02+0.02*sin(2*3.14*t/20))', '0:(0)', '0:(0)', '0:(10)', '0:(0)', '0:(0)', '0:(0)', False, '0:(0)', '0:(t%15)', '0:(0)', '0:(53)', '0: (0.08)', '0: (0.6)', '0: (1.0)', '0: (7)', '0: (40)', '0: (200)', '0: (10000)', '0: (t%4294967293)', 'Match Frame 0 LAB', 1, True, 0.3, 200.0, 10000.0, 40.0, 'border', 'bicubic', False, '/content/video_in.mp4', 1, False, False, '/content/video_in.mp4', False, 4, False, '20220829210106', '[\n    "a beautiful forest by Asher Brown Durand, trending on Artstation",\n    "a beautiful portrait of a woman by Artgerm, trending on Artstation"\n]\n', '{\n    "0": "apple:`where(cos(6.28*t/10)>0, 2*cos(6.28*t/10), 0.001)`, strawberry:`where(cos(6.28*t/10)<0, -2*cos(6.28*t/10), 0.001)`, snow, detailed painting by greg rutkowski --neg apple:`where(cos(6.28*t/10)<0, -2*cos(6.28*t/10), 0.001)`, strawberry:`where(cos(6.28*t/10)>0, 2*cos(6.28*t/10), 0.001)`",\n    "60": "a beautiful (((banana))), trending on Artstation",\n    "80": "a beautiful coconut --neg photo, realistic",\n    "100": "a beautiful durian, trending on Artstation"\n}\n', 512, 512, False, False, False, 0, 0, -1, 0, False, -1, 0, 0, 0, 21, 0.0, 1, False, 1, True, True, False, False, False, False, 'Deforum', '{timestring}_{index}_{prompt}.png', 'iter', False, False, True, 0, 'https://user-images.githubusercontent.com/14872007/195867706-d067cdc6-28cd-450b-a61e-55e25bc67010.png', False, False, False, True, 'https://www.filterforge.com/wiki/images/archive/b/b7/20080927223728%21Polygonal_gradient_thumb.jpg', 1.0, 5.0, False, 12.0, 'PIL gif', 'ffmpeg', False, 'snowfall.mp3', False, False, 200.0, 'x0_pred', '/content/drive/MyDrive/AI/StableDiffusion/2022-09/20220903000939_%05d.png', '/content/drive/MyDrive/AI/StableDiffusion/content/drive/MyDrive/AI/StableDiffusion/2022-09/kabachuha/2022-09/20220903000939.mp4', '<p style="font-weight:bold;margin-bottom:0.75em">Deforum v0.5-webui-beta</p>', "<p>Made by deforum.github.io, port for AUTOMATIC1111's webui maintained by kabachuha</p>", '<p>Original Deforum Github repo  github.com/deforum/stable-diffusion</p>', "<p>This fork for auto1111's webui github.com/deforum-art/deforum-for-automatic1111-webui</p>", '<p>Join the official Deforum Discord discord.gg/deforum to share your creations and suggestions</p>', '<p>User guide for v0.5 docs.google.com/document/d/1pEobUknMFMkn8F5TMsv8qRzamXX_75BShMMXV8IFslI/edit</p>', '<p>Math keyframing explanation docs.google.com/document/d/1pfW1PwbDIuW0cv-dnuyYj1UzPqe23BlSLTJsqazffXM/edit?usp=sharing</p>', '<p style="font-weight:bold;margin-bottom:0.75em">Import settings from file</p>', '<p style="font-weight:bold;margin-bottom:0.75em">Animation settings</p>', '<p style="margin-bottom:0.75em">Motion parameters:</p>', '<p style="margin-bottom:0.75em">2D and 3D settings</p>', '<p style="margin-bottom:0.75em">Prespective flip — Low VRAM pseudo-3D mode:</p>', '<p style="margin-bottom:0.75em">Coherence:</p>', '<p style="margin-bottom:0.75em">3D Depth Warping:</p>', '<p style="margin-bottom:0.75em">Video Input:</p>', '<p style="margin-bottom:0.75em">Interpolation (turned off atm)</p>', '<p style="margin-bottom:0.75em">Resume animation:</p>', '<p style="font-weight:bold;margin-bottom:0.75em">Prompts</p>', '<p>`animation_mode: None` batches on list of *prompts*. (Batch mode disabled atm, only animation_prompts are working)</p>', '<p style="font-weight:bold">*Important change from vanilla Deforum!*</p>', '<p style="font-weight:italic">This script uses the built-in webui weighting settings.</p>', '<p style="font-weight:italic">So if you want to use math functions as prompt weights,</p>', '<p style="font-weight:italic">keep the values above zero in both parts</p>', '<p style="font-weight:italic">Negative prompt part can be specified with --neg</p>', '<p style="font-weight:bold;margin-bottom:0.75em">Run settings</p>', '<p style="margin-bottom:0.75em">Sampling settings</p>', '<p style="margin-bottom:0.75em">3D Fov settings:</p>', '', '', '', '<p style="margin-bottom:0.75em">Batch settings</p>', '<p style="margin-bottom:0.75em">Init settings</p>', '<p style="margin-bottom:0.75em">3D settings</p>', '<p style="margin-bottom:0.75em">Generation settings:</p>', '<p style="margin-bottom:0.75em">Video output settings</p>', '<p style="margin-bottom:0.75em">To enable seed schedule select seed behavior — \'schedule\'</p>') {}
Traceback (most recent call last):
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/ui.py", line 185, in f
    res = list(func(*args, **kwargs))
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/webui.py", line 63, in f
    res = func(*args, **kwargs)
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/deforum/scripts/deforum.py", line 226, in run_deforum
    processed = DeforumScript.run(None, p, override_settings_with_file, custom_settings_file, animation_mode, max_frames, border, angle, zoom, translation_x, translation_y, translation_z, rotation_3d_x, rotation_3d_y, rotation_3d_z, flip_2d_perspective, perspective_flip_theta, perspective_flip_phi, perspective_flip_gamma, perspective_flip_fv, noise_schedule, strength_schedule, contrast_schedule, cfg_scale_schedule, fov_schedule, near_schedule, far_schedule, seed_schedule, color_coherence, diffusion_cadence, use_depth_warping, midas_weight, near_plane, far_plane, fov, padding_mode, sampling_mode, save_depth_maps, video_init_path, extract_nth_frame, overwrite_extracted_frames, use_mask_video, video_mask_path, interpolate_key_frames, interpolate_x_frames, resume_from_timestring, resume_timestring, prompts, animation_prompts, W, H, restore_faces, tiling, enable_hr, firstphase_width, firstphase_height, seed, sampler, seed_enable_extras, subseed, subseed_strength, seed_resize_from_w, seed_resize_from_h, steps, ddim_eta, n_batch, make_grid, grid_rows, save_settings, save_samples, display_samples, save_sample_per_step, show_sample_per_step, override_these_with_webui, batch_name, filename_format, seed_behavior, use_init, from_img2img_instead_of_link, strength_0_no_init, strength, init_image, use_mask, use_alpha_as_mask, invert_mask, overlay_mask, mask_file, mask_brightness_adjust, mask_overlay_blur, skip_video_for_run_all, fps, output_format, ffmpeg_location, add_soundtrack, soundtrack_path, use_manual_settings, render_steps, max_video_frames, path_name_modifier, image_path, mp4_path, i1, i2, i3, i4, i5, i6, i7, i8, i9, i10, i11, i12, i13, i14, i15, i16, i17, i18, i19, i20, i21, i22, i23, i24, i25, i26, i27, i28, i29, i30, i31, i32, i33, i34, i35, i36)
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/deforum/scripts/deforum.py", line 63, in run
    from scripts.deforum.render import render_animation, render_input_video
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/deforum/scripts/deforum/render.py", line 13, in <module>
    from .generate import generate, add_noise
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/deforum/scripts/deforum/generate.py", line 23, in <module>
    from .animation import sample_from_cv2, sample_to_cv2
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/deforum/scripts/deforum/animation.py", line 5, in <module>
    import py3d_tools as p3d
ModuleNotFoundError: No module named 'py3d_tools'

Issues regarding Masks and 3D mode

Hello,
Ive been getting two different errors that i am unable to workout on my own. The first being an issue with 3d mode. when using it, I get nothing but noise frames after the first image such as you see below. I also get an actual error if i try disabling 3d warping but not sure if thats normal.
Noisy Image: 20221027023619_00001

Kept all settings at default other than slightly raising the samples. (may notice the translation_z at 1 but same results at any number)
Settings:
20221027023619_settings.txt

The second error i have is when trying to use a mask. Everything works fine until the second i enable the mask. Also happens with the default mask link. Error and setting below.

Settings:
20221027024630_settings.txt

Error: Error completing request
Arguments: (False, '', '3D', 100, 'replicate', '0:(0)', '0:(1)', '0:(0)', '0:(0)', '0:(10)', '0:(0)', '0:(0)', '0:(0)', False, '0:(0)', '0:(t%15)', '0:(0)', '0:(53)', '0: (0.08)', '0: (0.6)', '0: (1.0)', '0: (7)', '0: (40)', '0: (200)', '0: (10000)', '0: (t%4294967293)', 'Match Frame 0 LAB', 1, True, 0.3, 200.0, 10000.0, 40.0, 'border', 'bicubic', False, '/content/video_in.mp4', 1, False, False, '/content/video_in.mp4', False, 4, False, '20220829210106', '[\n "a beautiful forest by Asher Brown Durand, trending on Artstation",\n "a beautiful portrait of a woman by Artgerm, trending on Artstation"\n]\n', '{\n "0": "apple:where(cos(6.28*t/10)>0, 2*cos(6.28*t/10), 0.001), strawberry:where(cos(6.28*t/10)<0, -2*cos(6.28*t/10), 0.001), snow, detailed painting by greg rutkowski --neg apple:where(cos(6.28*t/10)<0, -2*cos(6.28*t/10), 0.001), strawberry:where(cos(6.28*t/10)>0, 2*cos(6.28*t/10), 0.001)",\n "60": "a beautiful (((banana))), trending on Artstation",\n "80": "a beautiful coconut --neg photo, realistic",\n "100": "a beautiful durian, trending on Artstation"\n}\n', 512, 512, False, False, False, 0, 0, -1, 0, False, -1, 0, 0, 0, 30, 0.0, 1, False, 1, True, True, False, False, False, False, 'testing', '{timestring}{index}{prompt}.png', 'iter', True, False, True, 0.7, 'https://user-images.githubusercontent.com/14872007/195867706-d067cdc6-28cd-450b-a61e-55e25bc67010.png', True, False, False, True, 'https://www.filterforge.com/wiki/images/archive/b/b7/20080927223728%21Polygonal_gradient_thumb.jpg', 1.0, 5.0, False, 12.0, 'FFMPEG mp4', 'ffmpeg', False, 'snowfall.mp3', False, False, 200.0, 'x0_pred', '/content/drive/MyDrive/AI/StableDiffusion/2022-09/20220903000939_%05d.png', '/content/drive/MyDrive/AI/StableDiffusion/content/drive/MyDrive/AI/StableDiffusion/2022-09/kabachuha/2022-09/20220903000939.mp4', '

Deforum v0.5-webui-beta

', "

Made by deforum.github.io, port for AUTOMATIC1111's webui maintained by kabachuha

", '

Original Deforum Github repo github.com/deforum/stable-diffusion

', "

This fork for auto1111's webui github.com/deforum-art/deforum-for-automatic1111-webui

", '

Join the official Deforum Discord discord.gg/deforum to share your creations and suggestions

', '

User guide for v0.5 docs.google.com/document/d/1pEobUknMFMkn8F5TMsv8qRzamXX_75BShMMXV8IFslI/edit

', '

Math keyframing explanation docs.google.com/document/d/1pfW1PwbDIuW0cv-dnuyYj1UzPqe23BlSLTJsqazffXM/edit?usp=sharing

', '

Import settings from file

', '

Animation settings

', '

Motion parameters:

', '

2D and 3D settings

', '

Prespective flip — Low VRAM pseudo-3D mode:

', '

Coherence:

', '

3D Depth Warping:

', '

Video Input:

', '

Interpolation (turned off atm)

', '

Resume animation:

', '

Prompts

', '

animation_mode: None batches on list of prompts. (Batch mode disabled atm, only animation_prompts are working)

', '

Important change from vanilla Deforum!

', '

This script uses the built-in webui weighting settings.

', '

So if you want to use math functions as prompt weights,

', '

keep the values above zero in both parts

', '

Negative prompt part can be specified with --neg

', '

Run settings

', '

Sampling settings

', '

3D Fov settings:

', '', '', '', '

Batch settings

', '

Init settings

', '

3D settings

', '

Generation settings:

', '

Video output settings

', '

To enable seed schedule select seed behavior — 'schedule'

') {}
Traceback (most recent call last):
File "C:\Users\reidb\Desktop\SUPER SD\stable-diffusion-webui\modules\ui.py", line 221, in f
res = list(func(*args, **kwargs))
File "C:\Users\reidb\Desktop\SUPER SD\stable-diffusion-webui\webui.py", line 63, in f
res = func(*args, **kwargs)
File "C:\Users\reidb\Desktop\SUPER SD\stable-diffusion-webui\extensions\deforum\scripts\deforum.py", line 219, in run_deforum
processed = DeforumScript.run(None, p, override_settings_with_file, custom_settings_file, animation_mode, max_frames, border, angle, zoom, translation_x, translation_y, translation_z, rotation_3d_x, rotation_3d_y, rotation_3d_z, flip_2d_perspective, perspective_flip_theta, perspective_flip_phi, perspective_flip_gamma, perspective_flip_fv, noise_schedule, strength_schedule, contrast_schedule, cfg_scale_schedule, fov_schedule, near_schedule, far_schedule, seed_schedule, color_coherence, diffusion_cadence, use_depth_warping, midas_weight, near_plane, far_plane, fov, padding_mode, sampling_mode, save_depth_maps, video_init_path, extract_nth_frame, overwrite_extracted_frames, use_mask_video, video_mask_path, interpolate_key_frames, interpolate_x_frames, resume_from_timestring, resume_timestring, prompts, animation_prompts, W, H, restore_faces, tiling, enable_hr, firstphase_width, firstphase_height, seed, sampler, seed_enable_extras, subseed, subseed_strength, seed_resize_from_w, seed_resize_from_h, steps, ddim_eta, n_batch, make_grid, grid_rows, save_settings, save_samples, display_samples, save_sample_per_step, show_sample_per_step, override_these_with_webui, batch_name, filename_format, seed_behavior, use_init, from_img2img_instead_of_link, strength_0_no_init, strength, init_image, use_mask, use_alpha_as_mask, invert_mask, overlay_mask, mask_file, mask_brightness_adjust, mask_overlay_blur, skip_video_for_run_all, fps, output_format, ffmpeg_location, add_soundtrack, soundtrack_path, use_manual_settings, render_steps, max_video_frames, path_name_modifier, image_path, mp4_path, i1, i2, i3, i4, i5, i6, i7, i8, i9, i10, i11, i12, i13, i14, i15, i16, i17, i18, i19, i20, i21, i22, i23, i24, i25, i26, i27, i28, i29, i30, i31, i32, i33, i34, i35, i36)
File "C:\Users\reidb\Desktop\SUPER SD\stable-diffusion-webui\extensions\deforum\scripts\deforum.py", line 66, in run
render_animation(args, anim_args, root.animation_prompts, root)
File "C:\Users\reidb\Desktop\SUPER SD\stable-diffusion-webui\extensions\deforum\scripts\deforum\render.py", line 212, in render_animation
sample, image = generate(args, root, frame_idx, return_sample=True)
File "C:\Users\reidb\Desktop\SUPER SD\stable-diffusion-webui\extensions\deforum\scripts\deforum\generate.py", line 199, in generate
init_image.shape,
File "C:\Users\reidb\Desktop\SUPER SD\stable-diffusion-webui\venv\lib\site-packages\PIL\Image.py", line 517, in getattr
raise AttributeError(name)
AttributeError: shape

Any help is much appreciated. This is my 4th day trying to fix it.

Video mask not working on video input mode

After clicking on Generate I'm getting the following error:
File "D:\ai\stable-diffusion-webui\extensions\deforum\scripts\deforum.py", line 68, in run
render_input_video(args, anim_args, root.animation_prompts, root)#TODO: prettify code
File "D:\ai\stable-diffusion-webui\extensions\deforum\scripts\deforum\render.py", line 258, in render_input_video
render_animation(args, anim_args, animation_prompts, root)
File "D:\ai\stable-diffusion-webui\extensions\deforum\scripts\deforum\render.py", line 212, in render_animation
sample, image = generate(args, root, frame_idx, return_sample=True)
File "D:\ai\stable-diffusion-webui\extensions\deforum\scripts\deforum\generate.py", line 199, in generate
init_image.shape,
File "D:\ai\stable-diffusion-webui\venv\lib\site-packages\PIL\Image.py", line 517, in getattr
raise AttributeError(name)
AttributeError: shape

My init params:
image

Without video mask generation is working. Both mask and original video has same dimensions, length and frame rate.

Tried to change line 199 in generate.py to init_image.size, got next error:
File "D:\ai\stable-diffusion-webui\extensions\deforum\scripts\deforum.py", line 68, in run
render_input_video(args, anim_args, root.animation_prompts, root)#TODO: prettify code
File "D:\ai\stable-diffusion-webui\extensions\deforum\scripts\deforum\render.py", line 258, in render_input_video
render_animation(args, anim_args, animation_prompts, root)
File "D:\ai\stable-diffusion-webui\extensions\deforum\scripts\deforum\render.py", line 212, in render_animation
sample, image = generate(args, root, frame_idx, return_sample=True)
File "D:\ai\stable-diffusion-webui\extensions\deforum\scripts\deforum\generate.py", line 200, in generate
args.mask_contrast_adjust,
AttributeError: 'types.SimpleNamespace' object has no attribute 'mask_contrast_adjust'

Tried to hardcode mask_contrast_adjust=1, so lines 198-202 in generate.py will look like this:
mask = prepare_mask(args.mask_file if mask_image is None else mask_image, init_image.size, 1, args.mask_brightness_adjust, args.invert_mask)

Got this error:
File "D:\ai\stable-diffusion-webui\extensions\deforum\scripts\deforum.py", line 68, in run
render_input_video(args, anim_args, root.animation_prompts, root)#TODO: prettify code
File "D:\ai\stable-diffusion-webui\extensions\deforum\scripts\deforum\render.py", line 258, in render_input_video
render_animation(args, anim_args, animation_prompts, root)
File "D:\ai\stable-diffusion-webui\extensions\deforum\scripts\deforum\render.py", line 212, in render_animation
sample, image = generate(args, root, frame_idx, return_sample=True)
File "D:\ai\stable-diffusion-webui\extensions\deforum\scripts\deforum\generate.py", line 216, in generate
processed = processing.process_images(p)
File "D:\ai\stable-diffusion-webui\modules\processing.py", line 359, in process_images
res = process_images_inner(p)
File "D:\ai\stable-diffusion-webui\modules\processing.py", line 452, in process_images_inner
samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength)
File "D:\ai\stable-diffusion-webui\modules\processing.py", line 805, in sample
samples = self.sampler.sample_img2img(self, self.init_latent, x, conditioning, unconditional_conditioning, image_conditioning=self.image_conditioning)
File "D:\ai\stable-diffusion-webui\modules\sd_samplers.py", line 432, in sample_img2img
samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args={
File "D:\ai\stable-diffusion-webui\modules\sd_samplers.py", line 365, in launch_sampling
return func()
File "D:\ai\stable-diffusion-webui\modules\sd_samplers.py", line 432, in
samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args={
File "D:\ai\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "D:\ai\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 80, in sample_euler_ancestral
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "D:\ai\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "D:\ai\stable-diffusion-webui\modules\sd_samplers.py", line 310, in forward
denoised = self.init_latent * self.mask + self.nmask * denoised
TypeError: unsupported operand type(s) for *: 'Tensor' and 'Image'

Manage pickle permissions instead of '--disable-safe-unpickle'

It seems there's way to manage only the permissions of specific downloadable models, instead of disabling the checks for all. If so, it would both simplify the installation process and protect against malicious checkpoints.


u/RealAstropulse on Reddit:

Agreed, but most likely it’s because all “safe-unpickle” does is limit the types of variables a pickle file can have, and the 3d stuff needs kinds bot specified. It would be better to modify the pickle permissions instead, but i guess they didnt think of that.

The specific module is here: https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/master/modules/safe.py

ModuleNotFoundError: No module named 'py3d_tools'

I'm using Colab, fast_stable_diffusion_AUTOMATIC1111.ipynb to run the Deforum extension for animation. I'm attempting to generate, and have left it at the built-in parameters as a way to test.

I continue to get this error:
ModuleNotFoundError: No module named 'py3d_tools'

I've tried it with my own copy and the public version, and did a clean re-install mount of "fast" onto my Google Drive.

I came across a similar issue experienced for Disco Diff. I followed up by installing the code, and everything mounts fine but it still gives this error when I run the Deforum extension.

AttributeError: 'NoneType' object has no attribute 'ema_scope'

Hi,
It's a fresh install, I've never tried this extension before. I'm running Windows and Stable-Diffusion works fine.
I use the default settings, I don't change anything. I click Generate, I see two images being generated, and the third one stops at 3% with this error :

 File "C:\stable-diffusion-webui\extensions\deforum\scripts\deforum.py", line 69, in run
    render_animation(args, anim_args, root.animation_prompts, root)
  File "C:\stable-diffusion-webui\extensions\deforum\scripts\deforum\render.py", line 213, in render_animation
    sample, image = generate(args, root, frame_idx, return_sample=True)
  File "C:\stable-diffusion-webui\extensions\deforum\scripts\deforum\generate.py", line 215, in generate
    processed = processing.process_images(p)
  File "C:\stable-diffusion-webui\modules\processing.py", line 427, in process_images
    res = process_images_inner(p)
  File "C:\stable-diffusion-webui\modules\processing.py", line 487, in process_images_inner
    with torch.no_grad(), p.sd_model.ema_scope():
AttributeError: 'NoneType' object has no attribute 'ema_scope'

I know Windows but I don't know Python. Any help would be appreciated.

Find a way to unhardcode additional modules path location

As there were quite a lot of complaints regarding py3d and other src stuff not being found when users put their files in paths named not like 'deforum' or with renamed parent folder, it would be better to determine the actual deforum script path automatically, but it may be quite a challenge to do so because get_cwd returns automatic's root directory and not the folder with 'deforum'

Value 0 is not in list

There is a description in the web UI that when you load custom settings it just works once.

It doesn't work at all though. When you try to run the loaded settings then you will get the same error than if you try to load the custom settings twice.

loadsettings

valueerror

Unlock resolution dimensions below 512

There was is bug in vanilla Deforum breaking the pictures apart if the dimensions are below 512, but if Auto fixed it in webui's samplers (as Aeloi reports on Discord), unlocking it should be fine

Add an option to upscale the whole video before the output

Automatic1111 contains a number of upscalers and it should be pretty easy to convert each frame to a larger resolution. I think the best location for it will be in video output settings, so users will be able to preview the result before doing so

Problem with colors and noise.

When a frame is rendered, the second last step looks good but at the last step it seems like a color correction or a color conversion is added. Blacks are turning redish, pinkish , whites are burned and gradients aren t smooth anymore. I played a lot last week with deforum and i didn't have this problem.

This is an example of the issue. I tried playing with the noise and strength values but no good results

image

Here's another example where i tried to lower the contrast and somehow the problem become more visible

image

And here is the original init photo
20221018070807-2315062896-futuristic human with glowy skin, 4k, detailed, kodak vision

It looks somehow like a color conversion problem.

Video init not working, getting a keyerror from range.py

Trying to figure out why video init isn't working. I have Ffmpeg installed to Path, can't think of what else I can do

Traceback (most recent call last):
File "C:\Users\lahro\stable-diffusion-webui\modules\ui.py", line 217, in f
res = list(func(*args, **kwargs))
File "C:\Users\lahro\stable-diffusion-webui\webui.py", line 63, in f
res = func(*args, **kwargs)
File "C:\Users\lahro\stable-diffusion-webui\modules\img2img.py", line 126, in img2img
processed = modules.scripts.scripts_img2img.run(p, *args)
File "C:\Users\lahro\stable-diffusion-webui\modules\scripts.py", line 173, in run
processed = script.run(p, *script_args)
File "C:\Users\lahro\stable-diffusion-webui\scripts\deforum.py", line 50, in run
render_input_video(args, anim_args, root.animation_prompts, root)#TODO: prettify code
File "C:\Users\lahro\stable-diffusion-webui\scripts\deforum\render.py", line 258, in render_input_video
render_animation(args, anim_args, animation_prompts, root)
File "C:\Users\lahro\stable-diffusion-webui\scripts\deforum\render.py", line 36, in render_animation
keys = DeformAnimKeys(anim_args)
File "C:\Users\lahro\stable-diffusion-webui\scripts\deforum\animation.py", line 255, in init
self.angle_series = get_inbetweens(parse_key_frames(anim_args.angle), anim_args.max_frames)
File "C:\Users\lahro\stable-diffusion-webui\scripts\deforum\animation.py", line 295, in get_inbetweens
key_frame_series[0] = key_frame_series[key_frame_series.first_valid_index()]
File "C:\Users\lahro\stable-diffusion-webui\venv\lib\site-packages\pandas\core\series.py", line 981, in getitem
return self._get_value(key)
File "C:\Users\lahro\stable-diffusion-webui\venv\lib\site-packages\pandas\core\series.py", line 1089, in _get_value
loc = self.index.get_loc(label)
File "C:\Users\lahro\stable-diffusion-webui\venv\lib\site-packages\pandas\core\indexes\range.py", line 395, in get_loc raise KeyError(key)
KeyError: None

Video Input "Extract_Nth_Frame" does not do anything

Hello everyone

I have run a test on the feature "Extract_Nth_Frame" because I had noticed that it did not seem to be working, and I wanted to confirm with a very basic video.

I created a 30 frame video which looks like this:

frames used to create mp4 file

I ran deforum using the path to the MP4 itself (not the frames) in the video_init box, and had set the "Extract Nth Frame" to 5.

deforum settings

I noticed pretty immediately that the program is ignoring this setting... taking all 30 frames and editing them each.

batch cmd NOT skipping any frames

Once the program finished running I could see that indeed it gave me 30 frames in output (they are blank because I set the strength schedule to max thinking it might keep the numbers... but it did not. Oh well).

30 frames in output!

So... I don't think this feature works :(

Deforum doesn't load after latest updates

After updating the extension in SD Web UI this morning, Deforum doesn't load anymore in the UI and the console displays the following error messages:

Error executing callback ui_tabs_callback for C:\stable-diffusion-webui\extensions\deforum-for-automatic1111-webui\scripts\deforum.py
Traceback (most recent call last):
File "C:\stable-diffusion-webui\modules\script_callbacks.py", line 87, in ui_tabs_callback
res += c.callback() or []
File "C:\stable-diffusion-webui\extensions\deforum-for-automatic1111-webui\scripts\deforum.py", line 325, in on_ui_tabs
component_list = [ds.override_settings_with_file, ds.custom_settings_file, ds.animation_mode, ds.max_frames, ds.border, ds.angle, ds.zoom, ds.translation_x, ds.translation_y, ds.translation_z, ds.rotation_3d_x, ds.rotation_3d_y, ds.rotation_3d_z, ds.flip_2d_perspective, ds.perspective_flip_theta, ds.perspective_flip_phi, ds.perspective_flip_gamma, ds.perspective_flip_fv, ds.noise_schedule, ds.strength_schedule, ds.contrast_schedule, ds.cfg_scale_schedule, ds.fov_schedule, ds.near_schedule, ds.far_schedule, ds.seed_schedule, ds.color_coherence, ds.diffusion_cadence, ds.use_depth_warping, ds.midas_weight, ds.near_plane, ds.far_plane, ds.fov, ds.padding_mode, ds.sampling_mode, ds.save_depth_maps, ds.video_init_path, ds.extract_nth_frame, ds.overwrite_extracted_frames, ds.use_mask_video, ds.video_mask_path, ds.interpolate_key_frames, ds.interpolate_x_frames, ds.resume_from_timestring, ds.resume_timestring, ds.prompts, ds.animation_prompts, ds.W, ds.H, ds.restore_faces, ds.tiling, ds.enable_hr, ds.firstphase_width, ds.firstphase_height, ds.seed, ds.sampler, ds.seed_enable_extras, ds.subseed, ds.subseed_strength, ds.seed_resize_from_w, ds.seed_resize_from_h, ds.steps, ds.ddim_eta, ds.n_batch, ds.make_grid, ds.grid_rows, ds.save_settings, ds.save_samples, ds.display_samples, ds.save_sample_per_step, ds.show_sample_per_step, ds.override_these_with_webui, ds.batch_name, ds.filename_format, ds.seed_behavior, ds.use_init, ds.from_img2img_instead_of_link, ds.strength_0_no_init, ds.strength, ds.init_image, ds.use_mask, ds.use_alpha_as_mask, ds.invert_mask, ds.overlay_mask, ds.mask_file, ds.mask_contrast_adjust, ds.mask_brightness_adjust, ds.mask_overlay_blur, ds.skip_video_for_run_all, ds.fps, ds.output_format, ds.ffmpeg_location, ds.add_soundtrack, ds.soundtrack_path, ds.use_manual_settings, ds.render_steps, ds.max_video_frames, ds.path_name_modifier, ds.image_path, ds.mp4_path, ds.i1, ds.i2, ds.i3, ds.i4, ds.i5, ds.i6, ds.i7, ds.i8, ds.i9, ds.i10, ds.i11, ds.i12, ds.i13, ds.i14, ds.i15, ds.i16, ds.i17, ds.i18, ds.i19, ds.i20, ds.i21, ds.i22, ds.i23, ds.i24, ds.i25, ds.i26, ds.i27, ds.i28, ds.i29, ds.i30, ds.i31, ds.i32, ds.i33, ds.i34, ds.i35, ds.i36]
AttributeError: 'types.SimpleNamespace' object has no attribute 'mask_contrast_adjust'

Unable to use without being online

I have been getting multiple errors and unable to render out images. Guess my internet had went out and it eventually came back and allowed me to continue to make images. Is there a way to use this extension without having to be connected to the internet?

Merge denoising_strength_schedule with strength_schedule

The denoising_strength_schedule is basically strength_schedule, but within the webui. So it should be easy to map strength_schedule from vanilla Deforum to denoising_strength_schedule. So strength_schedule will be essentially converted denoising_strength_schedule and the 'denoising_strength_schedule' param will be removed from this script

image

[Bug]: ReferenceError: submit_deforum is not defined

after the last update every time i click on generate in deforum tab
I get this error
ReferenceError: submit_deforum is not defined
at eval (eval at (index.9828d028.js:54:2891), :3:14)
at index.9828d028.js:56:4850
at HTMLButtonElement. (index.9828d028.js:55:2100)
at index.9828d028.js:4:1266
at Array.forEach ()
at HTMLButtonElement.Zn (index.9828d028.js:4:1253)
at HTMLButtonElement._ (index.88652f98.js:1:2092)
at index.9828d028.js:4:1266
at Array.forEach ()
at HTMLButtonElement.Zn (index.9828d028.js:4:1253)

it seems to be an view problem because the CMD is not changing

image

Error in loading the new extension

I tried installing the new extension (removed the old script then used git clone https://github.com/deforum-art/deforum-for-automatic1111-webui/ extensions/deforum)
but am getting the following error on launching the Webui:

Error executing callback ui_tabs_callback for C:\Users\lhusa\SD\stable-diffusion-webui\extensions\deforum\scripts\deforum.py
Traceback (most recent call last):
File "C:\Users\lhusa\SD\stable-diffusion-webui\modules\script_callbacks.py", line 37, in ui_tabs_callback
res += c.callback() or []
File "C:\Users\lhusa\SD\stable-diffusion-webui\extensions\deforum\scripts\deforum.py", line 238, in on_ui_tabs
components = deforum_args.setup_deforum_setting_dictionary(None, True, True)
AttributeError: module 'scripts.deforum.args' has no attribute 'setup_deforum_setting_dictionary'

error msg when trying to make it work in linux (latest webui)

Hi followed instructions precisely. error i am getting while loading webui. any ideas...

Traceback (most recent call last):
  File "/home/user/Data/github/stable-diffusion/stable-diffusion-webui_AUTO/modules/script_callbacks.py", line 56, in ui_tabs_callback
    res += c.callback() or []
  File "/home/user/Data/github/stable-diffusion/stable-diffusion-webui_AUTO/extensions/deforum/scripts/deforum.py", line 238, in on_ui_tabs
    components = deforum_args.setup_deforum_setting_dictionary(None, True, True)
  File "/home/user/Data/github/stable-diffusion/stable-diffusion-webui_AUTO/extensions/deforum/scripts/deforum/args.py", line 210, in setup_deforum_setting_dictionary
    with gr.Accordion("Info and links", open=False):
AttributeError: module 'gradio' has no attribute 'Accordion'

No module named py3d_tools

Fresh Install when i try to generate get this error

File "D:\Apps\sd-webui\extensions\deforum\scripts\deforum.py", line 62, in run
from scripts.deforum.render import render_animation, render_input_video
File "D:\Apps\sd-webui\extensions\deforum\scripts\deforum\render.py", line 13, in
from .generate import generate, add_noise
File "D:\Apps\sd-webui\extensions\deforum\scripts\deforum\generate.py", line 23, in
from .animation import sample_from_cv2, sample_to_cv2
File "D:\Apps\sd-webui\extensions\deforum\scripts\deforum\animation.py", line 5, in
import py3d_tools as p3d
ModuleNotFoundError: No module named 'py3d_tools'

Video init isn't denoising/diffusing

I've recreated a the config from my colab notebook, using Video Input to process a video of a congo player. Same prompt and settings with anything that isn't in the Colab notebook left at default settings.. I've tried this with sd-1.4 & sd-1.5 checkpoints to make sure it wasn't an issue from using 1.5 (my old notebook has 1.4)

No matter how many steps I set or what I change, it doesn't appear to be denoising (diffusing) the image.

This is what I get when trying to use a video as the init
with_video_init

This is what I get without an init
no_video_init

Here are my settings
deforum_settings.txt

20221029175801_settings.txt

deforum_video-settings.txt

deforum tab not visible from the UI

I followed the instructions, but I can't see the deforum tab in UI.

I don't have anything that seems to be related to deforum.py in the script folder.

venv "D:\sources\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.8 (tags/v3.10.8:aaaf517, Oct 11 2022, 16:50:30) [MSC v.1933 64 bit (AMD64)]
Commit hash: be1596ce30b1ead6998da0c62003003dcce5eb2c
Installing requirements for Web UI
Launching Web UI with arguments: --listen --opt-split-attention --share --gradio-auth userXXX:testXXX--disable-safe-unpickle
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Loading weights [7460a6fa] from D:\sources\stable-diffusion-webui\models\Stable-diffusion\sd-v1-4.ckpt
Global Step: 470000
Applying cross attention optimization (Doggettx).
Model loaded.
Loaded a total of 0 textual inversion embeddings.
Running on local URL:  http://0.0.0.0:7860
Running on public URL: https://xxx.gradio.app

This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces

Add webui 'scripts' support

To use aesthetic gradients and stuff — obviously, can have weird side effects with more complex examples

Add tooltips

Tooltips would definitely help the newcomers, but the list is pretty large, so if you have some time to spare, I'd welcome your contributions. The list of tooltips is a simple JSON file in a form "label":"tooltip". The names of the widgets lie in https://github.com/deforum-art/deforum-for-automatic1111-webui/blob/automatic1111-webui/scripts/deforum.py and https://github.com/deforum-art/deforum-for-automatic1111-webui/blob/automatic1111-webui/scripts/deforum/args.py.

Examples from the main webui

titles = {
    "Sampling steps": "How many times to improve the generated image iteratively; higher values take longer; very low values can produce bad results",
    "Sampling method": "Which algorithm to use to produce the image",
	"GFPGAN": "Restore low quality faces using GFPGAN neural network",
	"Euler a": "Euler Ancestral - very creative, each can get a completely different picture depending on step count, setting steps to higher than 30-40 does not help",
	"DDIM": "Denoising Diffusion Implicit Models - best at inpainting",
}

Video init mode

Try to do some viedo init mode.
but got"integer division or modulo by zero"error.

I have FFMPEG installed and Path set.

Traceback (most recent call last):
File "F:\stable-diffusion-webui\modules\ui.py", line 212, in f
res = list(func(*args, **kwargs))
File "F:\stable-diffusion-webui\webui.py", line 64, in f
res = func(*args, **kwargs)
File "F:\stable-diffusion-webui\modules\img2img.py", line 124, in img2img
processed = modules.scripts.scripts_img2img.run(p, *args)
File "F:\stable-diffusion-webui\modules\scripts.py", line 162, in run
processed = script.run(p, *script_args)
File "F:\stable-diffusion-webui\scripts\deforum.py", line 50, in run
render_input_video(args, anim_args, root.animation_prompts, root)#TODO: prettify code
File "F:\stable-diffusion-webui\scripts\deforum\render.py", line 241, in render_input_video
vid2frames(anim_args.video_init_path, video_in_frame_path, anim_args.extract_nth_frame, anim_args.overwrite_extracted_frames)
File "F:\stable-diffusion-webui\scripts\deforum\animation.py", line 50, in vid2frames
if count % n == 0:
ZeroDivisionError: integer division or modulo by zero

I tried this below and it still didn't fix the problem
"
andywise — 今天05:50
re: zero division error, it looks like anim_args.extract_nth_frame is coming in as a bool type instead of a number 🤷‍♂️ A temporary fix for me was to hard code a new value for n (1 in my case) on line 50 of animation.py So, my new line 50 reads if count % 1 == 0:# 1 used to be n
"

Interpolation

Thanks for this!
Couldn't find anything about interpolation in the readme or issues. Is it planned or is there a major blocker that makes it hard/impossible to implement for now?

ModuleNotFoundError: No module named 'py3d_tools'

Hi, can anyone help me out how to get py3d_tools? This is the log running SD

Deforum script for 2D, pseudo-2D and 3D animations
v0.5-webui-beta
Additional models path: D:\StableDiffusion\stable-diffusion-webui\models/Deforum
Error completing request
Arguments: (False, '', '2D', 120, 'replicate', '0:(0)', '0:(1.02+0.02sin(23.14*t/20))', '0:(0)', '0:(0)', '0:(10)', '0:(0)', '0:(0)', '0:(0)', False, '0:(0)', '0:(t%15)', '0:(0)', '0:(53)', '0: (0.08)', '0: (0.6)', '0: (1.0)', '0: (7)', '0: (40)', '0: (200)', '0: (10000)', '0: (t%4294967293)', 'Match Frame 0 LAB', 1, True, 0.3, 200.0, 10000.0, 40.0, 'border', 'bicubic', False, '/content/video_in.mp4', 1, False, False, '/content/video_in.mp4', False, 4, False, '20220829210106', '[\n "a beautiful forest by Asher Brown Durand, trending on Artstation",\n "a beautiful portrait of a woman by Artgerm, trending on Artstation"\n]\n', '{\n "0": "apple:where(cos(6.28*t/10)>0, 2*cos(6.28*t/10), 0.001), strawberry:where(cos(6.28*t/10)<0, -2*cos(6.28*t/10), 0.001), snow, detailed painting by greg rutkowski --neg apple:where(cos(6.28*t/10)<0, -2*cos(6.28*t/10), 0.001), strawberry:where(cos(6.28*t/10)>0, 2*cos(6.28*t/10), 0.001)",\n "60": "a beautiful (((banana))), trending on Artstation",\n "80": "a beautiful coconut --neg photo, realistic",\n "100": "a beautiful durian, trending on Artstation"\n}\n', 512, 512, False, False, False, 0, 0, -1, 0, False, -1, 0, 0, 0, 21, 0.0, 1, False, 1, True, True, False, False, False, False, 'Deforum', '{timestring}{index}{prompt}.png', 'iter', False, False, True, 0, 'https://user-images.githubusercontent.com/14872007/195867706-d067cdc6-28cd-450b-a61e-55e25bc67010.png', False, False, False, True, 'https://www.filterforge.com/wiki/images/archive/b/b7/20080927223728%21Polygonal_gradient_thumb.jpg', 1.0, 5.0, False, 12.0, 'PIL gif', 'ffmpeg', False, 'snowfall.mp3', False, False, 200.0, 'x0_pred', '/content/drive/MyDrive/AI/StableDiffusion/2022-09/20220903000939_%05d.png', '/content/drive/MyDrive/AI/StableDiffusion/content/drive/MyDrive/AI/StableDiffusion/2022-09/kabachuha/2022-09/20220903000939.mp4', '

Deforum v0.5-webui-beta

', "

Made by deforum.github.io, port for AUTOMATIC1111's webui maintained by kabachuha

", '

Original Deforum Github repo github.com/deforum/stable-diffusion

', "

This fork for auto1111's webui github.com/deforum-art/deforum-for-automatic1111-webui

", '

Join the official Deforum Discord discord.gg/deforum to share your creations and suggestions

', '

User guide for v0.5 docs.google.com/document/d/1pEobUknMFMkn8F5TMsv8qRzamXX_75BShMMXV8IFslI/edit

', '

Math keyframing explanation docs.google.com/document/d/1pfW1PwbDIuW0cv-dnuyYj1UzPqe23BlSLTJsqazffXM/edit?usp=sharing

', '

Import settings from file

', '

Animation settings

', '

Motion parameters:

', '

2D and 3D settings

', '

Prespective flip — Low VRAM pseudo-3D mode:

', '

Coherence:

', '

3D Depth Warping:

', '

Video Input:

', '

Interpolation (turned off atm)

', '

Resume animation:

', '

Prompts

', '

animation_mode: None batches on list of prompts. (Batch mode disabled atm, only animation_prompts are working)

', '

Important change from vanilla Deforum!

', '

This script uses the built-in webui weighting settings.

', '

So if you want to use math functions as prompt weights,

', '

keep the values above zero in both parts

', '

Negative prompt part can be specified with --neg

', '

Run settings

', '

Sampling settings

', '

3D Fov settings:

', '', '', '', '

Batch settings

', '

Init settings

', '

3D settings

', '

Generation settings:

', '

Video output settings

', '

To enable seed schedule select seed behavior — 'schedule'

') {}
Traceback (most recent call last):
File "D:\StableDiffusion\stable-diffusion-webui\modules\ui.py", line 185, in f
res = list(func(*args, **kwargs))
File "D:\StableDiffusion\stable-diffusion-webui\webui.py", line 53, in f
res = func(*args, **kwargs)
File "D:\StableDiffusion\stable-diffusion-webui\extensions\deforum-for-automatic1111-webui\scripts\deforum.py", line 225, in run_deforum
processed = DeforumScript.run(None, p, override_settings_with_file, custom_settings_file, animation_mode, max_frames, border, angle, zoom, translation_x, translation_y, translation_z, rotation_3d_x, rotation_3d_y, rotation_3d_z, flip_2d_perspective, perspective_flip_theta, perspective_flip_phi, perspective_flip_gamma, perspective_flip_fv, noise_schedule, strength_schedule, contrast_schedule, cfg_scale_schedule, fov_schedule, near_schedule, far_schedule, seed_schedule, color_coherence, diffusion_cadence, use_depth_warping, midas_weight, near_plane, far_plane, fov, padding_mode, sampling_mode, save_depth_maps, video_init_path, extract_nth_frame, overwrite_extracted_frames, use_mask_video, video_mask_path, interpolate_key_frames, interpolate_x_frames, resume_from_timestring, resume_timestring, prompts, animation_prompts, W, H, restore_faces, tiling, enable_hr, firstphase_width, firstphase_height, seed, sampler, seed_enable_extras, subseed, subseed_strength, seed_resize_from_w, seed_resize_from_h, steps, ddim_eta, n_batch, make_grid, grid_rows, save_settings, save_samples, display_samples, save_sample_per_step, show_sample_per_step, override_these_with_webui, batch_name, filename_format, seed_behavior, use_init, from_img2img_instead_of_link, strength_0_no_init, strength, init_image, use_mask, use_alpha_as_mask, invert_mask, overlay_mask, mask_file, mask_brightness_adjust, mask_overlay_blur, skip_video_for_run_all, fps, output_format, ffmpeg_location, add_soundtrack, soundtrack_path, use_manual_settings, render_steps, max_video_frames, path_name_modifier, image_path, mp4_path, i1, i2, i3, i4, i5, i6, i7, i8, i9, i10, i11, i12, i13, i14, i15, i16, i17, i18, i19, i20, i21, i22, i23, i24, i25, i26, i27, i28, i29, i30, i31, i32, i33, i34, i35, i36)
File "D:\StableDiffusion\stable-diffusion-webui\extensions\deforum-for-automatic1111-webui\scripts\deforum.py", line 62, in run
from scripts.deforum.render import render_animation, render_input_video
File "D:\StableDiffusion\stable-diffusion-webui\extensions\deforum-for-automatic1111-webui\scripts\deforum\render.py", line 13, in
from .generate import generate, add_noise
File "D:\StableDiffusion\stable-diffusion-webui\extensions\deforum-for-automatic1111-webui\scripts\deforum\generate.py", line 23, in
from .animation import sample_from_cv2, sample_to_cv2
File "D:\StableDiffusion\stable-diffusion-webui\extensions\deforum-for-automatic1111-webui\scripts\deforum\animation.py", line 5, in
import py3d_tools as p3d
ModuleNotFoundError: No module named 'py3d_tools'

Add frame to frame interpolations

Google released their frame2frame interpolator this month under apache-2.0 license, so it may be feasible to include it into Deforum for transition smoothing

https://ai.googleblog.com/2022/10/large-motion-frame-interpolation.html?m=1

https://github.com/google-research/frame-interpolation

The only problem is that it requires Tensorflow to run, so either we can include it as is and just send the images via api, but it will require quite heavy Tensorflow installation along with PyTorch (easier) or translate the whole thing to PyTorch (harder and unlikely)


As hithere says on Discord, that way is too slow, so we could try implementing RIFE instead. It's under the MIT license and is compatibe with Deforum. Additionally, per hithere's tests, RIFE appears to be the best performing algorithm amongst RIFE, DAIN, FLAVR and XVFI, so it may be the best bet

https://github.com/megvii-research/ECCV2022-RIFE

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.