rewbs / sd-parseq Goto Github PK

Rather than having a separate SD script that does similar (but less / less mature) stuff compared to deforum, the UI could generate parameters for deforum (would require deforum script to have a mode that accepts a parseq-style JSON manifest)

https://github.com/deforum-art/deforum-for-automatic1111-webui

Aubio processes audio files and outputs a list of event timestamps. We could import such files and automatically generate keyframes, or add markers on the graph.

Rather than accepting a fixed number of prompts, allow a prompt template that can reference prompt_weight variables. For example the default would be:

my prompt 1 :${prompt_weight_1} AND
my prompt 2 :${prompt_weight_2} AND
my prompt 3 :${prompt_weight_3} AND
my prompt 4 :${prompt_weight_4}

Add share option to export to an unlisted pastebin doc, and a param to import from a remote pastebin id on load. This will allow easier sharing.

Currently batches and batch size are ignored (overridden to 1)

Currently neither of the following will do what you expect:

• 1 + 2 * 3 ---> 9
• 5 * (1+2) ---> parser error

Make it easier to see where on the grid a given graph point is and vice versa.

Currently with loop back, if say I want to do a 360° rotation from frames 0 to 100, if have to manually figure out this is a 3.6° per frame and set that as a fixed value for that range. If I want to ease in/out or oscillate, hitting 360° exactly is a pain in the ass.

We should render the delta/derivative values so that if I specify a smooth oscillation between 0 and 360°, parseq will figure out what the per-frame change should be and then the backend can use that instead of the absolute values.

Hardest part of the impl is figuring out the UX because you only want this behaviour on loopback, not vid2vid where prior transforms are not carried over.

Make the column chart editable for bonus points. :)

Include automated version increment.

Automated deploy to prod will come later. :)

Currently if all frames are set to -1, you'll get a random seed on every frame. This is likely not what people expect. Consider replacing -1 with a single random seed, or having a different random marker (e.g. random_value(N) to be able to create multiple fixed random values)

A feature that auto analises a mp3 or wav file you give it, or an easier way to create sound matching keyframes

According to chat on discord:

Provide a recommendation range of each parameter. For example, for strength, there is a range, write this on the site.

Also, for movements. How much is too much? If I put zoom 1000 for 1 frame, it will be artifact, but if this happens in 10000 frames gradually, then it's ok. So is there any formula that can calculate what is the safe range for for example rotation Z, before I generate my video and notice it's wrong and I need to do it again?

Is this something that parseq can calculate and warn us before we generate our video? For example it says, your zooming is too fast according to the numbers and probably you will get artifacts.

REWBS:
Including recommended ranges for each arg is a good idea. For some fields it'll be straight forward, but for others it'll be a bit of a grey area (the Deforum docs aren't always clear on this for all params so it'll require some experiments and/or code digging). I love the idea of emitting warnings if any fields are obviously out of range, and possibly even clamping them.

I get this errors each time that I try to run the parsec script on img to img

Traceback (most recent call last):
File "E:\Dev\stable-diffusion-webui\modules\call_queue.py", line 45, in f
res = list(func(*args, **kwargs))
File "E:\Dev\stable-diffusion-webui\modules\call_queue.py", line 28, in f
res = func(*args, **kwargs)
File "E:\Dev\stable-diffusion-webui\modules\img2img.py", line 137, in img2img
processed = modules.scripts.scripts_img2img.run(p, *args)
File "E:\Dev\stable-diffusion-webui\modules\scripts.py", line 317, in run
processed = script.run(p, *script_args)
File "E:\Dev\stable-diffusion-webui\scripts\parseq_script.py", line 44, in run
[all_images, info] = Parseq().run(p, input_img=original_input_image_resized, input_path=input_path, output_path=output_path,
File "E:\Dev\stable-diffusion-webui\scripts\parseq_core.py", line 93, in run
ffmpeg
File "E:\Dev\stable-diffusion-webui\venv\lib\site-packages\ffmpeg_run.py", line 284, in run_async
return subprocess.Popen(
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2288.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 971, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2288.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 1440, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] El sistema no puede encontrar el archivo especificado

I cant seem to make something longer than 119? is it limited?

Add BPM / frames per beat and time sigs as a 1st class concept so that:

• Frame positions can be shown as beat positions.
• Oscillator periods can be defined by beat.
• Visualiser grid can have indicators for beats / bars.
• Keyframes can be auto-generated for every beat / bar.
• Keyframes can be highlighted for every beat / bar.

Currently adding iterable parameters requires a code change.

Making them customisable (with a list of built in profiles) would make parseq much more flexible and adaptable to dependency changes.

Add checkbox to run in dry-run mode (which skips SD processing, can generate 100s of frames per second).

Consider generating 2nd companion video which renders parameter values.

Beat 0 isn't always on frame 0. Allow users to specify the offset (in frames or ms) of the first beat. Adust all beat/frame conversions accordingly.

Parseq currently overwrites most Deforum settings. It is very powerful for keyframe animations but it would be good to have the possibility to keep using all other deforum settings and only use parseq for camera movements.

3D rotations and zooming out naturally result in blank space in the input image. SD doesn't generally generate much in that space, especially with low denoising.

Look into border wrapping with perspective warping. This might not generate the right kinds of results and probably should be an option.

Look into how outpainting works. We may want to use the same technique.

To think about: does the noise need to be transformed in any way? Does it make conceptual sense to zoom/rotate the noise in accordance with the rest of the image?

UnboundLocalError: local variable 'parseqName' referenced before assignment

Allow dragging points up and down, and modifying interpolation formulae directly on the graph.

slate.js looks like a good component to try out: https://www.slatejs.org/examples/code-highlighting

ParseqUI is a bloated and would benefit from being broken down into subcomponents. Furthermore, the lack of types is annoying, so this would be a good opportunity to start converting those subcomponents to typescript.

hey there, I did every step with installing this parseq script but it doesn't show up below in keyframes by deforum.

after doing the steps I have closed the webui multiple times

Hey Rewbs, I've tried to render out some keyframes but I am perpetually stuck with "Please render to update the output" in the output no matter what. There is no error being thrown and I tried locally just in case, same issue. Since I can't render I haven't been able to see the visualized parameter flow either. The sparklines, when selected, just whites out the screen. Let me know if there is anything else you need to know. Thanks.

I.e if onset event 1 is dragged past onset event 2, it should become onset event 3

Initial thoughts for performance goals:

• We'll work with a dataset with 2000 frames, 20 keyframes, and a range of interpolation functions over 10 fields with points at 10 keyframes each.
• Baseline machine will be a 2018 macbook pro.
• Rendering can remain a relatively expensive operation but should not exceed ~500ms.
• The following non-rendering operations should always be snappy (<100ms):
  • Editing a value in the grid (without re-rendering).
  • Dragging a node in the graph (excluding re-rendering).
  • Changing the set of visible fields (should not trigger a re-render).
  • Changing any config values (excluding re-rendering).
  • Switching between percent and absolute values.
• Rendering with the default/demo keyframe values should be <50ms, so that auto-rendering can be enabled by default.

The A1111 Deforum extension integration is working well, but many users prefer plain old Deforum :)

There's no reason we can't do a Parseq integration there too.

Fixed mask or video/frame list with equal number of frames.

At the moment it's hard to see when key events occur on the grid and in the graph. Coloured markers would improve this.

Allow users to explicitly convert expression results between temporal offset values in parseqlang (f, b and s suffixes currently just work on number littérales)

Feature request for a button in the "output" section that copies the manifest to clipboard.

Currently keyframes are anchored to absolute keyframe offsets. Therefore, if you want to change the fps of your video it will most likely mess up sync. It would be good to have a way to solve this. E.g. option to make beats or time offsets the anchored temporal axis?

I am testing (congratulations for the work), for the moment only the execution of the "demo" in https://sd-parseq.web.app/?docId=doc-f9bbde46-fd5a-4c03-84b5-fb20efd7ae20, in Automattic webUI. I copy the generated json and paste it in the indicated place. When I hit generate I get the following error:

AttributeError: 'ParseqAnimKeys' object has no attribute 'kernel_schedule_series'

I don't know if I am forgetting some parameter or I am doing something wrong or it is some kind of error.

PS: The default animation in the extension runs correctly, I insist, without touching any configuration.

Thanks, and again great work!

Basic zooming and panning shouldn't be too hard to add since there are various plugins that do this for chart.js.

In fact, one is enabled right now but it doesn't work due to the click event being used to support adding/dragging nodes. The challenge is figuring out how to stop them conflicting (may be as simple as setting a modifier key).

Is the generated URL meant to change each time? I'm finding that the file I'm working on at the moment seems to change on every upload event, so i need to make sure that the manifest URL in Deforum GUI is updated each time I make changes... I am pretty sure it was maintaining the same URL upload to upload for a single document in the past...

I installed ffmpeg and followed the rest of your instructions, but I'm getting these errors upon launch:
Error loading script: parseq_core.py
Traceback (most recent call last):
File "A:\Desktop\00 AI Images\stable-diffusion-webui\modules\scripts.py", line 159, in load_scripts
exec(compiled, module.dict)
File "A:\Desktop\00 AI Images\stable-diffusion-webui\scripts\parseq_core.py", line 10, in
import ffmpeg
ModuleNotFoundError: No module named 'ffmpeg'

Error loading script: parseq_script.py
Traceback (most recent call last):
File "A:\Desktop\00 AI Images\stable-diffusion-webui\modules\scripts.py", line 159, in load_scripts
exec(compiled, module.dict)
File "A:\Desktop\00 AI Images\stable-diffusion-webui\scripts\parseq_script.py", line 8, in
from scripts.parseq_core import Parseq
File "A:\Desktop\00 AI Images\stable-diffusion-webui\scripts\parseq_core.py", line 10, in
import ffmpeg
ModuleNotFoundError: No module named 'ffmpeg'

Scheduling a ton of prompt changes leads to the parseq gui crashing to error code: out of memory

Occasionally an edit to the keyframe grid is ignored/lost after hitting enter. Not sure why.