fkunn1326 / openpose-editor

Openpose Editor for AUTOMATIC1111's stable-diffusion-webui

Openpose Editor

日本語 | English | 简体中文

Automatic1111/stable-diffusion-webui用のOpenpose Editor

• ポーズの編集
• ポーズの検出

ができます

• 「Add」: 人を追加する

• 「Detect from image」: 画像からポーズを検出する

• 「Add Background image」: 背景を追加する

• 「Save PNG」: PNGで保存する

• 「Send to ControlNet」: Controlnet拡張機能がインストールされている場合、画像をそこに送る

インストール方法

1. "Extension" タブを開く
2. "Install from URL" タブを開く
3. "URL for extension's git repository" 欄にこのリポジトリの URL (https://github.com/fkunn1326/openpose-editor.git) を入れます。
4. "Install" ボタンを押す
5. WebUIを再起動する

注意

ControlNetの "Preprocessor" には、何も指定しないようにしてください。

エラーの対策

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>

以下のファイルを開いてださい

/Applications/Python\ $version /Install\ Certificates.command

openpose-editor's Issues

ページが応答しない / Page hangs a lot

The page is reported by Chrome as being unresponsive, usually after moving some dots, and I have to wait a lot time until it stops being frozen so I can move points again or do anything. This is very inconvenient. Now the page just crashed with Error code: Out of Memory, maybe you have a memory leak in there.

Feature Request: way to delete/remove some dots

while in the editor i wish there was a way to delete or remove some of the dots

Multi-ControlNet doesn't work

ControlNet switched to tabs for Multi-ControlNet. Now send to... only works with the first one.

Use "Ellipses" instead of "Lines" to connect the joints.

Is your feature request related to a problem? Please describe.

Thank you so much for creating this posable OpenPose model extension, it is a great resource for me and others in the community.

I have noticed that the extension currently uses straight lines to represent the "bones". However, I believe that using ellipses for the bones would be more aligned with the data the ControlNet model was trained on, and in my tests, I get better results using the thin ellipse shapes instead of straight rectangular lines for the bones connecting the joints.

Describe the solution you'd like
Use ellipses instead of rectangular lines to connect the joints of the skeleton.

Additional context
This is a link to the ControlNet Paper showing the data it was trained on, on page 15:
https://arxiv.org/pdf/2302.05543.pdf (Page 15)
https://arxiv.org/abs/2302.05543 (Source)

I'm sorry that I don't have the ability to code this myself, I currently use photoshop which can be time consuming when moving each layer manually. I understand that this change may require additional work, but I wanted to share my suggestion in the hopes that someone with the proper abilities could create it.

Thank you for your continued efforts in maintaining and improving the project!

Paste image from Clipboard option

Paste image from Clipboard option for "detect from image" and "add background image".

Thank you. :D

[Question] Is there a size / Dimension limitation to the detect from Image function?

Is there a size / Dimension limitation to the detect from Image function?

Addition of Text Editor for depth map (for crisp text generation)

Is your feature request related to a problem? Please describe.
Text generation with stable diffusion is an issue with random letters an shapes appearing instead of clear words.
Using ControlNet, if we add a "fake" depth map with our white text on black background, and an inpainting mask for the space we want the text to appear in (should be the same spot on the fake depth map), we can have crisp, clean text based on our prompt.

Describe the solution you'd like
I saw that Fabric.js already has Text tools in its library, so maybe adding a small text editor (font size, font family dropdown) with the ability to drag and resize the text on the canvas to send it as a depth map to inpainting and controlnet, could allow us to have consistent text without the need of photoshop or other photo editors.

Additional context
here is the technique of using depth maps for clear text: https://www.youtube.com/watch?v=1jBnGx72mpg
also leaving this jsfiddle here as an example https://jsfiddle.net/gislef/Lvfpq57h/
But maybe all of this should be a separate project on github / tab on a1111

Feature Request: Delete segments

It would be nice to be able to delete segments directly in the editor rather than having to save to PNG then do it in a paint program.

Deleting segments has a lot of uses, here are some:

1. On images where certain body parts are out of frame
2. On images where body parts occlude others, ControlNet doesn't handle this well and removing the segments that are behind produces better results
3. On complex poses, especially with overlaps or large foreshortening, ControlNet suffers and deleting key segments can help a lot
4. Using some segments but not others allows you to "pin" certain body parts while still allowing random variation in others during generation

OpenPose editor txt2img and img2img error message

After adjusting the human skeleton in the OpenPose editor, clicking on "txt2img" to transfer the image to ControlNet and selecting "none" for preprocessing, the image cannot be generated and an error is reported in the console. However, adding the skeleton image manually from ControlNet allows for successful generation of images. It seems that there is a problem with the transfer of images from OpenPose to ControlNet, and no images can be generated thereafter. Disabling the ControlNet function does not solve the problem, and the only solution is to refresh the page with F5 to restore normal functionality. I can confirm that the ControlNet function is working properly and that both manual drawing and importing of skeletons can be used to complete the drawing. The problem only occurs when using the OpenPose editor's txt2img and img2img functions.

=============================================
Console error message：

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "E:\AI\novelai-webui\novelai-webui-aki-v2\py310\lib\site-packages\gradio\routes.py", line 337, in run_predict
output = await app.get_blocks().process_api(
File "E:\AI\novelai-webui\novelai-webui-aki-v2\py310\lib\site-packages\gradio\blocks.py", line 1013, in process_api
inputs = self.preprocess_data(fn_index, inputs, state)
File "E:\AI\novelai-webui\novelai-webui-aki-v2\py310\lib\site-packages\gradio\blocks.py", line 911, in preprocess_data
processed_input.append(block.preprocess(inputs[i]))
File "E:\AI\novelai-webui\novelai-webui-aki-v2\py310\lib\site-packages\gradio\components.py", line 1830, in preprocess
ff.run()
File "E:\AI\novelai-webui\novelai-webui-aki-v2\py310\lib\site-packages\ffmpy.py", line 98, in run
raise FFExecutableNotFoundError(
ffmpy.FFExecutableNotFoundError: Executable 'ffmpeg' not found

==========================================

How to Use?

I create and adjust my pose,
send to controlnet,
select and activate ControlNet
select openpose for model and preprocessor
enter a prompt and hit generate
the render looks nothing like the pose i created

Feature request: send to Img2img Controlnet (too)

Currently Send To ControlNet button only sends the image to the txt2Img tab. Either you should have a separate button to Send to img2img ControlNet or replace both Controlnet images.

skeleton windown dont showing

Hello i have just installed open pose,
im on Commit hash: 0cc0ee1bcb4c24a8c9715f66cede06601bfc00c8
I have updated pip and pytorch
i have tried 3 times but always in open pose editor tab the screen where skelleton should appear is blank, just a white border and nothing inside, i have disbale ad blocker, disable and enable GPU accel, but i dont know if this have any to do with the last Automatic Commits.

inspector shows may warnings and this error:

Uncaught SyntaxError: Identifier 'executed' has already been declared (at main.js?1676926043.611049:1:1)

thank you.

Add pose Presets

Is your feature request related to a problem? Please describe.
Its actually very difficult to use openpose for certain poses that are directly facing you. Standing with hands on hips is easy. Crouched sideways much harder.

Describe the solution you'd like
Adding simple presets like sideviews, crouched, etc would help. Also then other could share their presets. Also a way to rotate the whole skeleton around any of its axis accurately would be nice,

thanks, this is awesome for art.

[Feature Request] When adding detect from image may I suggest that the UI assume the new image Size

When adding detect from image may I suggest that the UI assume the new image Size

Multi ControlNet update

Hi,

the extension stopped working after Multi ControlNet update 9831aa30
it would be useful to be able to decide which ControlNet tab to send the pose to (ControlNet - 0, ControlNet - 1)

thank's for your job

Fit to frame, when adding a background image.

When adding a background image now, if the image's resolution is larger than the working area the background image is displayed in its original resolution which means if it's larger than 2048x2048 there's no way to fit it in the working area.
Would be useful if we could move, scale, crop the background image or at least fit it into the working area.

My Open Pose Editor Tutorial

Just published today

Thank you for the extension

18.) Automatic1111 Web UI - PC - Free
Fantastic New ControlNet OpenPose Editor Extension & Image Mixing - Stable Diffusion Web UI Tutorial

And here my other tutorials for those who are interested in

1.) Automatic1111 Web UI - PC - Free
Easiest Way to Install & Run Stable Diffusion Web UI on PC by Using Open Source Automatic Installer

2.) Automatic1111 Web UI - PC - Free
How to use Stable Diffusion V2.1 and Different Models in the Web UI - SD 1.5 vs 2.1 vs Anything V3

3.) Automatic1111 Web UI - PC - Free
Zero To Hero Stable Diffusion DreamBooth Tutorial By Using Automatic1111 Web UI - Ultra Detailed

4.) Automatic1111 Web UI - PC - Free
DreamBooth Got Buffed - 22 January Update - Much Better Success Train Stable Diffusion Models Web UI

5.) Automatic1111 Web UI - PC - Free
How to Inject Your Trained Subject e.g. Your Face Into Any Custom Stable Diffusion Model By Web UI

6.) Automatic1111 Web UI - PC - Free
How To Do Stable Diffusion LORA Training By Using Web UI On Different Models - Tested SD 1.5, SD 2.1

7.) Automatic1111 Web UI - PC - Free
8 GB LoRA Training - Fix CUDA Version For DreamBooth and Textual Inversion Training By Automatic1111

8.) Automatic1111 Web UI - PC - Free
How To Do Stable Diffusion Textual Inversion (TI) / Text Embeddings By Automatic1111 Web UI Tutorial

9.) Automatic1111 Web UI - PC - Free
How To Generate Stunning Epic Text By Stable Diffusion AI - No Photoshop - For Free - Depth-To-Image

10.) Python Code - Hugging Face Diffusers Script - PC - Free
How to Run and Convert Stable Diffusion Diffusers (.bin Weights) & Dreambooth Models to CKPT File

11.) NMKD Stable Diffusion GUI - Open Source - PC - Free
Forget Photoshop - How To Transform Images With Text Prompts using InstructPix2Pix Model in NMKD GUI

12.) Google Colab Free - Cloud - No PC Is Required
Transform Your Selfie into a Stunning AI Avatar with Stable Diffusion - Better than Lensa for Free

13.) Google Colab Free - Cloud - No PC Is Required
Stable Diffusion Google Colab, Continue, Directory, Transfer, Clone, Custom Models, CKPT SafeTensors

14.) Automatic1111 Web UI - PC - Free
Become A Stable Diffusion Prompt Master By Using DAAM - Attention Heatmap For Each Used Token - Word

15.) Python Script - Gradio Based - ControlNet - PC - Free
Transform Your Sketches into Masterpieces with Stable Diffusion ControlNet AI - How To Use Tutorial

16.) Automatic1111 Web UI - PC - Free
Sketches into Epic Art with 1 Click: A Guide to Stable Diffusion ControlNet in Automatic1111 Web UI

17.) RunPod - Automatic1111 Web UI - Cloud - Paid - No PC Is Required
Ultimate RunPod Tutorial For Stable Diffusion - Automatic1111 - Data Transfers, Extensions, CivitAI

Detect from image not working

I've tried the detect from image feature on several images but got nothing sent to the editor, is it still experimental or are there specific criteria I need to be aware of when selecting images?

Symmetry & lock length

Hello fkunn1326,

First off - thanks for this extension. It is MUCH appreciated.
This is a game changer for SD and Openpose-Workflows.

I have some ideas I hope you'll be able to implement:

1. Edit Limbs (arms, legs) with Symmetry (if possible turned on / off with a button or right clicking on the canvas)

2. Mode that locks the length of a segment when posing points / joints (either all at once or individually, if possible)

3?) Is there a way in openpose to differentiate different skeletons? With slight hue-changes to the openpose skeletons for example?

Thank you for your hard work.

Txt2Img renders a skeleton and not an image posed as the sekelton

In my case, I have two skeletons (but have tried with just one). I post them and then hit the Send to ControlNet button. That sends my skeleton into the ControlNet section of the Txt2Img tab. I check that ControlNet is enabled and canny is selected for the preprocessor and model. The result is the skeleton with some shading and color. My prompt is "a photo of two men fighting"

MMPose support in the future?

Is your feature request related to a problem? Please describe.
It would be nice to have both types of figures, and I believe the t2iadapter one is different.
Describe the solution you'd like
MMPose figure addition if possible.

Additional context
MMPose figure.

Good for combining with depth, faster and less memory.

ffmppeg ffmpy bug

had to revert to an earlier commit for it work. see here: AUTOMATIC1111/stable-diffusion-webui#8211

Overlapping skeletons

Skeletons lack some kind of detection of the the depth of characters, determine which are closer, which are farther, and what limbs or body parts are behind the body or objects. For example if the model put her hands behind back, it thinks just that they are not visible and make non visible parts random but in front, not behind. And same for characters that are behind other characters, it tries to draw them not behind. Or maybe there is a way to show that limb or part body are behind something with this pose skeleton? Maybe make a special coloration, that turns brighter \ different color if it is closer\further to viewer or if it is behind something.

open pose is not usable on mobile

open pose is not usable on mobile, when I try to move the limbs around, it select part and does not move them.

Maybe you can find something to make openPose work on smartphone.

Feature request: 3D view

This will enable creating the same scenes from different perspective.

[Bug] When there are no legs we get crazy results

See image

[Bug] Incorrect Position of Nodes when Manipulating Flipped Pose

Describe the bug
Sometimes we want flipped pose when the back of body is heading to viewer. However, when trying to flip the body, the node position stays unflipped while the lines between nodes behave correctly.

To Reproduce
Steps to reproduce the behavior:

1. Generate a normal pose
   
2. drag the handle to horizontally flip the pose. you can see the nodes stay unflipped, but the lines position are correct.
   
3. if you trying to move the node, the lines will reconnect to the node again.
   

Expected behavior
The node should be flipped correctly like the lines.

Environment

• OS: Tested on Windows 11, Android
• Browser: Edge, Chrome, Android Webview

Additional context
None

Select ALL

Is your feature request related to a problem? Please describe.
Making closeups is almost impossible

Describe the solution you'd like
Keyboard shortcut to select all, or douple-click on a character to select all of its bones.
Selecting, scaling, rotating and moving through keyboard shortcuts.

Colors of body lines are reversed

Referring to this image which shows a figure facing the camera, your body line colors are horizontally reversed from what they should be. The head is correct as is.

Feature request: Rotate

Currently only Scale and Move operations are working.

Is it possible to save the adjusted skeleton position?

There will be deviation when identifying the skeleton from the image, after adjusting it and sending it to ControlNet, after switching back to the openpose-editor tab again, the originally adjusted skeleton will change, and I hope to save the skeleton position.

'Send to ControlNet' button is not showing up

After installation, the 'Send to ControlNet' button is not showing up on the plugin. Could you help me with this?

[Question] Why the image size is different between "save PNG" and Canvas setting?

Hi,

In openpose editor, I setting for example is 512x512
But when I press "Save PNG", the save image size change to 704x704?
Seem it will *1.375?

send to txt2img doesn't work When I use the Chinese language pack

send to txt2img doesn't work When I use the Chinese language pack
When I use English, it works.
https://github.com/dtlnor/stable-diffusion-webui-localization-zh_CN
当我使用中文语言包的时候，发送到文生图不起作用
当我把语言切换到英语时，又正常了。

Pose ignored

Hi. Thanks for sharing this project with the community.

I tried the editor but I must be doing something wrong:

1. I start the OpenPose Editor and arrange the pose I like.
2. I save pose.png
3. I go to the img2img tab and load pose.png to both the img2img upload area and the ControlNet upload area.
4. I set Denoising Strength to 1 in the img2img settings
5. I enable ControlNet and select openpose as Preprocessor and control_sd15_openpose [fef5e48e] as model.
6. I select v1-5-pruned-emaonly.ckpt [cc6cb27103] as my Stable Diffusion model.

The images generated are both incredibly dark or black and the pose is completely ignored, no matter the prompt I use.

So I thought "OK, that's correct because the input image is a black canvas with the pose."
But then, I get the same results if I load pose.png in the ControlNet section of txt2img tab.

What am I doing wrong?

Hand Pose

Is your feature request related to a problem? Please describe.
Is it possible to add hand pose also?

Describe the solution you'd like
Create hand pose editor, maybe a dropdown menu to select between body pose only and hand pose also

Feature request: Select all (joint points)

I often re-create poses from full body shots. For this, I mostly scale the image up and refine the pose. Then, mostly I want to reduce to, say, a portrait size. I shrink the preview image to the final size (536x768). But after that, I can't select all joints to resize it to match.
So, it would be nice to have a button "Select all" to get all joints selected, no matter if outside the frame or not.
Thanks.

Feature request: is it possible to add hand

https://twitter.com/Artoid_XYZ/status/1627670405849960448

オブジェクトオブジェクトのスケルトンのみを追加できますか?

オブジェクトオブジェクトのスケルトンのみを追加できますか?

Add annotated instruction images to readme

Request to add annotated images explaining purpose of individual dots.
Easiest to add to the read.me, would be nice to see it in the extension as some kind of hint.

[Bug] dots and lines should not change their size

problem demonstration:

bug.mp4

Feature request: multi person detect and clone

The ControlNet extension already supports multi-person detection, but unfortunately your extension does not (to scan the Detect from image button).
I'd like you to support it, because I have some images with multiple people in them.
Also it would be nice if you could clone a figure that is already inserted, because I would only change a few movements.

[Feature Request] Inverse Kinematics

Would love to be able to use inverse kinematics when moving joints, it would be so much easier to edit the pose. One way I see this could work is to rotate i.e. the wrist along with the elbow when moving the elbow.

This may have the effect of making it easier to make 2D poses, but at this stage the controlnet models are not precise enough for this to matter AFAIK.

Multi-ControlNet support

Can you please support Multi-ControlNet for Openpose Editor?

Currently, it is possible to send a stickman picture to the (single) ControlNet panel in the txt2img or img2img tabs.

However, with Multi-ControlNet enabled, it is not possible to send a stickman picture because it is divided into multiple panels such as ControlNet - 0, ControlNet - 1, ControlNet - 2....

With Multi-ControlNet enabled, I think it is necessary to be able to select which ControlNet panel to send the stickman picture to.

Inpaint compatibility?

First of all I have to say I have been enjoying this extension greatly. I just wonder (as a non-coder) if it would be possible to make it compatible with inpaint models.

Currenty you get a channel expectation error if you try and the workaround is extremely tedious and time consuming.

If such a feature isn't possible it would be nice to hear as to not get my hopes up needlessly.

Thank you in advance-

nothing is sent when clicking "send to txt2img" or "sent to img2img'

To Reproduce
Steps to reproduce the behavior:

1. Go to 'openpose editor'
2. Click on 'send to txt2img'
3. It automatically goes to the txt2img tab, but found nothing in ControlNet
   send to img2img has the same bug

Expected behavior
the openpose image is sent to the tab

Environment

• OS: windows
• Browser: Microsoft Edge

No "Send to ControlNet" option

Describe the bug
Hello guys, there is no "Send to ControlNet" option on my PC when openpose editor is installed.
Then I "send to img2img" to generate some picture, but it seems the pose is not loaded.

Screenshots

Environment

• windows

Cannot read properties of undefined (reading 'forEach')

Describe the bug
Clicking "detect from image" or "add background image" throws error in console.

To Reproduce
Click one of the buttons

Expected behavior
No errors

Additional context

Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'forEach')
    at st (index.4395ab38.js:76:2728)

Below, data is undefiend

function st(xt) {
    xt.data.forEach((dt,Gt)=>{
        if (typeof dt == "object" && dt !== null && dt.__type__ === "update") {
            for (const [zt,Lt] of Object.entries(dt))
                zt !== "__type__" && n(11, L[jt[Gt]].props[zt] = Lt, L);
            n(9, m)
        } else
            n(11, L[jt[Gt]].props.value = dt, L)
    }
    )
}

Extra parts/Other Skeletons

Is your feature request related to a problem? Please describe.
I was just thinking it might be useful to have either optional parts or even full skeletons for more fantasy body types. Like for instance, a model that has a part that could be posed for a tail, or maybe wings. Not sure if it's possible but I personally would like it as an option

Detect from image fails with TypeError: xd.data is undefined

Detect from image does not work in Firefox 88 and Chrome 110 (old versions) or Edge 109 (current), throwing the following uncaught error:

Firefox

Uncaught (in promise) TypeError: xt.data is undefined
    st http://127.0.0.1:7860/assets/index.4395ab38.js:76
[index.4395ab38.js:76:2720](http://127.0.0.1:7860/assets/index.4395ab38.js)
SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data
    json_observer http://127.0.0.1:7860/file=B:/stable-diffusion-webui/extensions/openpose-editor/javascript/main.js?1678118264.4358704:316
    initCanvas http://127.0.0.1:7860/file=B:/stable-diffusion-webui/extensions/openpose-editor/javascript/main.js?1678118264.4358704:312
    observer http://127.0.0.1:7860/file=B:/stable-diffusion-webui/extensions/openpose-editor/javascript/main.js?1678118264.4358704:527
    MutationCallback* http://127.0.0.1:7860/file=B:/stable-diffusion-webui/extensions/openpose-editor/javascript/main.js?1678118264.4358704:524
    EventListener.handleEvent* http://127.0.0.1:7860/file=B:/stable-diffusion-webui/extensions/openpose-editor/javascript/main.js?1678118264.4358704:523
[main.js:347:28](http://127.0.0.1:7860/file=B:/stable-diffusion-webui/extensions/openpose-editor/javascript/main.js?1678118264.4358704)
SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data
    json_observer http://127.0.0.1:7860/file=B:/stable-diffusion-webui/extensions/openpose-editor/javascript/main.js?1678118264.4358704:316
    initCanvas http://127.0.0.1:7860/file=B:/stable-diffusion-webui/extensions/openpose-editor/javascript/main.js?1678118264.4358704:312
    observer http://127.0.0.1:7860/file=B:/stable-diffusion-webui/extensions/openpose-editor/javascript/main.js?1678118264.4358704:527
    MutationCallback* http://127.0.0.1:7860/file=B:/stable-diffusion-webui/extensions/openpose-editor/javascript/main.js?1678118264.4358704:524
    EventListener.handleEvent* http://127.0.0.1:7860/file=B:/stable-diffusion-webui/extensions/openpose-editor/javascript/main.js?1678118264.4358704:523
[ma](http://127.0.0.1:7860/file=B:/stable-diffusion-webui/extensions/openpose-editor/javascript/main.js?1678118264.4358704)

Edge

Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'forEach')
    at st (index.4395ab38.js:76:2728)
2main.js?1678118264.4358704:347 SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at MutationObserver.<anonymous> (main.js?1678118264.4358704:316:31)

Input file: