hnmr293 / posex

Posex - Estimated Image Generator for Pose2Image

Posex - Estimated Image Generator for Pose2Image

Quick Start with Web UI

1. Install Mikubill/sd-webui-controlnet.
2. Install Posex (this).
3. Open Posex accordion in t2i tab (or i2i as you like). Enable Send this image to ControlNet checkbox. Editor will appear.
4. Configure ControlNet as below.

Preprocessor: none
Model: control_sd15_openpose

5. Make pose.
6. Generate images.

How to use

Click:      select body
Left Drag:  move joint (on joint)
            rotate camera (otherwise)
Right Drag: move whole body (if selected)
            move camera (otherwise)
Wheel:      zoom

Installation

• Web UI Extension
• Online (Github Pages)
• Standalone

Web UI Extension

Prerequirement: You need to install Mikubill/sd-webui-controlnet to use Posex with ControlNet Pose2Img.

Go to Extensions tab, then select Install from URL tab and input https://github.com/hnmr293/posex.

Or move to extensions directory and type git clone https://github.com/hnmr293/posex.

In webui, open Posex accordion in txt2img or img2img tab, then click a checkbox. The canvas will be opened.

Online

See Github Pages.

https://hnmr293.github.io/posex

Standalone

$ pip install -r requirements.txt
$ python app.py

Then open localhost:55502 or 127.0.0.1:55502 in your browser.

Example

Web UI

sample 1:

result:

sample 2:

result:

Standalone

History

v0.3 -> v0.4 features

• background setting
• save/load pose
• fixed camera roll

v0.2 -> v0.3 features

• Web UI extension

v0.1 -> v0.2 features

• copying the image to clipboard
• multiple bodies
• canvas size changing
• UI has become ugly >_<;

posex's Issues

posex UI show nothing

[Bug]: TypeError: Script.postprocess() takes from 3 to 5 positional arguments but 6 were given

Currently, the Posex extension doesn't show up, even when I click the "Send this image to ControlNet." checkbox.

But, I get this error in the console after every image generation, even if Posex isn't activated.

Error running postprocess: C:\SD\AUTOMATIC1111\stable-diffusion-webui\extensions\posex\scripts\posex.py
Traceback (most recent call last):
  File "C:\SD\AUTOMATIC1111\stable-diffusion-webui\modules\scripts.py", line 404, in postprocess
    script.postprocess(p, processed, *script_args)
TypeError: Script.postprocess() takes from 3 to 5 positional arguments but 6 were given

Feature: upload a image as coordinate

Let's suppose, I want a pose, a man sitting on a chair. I want to upload a image of a chair as coordinate.

Vertical lines when saving pose

Hello, I believe I've correctly followed the install instructions. However when saving a pose, it only saves an image with vertical lines. Same issue whether I save inside of posex or download pose to pc. All extensions are up to date.

Not sure what the problem could be, any help is appreciated.
Thank you!

I don't quite understand what the mistake is. Error loading script: posex.py

Error loading script: posex.py Traceback (most recent call last): File "C:\Users\123123123\A1111 Web UI Autoinstaller\stable-diffusion-webui\modules\scripts.py", line 218, in load_scripts script_module = script_loading.load_module(scriptfile.path) File "C:\Users\123123123\A1111 Web UI Autoinstaller\stable-diffusion-webui\modules\script_loading.py", line 13, in load_module exec(compiled, module.__dict__) File "C:\Users\123123123\A1111 Web UI Autoinstaller\stable-diffusion-webui\extensions\posex\scripts\posex.py", line 14, in <module> posex.set_save_dir(os.path.join(os.path.dirname(__file__), '..', 'saved_poses')) NameError: name '__file__' is not defined

All the data is in the log, why doesn't it work on my machine?

saving as a json...

Open Pose editor allows saving the data of the 18 points, x/y coordinates, as a json file.
Looking over your saved info in the png file, it's not nearly as simple, it's got camera, and 3D coordinates etc.
I realize you have much more data than Open Pose, but there is value to a shared data format (beyond just an image), and allowing manipulation of the pose data in ways other than your editor.

I realize I could extract the data from the png, and write a translator to open pose's format. Hoping you have access to the info in a more direct way (ie 2d coordinates in the display, which have already translated the 3d coordinates (and camera etc), into 2d coordinates.

cant not continue working 3day ago with 404

acttion:
i am going to click add to add pose.

expect : done

actutal: nothing working.

why:?
net::ERR_HTTP_RESPONSE_CODE_FAILURE in web console,it tell me the file 404: https://hnmr293.github.io/sm/a10f0fbff9ef9e01c8548236b6b20610cf5d5cba1f631d8ebab539add9c04974.map

The content displayed on the panel is incorrect

with the latest webui / ControlNet / posex,
and ControlNet enabled, no posex code modified,
the posex panel shows :
<"Send this image to ControlNet."> <"Target ControlNet number">
Run button
TextBox
TextBox
....
Run button
TextBox
TextBox

[Bug] Sends to All ControlNet Networks (Multi ControlNet)

ControlNet supports running multiple networks with different images on a single generation through the Multi ControlNet setting. Currently Posex's "Send this image to ControlNet" checkbox sends the pose to all enabled ControlNet networks at once, overwriting any images already uploaded. This prevents the use of Posex alongside other ControlNet models or additional images. It would be ideal to specify which network number (0 up to 9 I believe) to send the pose to. This specification is present in fkunn1326's OpenPose Editor https://github.com/fkunn1326/openpose-editor for reference.

general notes

very cool extension but need to make couple of notes:

• why run separate web server (flask) just to expose several apis to its own js code? webui already uses fastapi, you can register extra endpoints easily.
• installer in app.py is custom and prone to failures, but extensions framework already provides for installer features? why have custom installer?
• location of editor in scripts is strange as it requires a lot of scroll up/down. plus many users use much more narrow left pane and wide results pane. this should realistically be an extension, not a script (just like openpose is).
• given that ui is written in js, it doesn't have look&feel of webui. controls should be inplemented in gradio, only canvas with three.js should be js code.

Pose bones are "lines" instead of "ellipses"

Thank you so much for creating this posable OpenPose model extension, it is a great resource for me and others in the community.

I have noticed that the extension currently uses straight lines to represent the "bones". However, I believe that using ellipses for the bones would be more aligned with the data the ControlNet model was trained on, and in my tests, I get better results using the thin ellipse shapes instead of straight rectangular lines for the bones connecting the joints.

This is a link to the ControlNet Paper showing the data it was trained on, on page 15:
https://arxiv.org/pdf/2302.05543.pdf (Page 15)
https://arxiv.org/abs/2302.05543 (Source)

I'm sorry that I don't have the ability to code this myself, I currently use photoshop which can be time consuming when moving each layer manually. I understand that this change may require additional work, but I wanted to share my suggestion in the hopes that someone with the proper abilities could create it.

Thank you for your continued efforts in maintaining and improving the project!

Loading a saved set of poses and then trying to add a body will override "body_2" then "body_3"

When loading a pose it doesn't remember how many bodies are in the saved file so adding more replaces ones that are already done.

Posex panel is empty

copied poser images into poseX dir/Poses will not show up

there is a folder for poses in the PoseX_dir, i tryed to copy img/pose files into this Folder but no Poses images are showing up, i´m able to see poser img only when is save it in the PoseX ui interface.
I suppose there is a special file format for those pose file, care to enlighten us?

[Feature request] Integrate a simple Poser doll UI

https://webapp.magicposer.com/

Could it be possible to integrate something like this one?
And for the sketches, could there be a preset sketches for "correct anatomy"?

That is, for example, 2d characters has very big heads and head to body ratio is smaller.
Using that body sketch for realistic drawing would be incorrect.

Can there be a set of sketch for 2d and 3d images separately?
(actually that site have these preset dolls as well)

Doesn't working at all

After latest update

Havent been able to get posex to work for a month or so

Likely something I ma doing wrong, but I need some help. I have updates automatic1111, and all extentions, controlnet, etc. I click on the arrow to open posex, click on "Send to controlnet" I get an incomplete UI,

and this in the output:

Traceback (most recent call last):
File "O:\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 394, in run_predict
output = await app.get_blocks().process_api(
File "O:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1075, in process_api
result = await self.call_function(
File "O:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 884, in call_function
prediction = await anyio.to_thread.run_sync(
File "O:\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "O:\stable-diffusion-webui\venv\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "O:\stable-diffusion-webui\venv\lib\site-packages\anyio_backends_asyncio.py", line 867, in run
result = context.run(func, *args)
File "O:\stable-diffusion-webui\extensions\posex\scripts\posex.py", line 168, in f
v = fn(*args, **kwargs)
File "O:\stable-diffusion-webui\extensions\posex\scripts\posex.py", line 173, in all_pose
return json.dumps(list(posex.all_poses()))
File "O:\stable-diffusion-webui\extensions\posex\common\posex_utils.py", line 57, in all_poses
'name': img.text['name'], # type: ignore
KeyError: 'name

Posex Ui not working correctly

Posex is showing up in stable diffusion however I cannot use it since the ui seems to be bugged

I have installed control net/posex through the extension manager as can be seen below

This might be an issue on my part since I did not find many posts regarding this issue however I cannot find any reasons for why that might be since I followed the instructions and have done everything accordingly.

This is the only post going through the same issue as me : https://www.reddit.com/r/SDtechsupport/comments/11kh71g/posex_ui_not_appearing_correctly_on_sd/

AssertionError: Controlnet is enabled but no input image is given

I use the poseX and controlnet, but the poseX picture cant sent to controlnet.

Bug: ControlNet and Pose - Load order

Overview

Alright found out a funny interaction with ControlNet and Pose.
It's related to folder named and load order between these two.

What's happening?

I have both my extension folders named, Control-Net and Posex respectively.
• So what happening?
I was always having an issue related to Control-Net saying it was active but no image was used.
After a couple trial and errors, switching back and forth between Posex commits, I decided to a quick test.
Namely making pose "load" before control net by renaming their folders to 00-Posex and 01-Control-net.
For my surprise it worked, no longer those errors.

Commit where this happens:

233daf8e1460c90d24a0f8c21bc5ec712ee258df

And prior.

Additional Notes:

Could we have a retro-compatibility for gradio 3.16 ?
After most recent Posex updates starting from commit a1cdda Posex stopped working with gradio 3.16.
It shows up in WEBUI, but it doesn't launch when clicked in [ ] send to controlnet.

Automatic1111 was having too much issues, so I switched anapnoe fork.

• Thanks for your amazing work •

Feature: ctrl+z

Great job.

It would be amazing to have ctrl+z feature here.

How to rotate two poses separately?

Hi. I click on one but rotate both at once, how to do that separately?
I use https://hnmr293.github.io/posex/#

posex panle doesnt show, but......

Things went back to normal after going back to the March 7th version.
what did you do to her？

feature req: ability to move multuiple limbs at once and ability to merge two poses into a scene

This is awesome. thanks,

however i find i cant get the control i need if i want to make a character jump i have to move each part at a time. Be nice to select multiple limbs and move them at once.

also if i create 2 or more amazing poses, i cant seem to merge them both into the same scene. Would be incredible for more complicated scenes.

and a third req. the ability to rotate limbs around the current camera angle axis. for quick tuning of a pose.

ty again, awesome

rotate the whole body

I want to rotate the whole body, how do I do that

net

Nothing works bro

Hi, yes after looking for multiple solutions only I'm posting here. I tried downgrading also have canvas zoom installed but nothing works. can you update your extension so it'd work with latest version?

stopped working after UI update

Latest webui brakes posex

latest pull from this repo and auto webui on commit : a0d07fb5807ad55c8ccfdfc9a6d9ae3c62b9d211

It shows the "Send this image to ControlNet" option, but nothing happens when clicked.
I think this may have to do with new gradio, but everything is wonky right about no, so it could be something else.

these is nothing in tab

my version

is this broken?
最新版的插件不显示内容了，只有截图中的两个按钮，作者看看哪里出问题了，这个插件非常棒

waiting for unpkg.com / memory leak

Hello there, thank you for this tool! It takes a while to the files needed to make this works, there is also a memory leak

[Feature Request]: Variable Joint Size

Feature Requested
A way to change the radius of the joints, similar to the "Limb Width" slider.

---

Ref: PR #7
This is so much better than what I had in mind! Being able to change the width and Elliptic/Line state opens up so much versatility in the creative possibilities. Since you added this type of control, I've been experimenting with how the ControlNet model responds to these types of changes in the skeleton's form, and I think it would be useful to be able to change the circle/joint size too.

This has also inspired me to start thinking about how I can experiment even further through changing the joint and limb colors in photoshop, hiding/showing different and additional limbs and joints. New discoveries in interesting and unexpected ways.

Thanks for making this cool toy/tool and for adding this ellipse feature in an enhanced and non-constraining way.

posex panle doesnt show

Failed to execute 'observe' on 'MutationObserver': parameter 1 is not of type 'Node'. TypeError: Failed to execute 'observe' on 'MutationObserver': parameter 1 is not of type 'Node'.

f12 ， when click ‘Send this image to ControlNet.’ ， the error above show once more

Generate button doesn't work

After "removeChild fix" there is another bug now
If you enable and disable posex tab and click "Generate" it throws error:

posex-webui.js?1680351184.6494553:41 Uncaught (in promise) TypeError: Cannot read properties of null (reading 'click')
at posex-webui.js?1680351184.6494553:41:63
at new Promise ()
at js2py (posex-webui.js?1680351184.6494553:22:16)
at HTMLElement. (posex-webui.js?1680351184.6494553:311:23)

Vid:
https://youtu.be/wpECwgbiOgs

Env:

3D editor uses 100% CPU

After clicking the Send this image to ControlNet checkbox and the 3D editor appears, it uses 100% of the CPU if your browsers hardware acceleration is turned off. Please add some sort of FPS limiter to the 3D app if possible.

In Edge the setting is called "Use hardware acceleration when available".

How do all objects move? I want to make smartphone support, but there is no right mouse button

camera position and zoom is not saved to presets

when i load my presets then theyre all over the place, i have to manually set them up again cause armature is offscreen on some, or zoomed in too much, its cause pos and zoom is not saved to presets, can you please save them as well? so when you click on preset then what appears is exactly the same thing as on thumbnail, currenty it doesnt work too well, the pose itself is saved but not camera.

[Feature] Infer pose from an image

The other pose editor extension, fkunn1326/openpose-editor, supports pasting/uploading an image and it infers the pose skeleton from it (not very successfully most of the time). Same is done, kind of, by the openpose preprocessor from ControlNet (quite successfully).

It would be a killer feature to add that here. IMO it's hard and time-consuming to arrange a pose from scratch. Hopefully you can take code from somewhere already, you won't be inventing this from scratch

Personally my main workflow would be:

1. Upload image, get a pose
2. Fix it
3. Move/Zoom it around
4. Use it

[Request] Ability to hide nodes\Node Limbs

When creating the back view of someone, it helps to not show the eyes and nose node, is it possible to "disable" nodes, to hide them before it is sent to controlnet.

editor doesn't appear when I click send to control net

Not sure why it's happening. There's no error in the console. As of this post I'm up to date on commits.