Step1: Setup backend service
Set following command line arguments in webui-user.bat:
set COMMANDLINE_ARGS=--api --cors-allow-origins https://huchenlei.github.io [Rest of ARGS...]For SDNext(V1111) users, set following arguments:
set COMMANDLINE_ARGS=--cors-origins https://huchenlei.github.io [Rest of ARGS...]Step2: Click Window>Plugin
Step3: Search for stable-diffusion-ps-pea
🔥[New Feature][2023-11-26] Realtime rendering powered by LCM
Recent advancement in LCM(Latent Consistency Model) has significantly increased the speed of inference of stable diffusion. The inference time now can be so fast that we can do real-time rendering of the canvas.
Some preparations before you start exploring the real-time rendering tab:
- Make sure to download the latest version of
config_sharing/huchenlei_configs.json5and upload it in the config tab. The new config file provideslcm_base,lcm_lora_sd15,lcm_sd15_scribbleconfigs that are necessary. - Make sure you have LCM LoRA named
lcm_lora_sd15.safetensorin A1111. Or you can change the name of LoRA in configlcm_lora_sd15. You can download LCM LoRAs here.
After these preparations, you can now navigate to the real-time render tab (📹).
- Select
lcm_base,lcm_lora_sd15in RealtimeConfig. - Start drawing on canvas and enjoy!
Other features:
- If you have any selections on canvas, LCM will only render the selected area.
- You can add
lcm_sd15_scribbleto RealtimeConfig, which will invoke ControlNet scribble model on canvas content. Make sure you have solid black brush color active when scribbling. - You can click
Send to canvasto send the rendered view to canvas.
...More documentation work in progress...
Reference range selection
In A1111 img2img inpaint, one painpoint is that the inpaint area selection is either WholeImage or OnlyMasked. This
might not be an issue when the image is within reasonable size (512x512). Once the image becomes big (1024x1024+), the
time and resouce required for inpaint area to be WhileImage grows exponentially, which makes this option not viable, but
sometimes we do want to reference a limited range of surroundings. In this situation, one need to crop the image in an
image editor, ask A1111 to only process the cropped image, then put the cropeed image back to the original big image.
This is a tedious process, but now we have this behaviour as default in stable-diffusion-ps-pea. Everytime you do an
img2img, optionally you can apply a reference range (%/px), or you can just manually specify the range by creating another
selection on canvas.
Scale ratio In whole body generation, some body parts (hand/face) often becomes bad in quality, because there are just not enough pixels for the diffusion model to add details to. The diffusion model also performs less well on aspect ratios other than the ratios it was trained on (512x512 for SD1.5, 1024x1024 for SDXL), so doing inpaint in a small area only help a little. The solution is simple here, when inpainting a small area, we let A1111 target a bigger area closer to diffusion model's trained aspect ratio and resize the output to put the result image back to the original inpaint spot. The very popular extension ADetailer is doing this exact process but using image detection models to automatically detect face/hand/body to fix.
ControlNet Majority of ControlNet models can be applied to a specific part of the image (canny, depth, openpose, etc). However, in normal A1111 ControlNet UI, you cannot easily visualize the spatial relationship between each ControlNet unit.
One example is shown in following video. The author uses openpose to control body pose, and softedge to control hand
detail. Noting that he is using a image editor to edit the softedge map to keep only the hand part.
This type of operation now becomes very easy in stable-diffusion-ps-pea. The ControlNet maps can easily overlay
on top of each other. Here I am using a openpose unit and a lineart unit.
Initial proposal to implement layer control in ControlNet's repo: Issue #1736.
Config System One pain point about A1111 is that it is hard to define workflow. There are many configuration I wish can be restored later when I was using A1111. So here I designed a configuration system that let users easily define workflow.
There are 3 types of config in stable-diffusion-ps-pea:
- Base: The config representing the hardcoded default values for each generation parameters.
- Default: The default config when each time you enter the UI or click the
refreshbutton at the bottom right corner. Clicking the checkmark will activate the current selected config as default.
- Toolbox: The addon configs that only applied temporarily on the generation triggered by clicking the corresponding
toolbox button. This is where you can define you customized workflow.
Configs are defined as the delta to apply on top of the current UI states. Here are some examples I wrote and you can download config_sharing/huchenlei_configs.json5 and upload it in config panel to get access to them.
LamaGenFill: Use ControlNet's inpaint_only+lama to achieve similar effect of adobe's generative fill, and magic eraser. We accept JSON5 as config format, so you can actually add comment in config file.
"LamaGenFill": [
{
"kind": "E",
"path": [
"img2imgPayload",
"denoising_strength"
],
"lhs": 0.75,
"rhs": 1
},
{
"kind": "E",
"path": [
"img2imgPayload",
"inpainting_fill"
],
"lhs": 1,
"rhs": 3, // Inpaint fill is latent nothing.
},
{
"kind": "E",
"path": [
"img2imgPayload",
"inpaint_full_res"
],
"lhs": 0,
"rhs": 0, // Make sure inpaint reference range is whole image.
},
{
"kind": "A",
"path": [
"controlnetUnits"
],
"index": 0,
"item": {
"kind": "N",
"rhs": {
"batch_images": "",
"control_mode": 2,
"enabled": true,
"guidance_end": 1,
"guidance_start": 0,
"input_mode": 0,
"low_vram": false,
"model": "control_v11p_sd15_inpaint [ebff9138]",
"module": "inpaint_only+lama",
"output_dir": "",
"pixel_perfect": false,
"processor_res": 512,
"resize_mode": 1,
"threshold_a": 64,
"threshold_b": 64,
"weight": 1,
"linkedLayerName": ""
}
}
}
],Generative Fill using LamaGenFill workflow:
Magic Eraser using LamaGenFill workflow:
TileUpscale2x As previously demoed about scale ratio, this workflow is used to fix hand/face, and add details to the selected region.
"TileUpscale2x": [
{
"kind": "E",
"path": ["imageScale"],
"lhs": 1,
"rhs": 2,
},
{
"kind": "A",
"path": [
"controlnetUnits"
],
"index": 0,
"item": {
"kind": "N",
"rhs": {
"batch_images": "",
"control_mode": 0,
"enabled": true,
"guidance_end": 1,
"guidance_start": 0,
"input_mode": 0,
"low_vram": false,
"model": "control_v11f1e_sd15_tile [a371b31b]",
"module": "tile_resample",
"output_dir": "",
"pixel_perfect": false,
"processor_res": 512,
"resize_mode": 1,
"threshold_a": 1,
"threshold_b": 64,
"weight": 1,
"linkedLayerName": ""
}
}
}
],Here is a video demo using it: https://www.loom.com/share/fb11c0206d7045469b82fe9d6342bd15
Overall, the config system gives users full capability on A1111 API. Even the plugin does not build UI support for some
extensions, users can still invoke the extensions they want by setting entries of alwayson_scripts.
Interfacing with A1111: Optionally you can use https://github.com/yankooliveira/sd-webui-photopea-embed to send images between photopea and A1111.
The dev server needs to run under HTTPS because the plugin runs in an iframe that is embedded in an HTTPS environment. Using HTTP will make the browser complain about mixing HTTP/HTTPS content on a page.
Linux/Mac bash
openssl req -x509 -nodes -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -subj "/CN=localhost"
Windows bash
openssl req -x509 -nodes -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -subj "//CN=localhost"
Add --cors-allow-origins https://localhost:5173 to WebUI command line args for local development.
I do not make my dev plugin setup public as it might confuse user which plugin to install. So if you are planning to do development of this plugin, I kindly ask every developer to add their own dev plugin to photopea plugin store following these steps:
Step1: Click Window>Plugin
Step2: Click Add Plugin
Step3: Click New
Step4: Fill the Form
- File: upload
photopea_dev.jsonin project root directory - Thumbnail: Use any image link with proper size. I use
https://huchenlei.github.io/stable-diffusion-ps-pea/sd.png - Make sure to check
Make Public.
Step5: Install the plugin
You should be able to find the newly added plugin in the plugin store.
Step6: Make the plugin private
Go back to Step3 panel, and click Edit on the plugin you just added. Uncheck Make Public.
VSCode + Volar (and disable Vetur) + TypeScript Vue Plugin (Volar).
TypeScript cannot handle type information for .vue imports by default, so we replace the tsc CLI with vue-tsc for type checking. In editors, we need TypeScript Vue Plugin (Volar) to make the TypeScript language service aware of .vue types.
If the standalone TypeScript plugin doesn't feel fast enough to you, Volar has also implemented a Take Over Mode that is more performant. You can enable it by the following steps:
- Disable the built-in TypeScript Extension
- Run
Extensions: Show Built-in Extensionsfrom VSCode's command palette - Find
TypeScript and JavaScript Language Features, right click and selectDisable (Workspace)
- Run
- Reload the VSCode window by running
Developer: Reload Windowfrom the command palette.
See Vite Configuration Reference.
npm installnpm run devnpm run buildRun Unit Tests with Vitest
npm run test:unitRun End-to-End Tests with Nightwatch
# When using CI, the project must be built first.
npm run build
# Runs the end-to-end tests
npm run test:e2e
# Runs the tests only on Chrome
npm run test:e2e -- --env chrome
# Runs the tests of a specific file
npm run test:e2e -- tests/e2e/example.ts
# Runs the tests in debug mode
npm run test:e2e -- --debugRun Headed Component Tests with Nightwatch Component Testing
npm run test:unit
npm run test:unit -- --headless # for headless testingLint with ESLint
npm run lint
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent
