holahapi / koi

A plug-in for Krita that enables the use of AI models for img2img generation.

koi 🎣

koi is an open source plug-in for Krita that allows you to use AI to accelerate your art workflow!

Disclaimer ✋

In the interest of getting the open source community on board--I have released this plug-in early. In its current state you may run into issues (particularly during the setup process). If you do, I encourage you to open an issue here on GitHub and describe your problems so that it can be fixed it for you and others!

---

Overview 😄

The goal of this repository is to serve as a starting point for building increasingly useful tools for artists of all levels of experience to use.

Link to original twitter thread

This plug-in serves as a working example of how new A.I. models like Stable Diffusion can lower the barrier of entry to art so that anyone can enjoy making their dreams a reality!

Because this is an open source project I encourage you to try it out, break things, and come back with suggestions!

Getting Started 🏁

If you are new to git, or get stuck during the installation process, Lewington made a nice step-by-step video.

The easiest way to get started is to follow the plug-in installation process for krita. Then use the google colab backend server (button at the top of this readme)! This should give you a good introduction to the setup process and get you up and running fast!

---

Installation 🔨

Krita has a few plug-in installation methods, however, I will refer you to the one I use.

• Step 1: Find your operating system's pykrita folder reference
• Step 2: Clone the repository, and copy the koi folder and koi.desktop to pykrita. (restart krita now if it is open)
• Step 3: Open Krita and navigate to the python plug-in menu reference
• Step 4: Enable the koi plugin and restart Krita to load the plug-in.

The next thing you will need to do is setup the backend server that do all the computation!

• Step 0: Ensure you have a GPU-accelerated installation of pytorch. (You can skip this step if you are using Colab or already have it)

  • Follow the installation instructions on pytorch's official getting started.

• Step 1: Get the latest version of HuggingFace's diffusers from source by going to the GitHub repo

  • From here you can clone the repo git clone https://github.com/huggingface/diffusers.git & cd diffusers to move into the directory.
  • Install the package with pip install -e .

• Step 2: Install this package! I recommend moving out of the diffusers folder if you haven't already (eg. cd ..)

  • git clone https://github.com/nousr/koi.git, then cd koi and pip install -e .

Note 🙋

Before continuing, make sure you accept the terms of service for the diffusers model link to do so here.

Next, inside your terminal run the huggingface-cli login command and paste a token generated from here. If you don't want to repeat this step in the future you can then run git config --global credential.helper store. (note: only do this on a computer you trust)

• Step 3: Run the server by typing python server.py
  • If you did everything correctly you should see an address spit out after some time (eg. 127.0.0.1:8888)
• Step 4: Open Krita, if you haven't already, and paste your address into the endpoint field of the plugin
  • You will also need to append the actual API endpoint you are using. By default this is /api/img2img
  • If you are using all of the default settings your endpoint field will look something like this http://127.0.0.1:8888/api/img2img

---

Inference 🖌️

This part is easy!

• Step 1: Create a new canvas that is 512 x 512 (px) in size and make a single-layer sketch (note: these are temporary restrictions).
• Step 2: Fill out the prompt field in the koi panel (default location is somewhere on the right of your screen).
• Step 3: Make any additional changes you would like to the inference parameters (strength, steps, etc.)
• Step 4: Copy and paste your server's endpoint to the associated field
• Step 5: Click dream!

---

FAQ ❔

• What does koi stand for?
  • Krita Open(source) Img2Img: While support for StableDiffusion is first, the goal is to have this plug-in be compatible with any model!
• Why the client/server setup?
  • The goal is to make this as widely available as possible. The server can be run anywhere with a GPU (i.e. colab) and allow those with low-powered hardware to still use the plug-in!
• I'm getting an error with "set-size"
  • This usually happens when you either forget "/api/img2img" at the end of your endpoint when copying it into the plugin OR you have some issue with your backend server (check the output on your server for more information).

---

TODO

• Add colab backend example
• Flesh out UI
• Move to CompVis repo
• Add CI
• Abstract away drop-in models for the server
• Improve documentation
• Add DreamStudio API support
• Add support for arbitrary canvas size & selection-based img2img
• Add support for masks
• Offer more configuration options