- Markdown is a lightweight markup language for creating formatted text using a plain-text editor. You can easily format your texts with Markdown format. Prompts can be difficult to organize. So with this GUI and CLI based solution you can easily structure your prompts with Markdown file format and get positive and negative prompts as seperate txt files.
- Now you can create datasets, train your prompt generator model, evaluate your model and generate prompts with it.
- The cursor updates in the CivitAITab and CLICivitai files are changed. If the nextCursor is not available in the json file, it adds to itself the total image count in the json file. The nextCursor is not available error may be occured if you do not use Newest for the sort variable so this update was made to fix this bug.
- CivitAI tab and CLI functionality is updated based on the update of the CivitAI REST API.
- Refactoring code base is done.
- Custom prompt generator node for ComfyUI is added. You can access the repository with this link.
- female_positive-gpt2-141972-5-1 prompt generator is added to Drive. Trained with gpt2 model, 141972 unique prompts, 5 epochs and 1 batch size.
- Improvements and bug fixes.
- WebUI is added. You can use the same functionalities with upscale image functionality by AIUpscale.
- Civitai Tab is added for GUI application.
- Civitai functionality is added.
- New texts have been added to understand what action is currently being taken.
- Virtual environment is gathered into one "venv" folder.
- CLI application is added.
- Auto-complete with
TABbutton functionality is added. - Reinstalling environment, updating packages in the virtual environment and passing virtual environment check functionalities are added.
- Virtual environments are seperated.
- Importing module optimizations are done.
- Update packages inside virtual environment functionality is added with command line argument.
- Reinstalling virtual environment functionality is added with command line argument.
- Optimization is done for importing modules.
- SelfRecursive algorithm is added.
- Preprocess is added for generating tab to get prompt more accurately.
- Adding datasets to previous datasets functionality is added
- Removing duplicates from datasets when adding new data functionality is added
- New models are trained and added to Google Drive link.
- Startup class is added. Automation commands are added for Linux environment.
- GUI is divided into tabs such as parse, dataset, train, evaluate, generate.
- Parse tab has the functionality of the parser which is translating prompts and parsing Markdown files.
- In the dataset tab, you can create prompt datasets to use in other tabs.
- In the train tab, you can train your prompt transformer such as GPT2.
- In the evaluate tab, you can get loss score of your model for the given dataset.
- In the generate tab, you can use your model to generate texts. You can enter seed text and configure recursive level.
- Virtual environment is using bat files to activate and deactivate.
- GUI appearance can be changed between light and dark mode.
- New translators are added but Google is the only one working without errors.
- Translator is added with deep-translator
- Modern GUI style is added with customtkinter
- Virtual environment automation is added. No need to configure manually. Just run
python start.pycommand. - Progress bar is added.
- Parsing and translation is done in threads so no freezing when parsing.
- Tested in Windows OS environment.
- Minimum 3.7 Python version is required for the GUI application.
- Tested with Python 3.10.6 and 3.11.3. As a note for Python 3.10.6 and newer versions Tkinter comes as default. You can easily test it with these commands:
- From command line enter
pythoncommand and press Enter button. - Write
import tkintercommand and press Enter button. If there are no errors. You are ready to go. - Write
exit()command and press Enter button to exit.
- Clone the repository with
git clone https://github.com/alpertunga-bile/prompt-markdown-parser.gitcommand. - Get into folder with
cd prompt-markdown-parsercommand.
- Run
python start.pycommand.
download_compressed.mp4
- Try
python start.pycommand. If it fails, run these commands
python -m venv venv source venv/bin/activate pip3 install -r requirements.txt deactivate
- Start the application with
python start.py --clicommand. This command is going to look for 'venv' file for virtual environment. It is going to setup the dependencies and start the application. After the first setup, it is just going to start the CLI application. - There are 8 commands in the main menu. These are parse, create, train, evaluate, generate, clear, cls and exit. With first 5 commands you can access to different menus. With clear and cls commands you can clear the terminal. With exit command you can terminate the application.
- You do not have to write all the commands you can write first 2 or 3 letters and press
TABbutton for auto complete.
- In parse section there are 3 commands, allParse, parse and exit. You can use
TABbutton here. - You have to have
promptsfolder to continue with allParse command. It will found all .md files underpromptsfolder and print the total files it can found. After that it is going to ask you to want to translate. You can use auto complete functionality here. Then it is going to parse all files and write them underpromptsfolder. - In parse command. You specify a filepath to parse and it is going to ask translation and after that it is going to parsed and saved to the same folder with .md file.
cli-parse.mp4
- There are three functionalities you can use.
- Files have to be in
datasetfolder and just write the filename not path. - Enhance functionality is for creating dataset. You can specify positive and negative filenames. Then you can specify image limit in [1, 200] range. Then specify image cursor to start, when to finish as hour and minute. Specify wanted and unwanted prompts with comma seperated. You can use default prompts. Select sort, period and nsfw parameters.
- Prune functionality is for clearing duplicates in files. Enter your filenames and wait for "Done!!!" text to show.
- Frequency functionality is for looking how many times a word is used. Specify filename and when it is done look for
dataset/frequency.txtfile.
cli-civitai_compressed.mp4
- In the create menu, you have to give dataset path. So how to create dataset file? It is quite easy actually, go to CivitAi site and find images you like. Then copy their link and paste in a txt file.
- Then specify positive filename. If you have already positive dataset you can specify it too but it has to be under the dataset folder.
- Then specify negative filename. If you have already negative dataset you can specify it too but it has to be under the dataset folder.
- You can write with or without file extensions.
- Aftert that wait for progress bar to finish and
DONE !!!text on the terminal.
cli-create.mp4
- Give your dataset path. Which can be your positive or negative datasets. Not the links dataset.
- Enter your model name for example
gpt2. You can found the model names. If you are going to use this site, write all the model names for examplebigscience/bloom-560m. - Enter epochs.
- Enter batch size.
- Enter model save folder name. The model is going to save under dataset folder. So just right the name of the folder you want to save.
- Wait for
DONE!!!text on the terminal.
cli-train.mp4
- Give your dataset path. Which can be your positive or negative datasets. Not the links dataset.
- Enter your model name for example
gpt2. You can found the model names. If you are going to use this site, write all the model names for examplebigscience/bloom-560m. - Enter the model path.
- Wait for
Evaluation Score (Loss)text on the terminal.
cli-evaluate.mp4
- Enter your model name for example
gpt2. You can found the model names. If you are going to use this site, write all the model names for examplebigscience/bloom-560m. - Enter the model path.
- Enter the minimum length that your generator can generate.
- Enter the maximum length that your generator can generate.
- Choose if you want do sample functionality.
- Choose if you want early stopping functionality.
- Specify recursive level.
- Choose if you want self recursive functionality.
-
Let's say we give
a,as seed and recursive level is 1. I am going to use the same outputs for this example to understand the functionality more accurately. -
With self recursive, let's say generator's output is
b. So next seed is going to beband generator's output isc. Final output isa, c. It can be used for generating random outputs. -
Without self recursive, let's say generator's output is
b. So next seed is going to bea, band generator's output isa, b, c. Final output isa, b, c. It can be used for more accurate prompts. -
Now there are 6 commands you can use. These are generate, set, print, clear, cls and exit commands. clear and cls commands are clearing the terminal and exit command is return you to CLI applications main menu.
- In the generate menu, you can enter seed and get the generated text.
- You can choose a variable to set its value. You can choose the variables that you specify before except model name and model path.
- Print the current variables values.
cli-generate_compressed.mp4
- Start the application with
python start.py --guicommand. This command is going to look for 'venv' file for virtual environment. It is going to setup the dependencies and start the application. After the first setup, it is just going to start the GUI application.
- You can choose Translate Prompts checkbox to translate your prompts to English. You can write prompts in a mixture of English and your native language. GoogleTranslator is used for translation. This checkbox works with Parse All Files In 'prompts' Folder and Parse And Save functionalities.
- Parse All Files In 'prompts' Folder button is getting all markdown files under 'prompts' folder which is located in repo directory and parse and save them.
- Select Prompt Files button is going to show a window to let you select your Markdown files. You can choose multiple files.
- After the selection press Parse And Save button and it's done. You can find your text files in the folder where are your selected Markdown files. You can see that it seperates negative and positive prompts with _negative and _positive names.
- DO NOT EDIT Positive Prompts and Negative Prompts strings in Markdown files. These are used to seperate prompts. You can add or reduce # symbol but do not edit the strings.
- DO NOT ADD comma "," at the end of your rows. Parser is adding for you.
gui-parse.mp4
- Save prompt links in a txt file. Just select image from CivitAi and copy and paste its link into a txt file.
- Choose your link dataset which is provided in txt file.
- Write positive and negative filenames as you want to name them.
- Click Create Dataset button and wait. You can watch the progress with progressbar in GUI and in terminal.
gui-create.mp4
- Locate
wantedPrompts.txtandunwantedPrompts.txtfiles under dataset folder and change for wanted and unwanted prompts. Seperate your prompts with comma. - Give positive and negative filenames for dataset. Do not give path. Files are going to be created in dataset folder.
- Select attributes.
- Select
Allfor NSFW if you want to get all images. - Click
Enhancebutton and wait for loading bar to complete in terminal then check dataset folder.
gui-civitai_compressed.mp4
- Enter your model name. You can found the model names.
- Enter epochs, batch size.
- Enter folder name for your model. Your model is going to be saved into 'dataset' folder.
- Choose dataset to train with model.
- Click Train button and wait for Done!!! text to appear above the button.
gui-train.mp4
- Choose your dataset which is used for training.
- Enter your model name that you used for training. DO NOT enter the folder name of your model.
- Choose your model's saved folder.
- Click Evaluate button and wait for a Evaluation score output above the button.
gui-evaluate.mp4
- Enter your model name that you used for training. DO NOT enter the folder name of your model.
- Choose your model's saved folder.
- Enter min length that model can generate.
- Enter max length that model can generate.
- Tick the checkboxes if you want these features.
- Set recursive level with slider. It is going to give model the previous result in each step.
- Select if you want self recursive.
- Enter your seed and click Generate Text button and wait for text to be appeared in the textbox.
gui-generate.mp4
- Let's say we give
a,as seed and recursive level is 1. I am going to use the same outputs for this example to understand the functionality more accurately. - With self recursive, let's say generator's output is
b. So next seed is going to beband generator's output isc. Final output isa, c. It can be used for generating random outputs. - Without self recursive, let's say generator's output is
b. So next seed is going to bea, band generator's output isa, b, c. Final output isa, b, c. It can be used for more accurate prompts.
- You can access the Markdown file that is used for the below image.
- Image is upscaled with AIUpscaleGUI project.
- Some typo fixes are done for the generated prompts.
- female_positive_gpt2-75_model is used for generator examples. Used model can be found here.
- Model is trained with 75 epochs and 1 batch size.
- Model Name : gpt2
- Min Length : 10
- Max Length : 50
- Features are off
- Recursive Level : 0
- Self Recursive : Off
- Seed : mature woman, mechanical halo
- Generated Prompt : mature woman, mechanical halo, (blue hair:1.2), intricate, high detail, sharp focus, dramatic, beautiful girl, RAW photo, 8k uhd, film grain, caustics, subsurface scattering, reflections, (cowboy shot:1)
- Model Name : gpt2
- Min Length : 10
- Max Length : 50
- Features are off
- Recursive Level : 1
- Seed : goddess
- Self Recursive : Off
- Generated Prompt : goddess, (intricate detailed skin texture:1.2), (electric spark, broken machine:1.1), (machine body:1.2), looking at the viewer, (smart sharpen:1.2), medium breasts, (Ghost in the Shell), depth of field, gradient background, backlit, rim lighting, dramatic lighting, ambient occlusion, volumetric lighting, professional studio lighting, closed mouth, insanely detailed,, ((masterpiece)), absurdres, HDR
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent