Bitsandbytes Error about sd_dreambooth_extension

[d8ahazard]

line 796, in main

I see the issue. Should be fixed with 1ddfcfc

So far its fixed, yhea, it generated till 100 steps as a test and it successfully made an image and saved a ckpt file. However....it got stuck on a loop saying training complete? and it kept saving a ckpt file every 1 step.

Wheeee!

Allright, I'll se what I did wrong now. :P

[derekleighstark]

just got the same error.

[YakuzaSuske]

supposedly Nerdy Rodent said it had to do with export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH.
Something that it had to be in the pastebin for it to work or something. But i haven't figured it out.

bitsandbytes-foundation/bitsandbytes#52

[d8ahazard]

supposedly Nerdy Rodent said it had to do with export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH. Something that it had to be in the pastebin for it to work or something. But i haven't figured it out.

TimDettmers/bitsandbytes#52

I don't think that's the issue here. That's for WSL, we're trying to do this thing without WSL.

I think the issue is just that we need to do 'conda install cudatoolkit=11.3', but getting an issue running from my install script.

[YakuzaSuske]

supposedly Nerdy Rodent said it had to do with export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH. Something that it had to be in the pastebin for it to work or something. But i haven't figured it out.
TimDettmers/bitsandbytes#52

I don't think that's the issue here. That's for WSL, we're trying to do this thing without WSL.

I think the issue is just that we need to do 'conda install cudatoolkit=11.3', but getting an issue running from my install script.

Is there another way to install it, if it doesn't work?

[Evil-Dragon]

Is this because we don't use conda for the virtual environment? Basically you can't install cudatoolkit using pip which what the issue is. PyTorch doesn't install everything that this needs to run under Windows. I think.

Not an expert on this. I've tried all kinds of instructions from various sources and had no luck getting 8bit Adam to work in Windows with A111 venv.

[d8ahazard]

Is this because we don't use conda for the virtual environment? Basically you can't install cudatoolkit using pip which what the issue is. PyTorch doesn't install everything that this needs to run under Windows. I think.

Not an expert on this. I've tried all kinds of instructions from various sources and had no luck getting 8bit Adam to work in Windows with A111 venv.

Not 100%, but that's what I'm leaning towards as the issue? Seems like we might need to manually use conda to install cuda-toolkit to the venv, or figure out why it's failing from within the install.py script.

[Raibeat]

I fixed this by changing the line in cextension.py:

binary_name = evaluate_cuda_setup()

to hardcode the cuda DLL path to libbitsandbytes_cuda116.dll

[LaikaSa]

I think there is already a solution for this in Windows here: bitsandbytes-foundation/bitsandbytes#30 (comment)
Basically, you download 2 dll files from this repo: https://github.com/DeXtmL/bitsandbytes-win-prebuilt
and drop them both into webui's sitepackage bitandbytes folder, should be following: stable-diffusion-webui\venv\lib\site-packages\bitsandbytes
After that, edit and replace cextension.py file in the same folder with this: https://pastebin.com/jjgxuh8V
and go to cuda_setup folder, replace the main.py file with this: https://pastebin.com/BsEzpdpw
That is all, problem solved, I tried and no more CUDA_SETUP: WARNING!

[d8ahazard]

Found the problem, as LaikaSA pointed out.

I was copying the main.py to the wrong spot.

Should be fixed and working now on windoze with c42b4f3

[YakuzaSuske]

Found the problem, as LaikaSA pointed out.

I was copying the main.py to the wrong spot.

Should be fixed and working now on windoze with c42b4f3

Yhea it worked, but now i got CUDA out of memory on my 3080TI, however disabling Text encoder fixed it and now its training. Running at 11.7 GB /12. Getting about 3it/s

[YakuzaSuske]

Now i got this.

Traceback (most recent call last):
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\modules\ui.py", line 185, in f
res = list(func(*args, **kwargs))
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\webui.py", line 54, in f
res = func(*args, **kwargs)
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\extensions\sd_dreambooth_extension\dreambooth\dreambooth.py", line 256, in start_training
trained_steps = main(config)
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 796, in main
text_enc_model = CLIPTextModel.from_pretrained(args.model_name,
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\venv\lib\site-packages\transformers\modeling_utils.py", line 1966, in from_pretrained
config, model_kwargs = cls.config_class.from_pretrained(
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\venv\lib\site-packages\transformers\models\clip\configuration_clip.py", line 133, in from_pretrained
config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\venv\lib\site-packages\transformers\configuration_utils.py", line 559, in get_config_dict
config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\venv\lib\site-packages\transformers\configuration_utils.py", line 614, in _get_config_dict
resolved_config_file = cached_file(
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\venv\lib\site-packages\transformers\utils\hub.py", line 424, in cached_file
raise EnvironmentError(
OSError: Luvelia is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with use_auth_token or log in with huggingface-cli login and pass use_auth_token=True.

[d8ahazard]

line 796, in main

I see the issue. Should be fixed with 1ddfcfc

[YakuzaSuske]

line 796, in main

I see the issue. Should be fixed with 1ddfcfc

So far its fixed, yhea, it generated till 100 steps as a test and it successfully made an image and saved a ckpt file.
However....it got stuck on a loop saying training complete? and it kept saving a ckpt file every 1 step.

[YakuzaSuske]

line 796, in main

I see the issue. Should be fixed with 1ddfcfc

So far its fixed, yhea, it generated till 100 steps as a test and it successfully made an image and saved a ckpt file. However....it got stuck on a loop saying training complete? and it kept saving a ckpt file every 1 step.

Wheeee!

Allright, I'll se what I did wrong now. :P

Forgot to mention it only happened clicking the blue button to restore after hitting cancel and then train again from 100

[d8ahazard]

line 796, in main

I see the issue. Should be fixed with 1ddfcfc

So far its fixed, yhea, it generated till 100 steps as a test and it successfully made an image and saved a ckpt file. However....it got stuck on a loop saying training complete? and it kept saving a ckpt file every 1 step.

Wheeee!
Allright, I'll se what I did wrong now. :P

Forgot to mention it only happened clicking the blue button to restore after hitting cancel and then train again from 100

Can you post the parameters it spits out in the console when you start training after doing this?

[YakuzaSuske]

let me check if i can replicate it...

Trying to parse: [{'instance_prompt': 'Luvelia', 'class_prompt': 'person', 'instance_data_dir': 'C:\Users\Hector\Downloads\New folder\New folder', 'class_data_dir': ''}]
Unable to load concepts as JSON, trying as file.
Loaded model.
Allocated: 0.0GB
Reserved: 0.0GB

The config attributes {'set_alpha_to_one': False, 'skip_prk_steps': True, 'steps_offset': 1} were passed to DDPMScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
Scheduler Loaded
Allocated: 0.2GB
Reserved: 0.2GB

Total target lifetime optimization steps = 2000
CPU: False Adam: True, Prec: fp16, Prior: False, Grad: True, TextTr: True
Allocated: 3.8GB
Reserved: 3.9GB

Steps: 0%| | 1/1000 [00:00<10:28, 1.59it/s, loss=0.00983, lr=5e-6]Error completing request
Arguments: ('Luvelia', 'C:\Users\Hector\Downloads\New folder\New folder', '', 'Luvelia', 'person', '', '', 1.0, 7.5, 40.0, 0, 512, False, True, 1, 1, 10, 1000, 1, True, 5e-06, False, 'constant', 0, True, 0.9, 0.999, 0.01, 1e-08, 1, 100, 1000, 'fp16', True, "[{'instance_prompt': 'Luvelia', 'class_prompt': 'person', 'instance_data_dir': 'C:\\Users\\Hector\\Downloads\\New folder\\New folder', 'class_data_dir': ''}]", False) {}
Traceback (most recent call last):
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\modules\ui.py", line 185, in f
res = list(func(*args, **kwargs))
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\webui.py", line 54, in f
res = func(*args, **kwargs)
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\extensions\sd_dreambooth_extension\dreambooth\dreambooth.py", line 256, in start_training
trained_steps = main(config)
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 766, in main
accelerator.backward(loss)
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\venv\lib\site-packages\accelerate\accelerator.py", line 882, in backward
self.scaler.scale(loss).backward(**kwargs)
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\venv\lib\site-packages\torch_tensor.py", line 396, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\venv\lib\site-packages\torch\autograd_init_.py", line 173, in backward
Variable.execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\venv\lib\site-packages\torch\autograd\function.py", line 253, in apply
return user_fn(self, *args)
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\venv\lib\site-packages\torch\utils\checkpoint.py", line 146, in backward
torch.autograd.backward(outputs_with_grad, args_with_grad)
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\venv\lib\site-packages\torch\autograd_init.py", line 173, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0; 12.00 GiB total capacity; 10.19 GiB already allocated; 0 bytes free; 10.74 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Steps: 0%| | 1/1000 [00:02<45:06, 2.71s/it, loss=0.00983, lr=5e-6]
Starting Dreambooth training...
VRAM cleared.
Allocated: 0.0GB
Reserved: 8.2GB

Trying to parse: [{'instance_prompt': 'Luvelia', 'class_prompt': 'person', 'instance_data_dir': 'C:\Users\Hector\Downloads\New folder\New folder', 'class_data_dir': ''}]
Unable to load concepts as JSON, trying as file.
Loaded model.
Allocated: 0.0GB
Reserved: 8.2GB

The config attributes {'set_alpha_to_one': False, 'skip_prk_steps': True, 'steps_offset': 1} were passed to DDPMScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
Scheduler Loaded
Allocated: 0.2GB
Reserved: 8.2GB

Total target lifetime optimization steps = 2000
CPU: False Adam: True, Prec: fp16, Prior: False, Grad: True, TextTr: True
Allocated: 3.8GB
Reserved: 8.2GB

Steps: 2%|▊ | 17/1000 [00:06<06:08, 2.67it/s, loss=0.0705, lr=5e-6] Loaded pipeline for preview...
Allocated: 7.1GB
Reserved: 11.2GB

Saving checkpoint at step 1017.
Successfully trained model for a total of 1017 steps, converting to ckpt.
Pipeline cleared...
Allocated: 5.7GB
Reserved: 6.7GB

Training complete??
Error completing request
Arguments: ('Luvelia', 'C:\Users\Hector\Downloads\New folder\New folder', '', 'Luvelia', 'person', '', '', 1.0, 7.5, 40.0, 0, 512, False, True, 1, 1, 10, 1000, 1, True, 5e-06, False, 'constant', 0, True, 0.9, 0.999, 0.01, 1e-08, 1, 100, 1000, 'fp16', True, "[{'instance_prompt': 'Luvelia', 'class_prompt': 'person', 'instance_data_dir': 'C:\\Users\\Hector\\Downloads\\New folder\\New folder', 'class_data_dir': ''}]", False) {}
Traceback (most recent call last):
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\modules\ui.py", line 185, in f
res = list(func(*args, **kwargs))
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\webui.py", line 54, in f
res = func(*args, **kwargs)
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\extensions\sd_dreambooth_extension\dreambooth\dreambooth.py", line 256, in start_training
trained_steps = main(config)
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 766, in main
accelerator.backward(loss)
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\venv\lib\site-packages\accelerate\accelerator.py", line 882, in backward
self.scaler.scale(loss).backward(**kwargs)
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\venv\lib\site-packages\torch_tensor.py", line 396, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\venv\lib\site-packages\torch\autograd_init_.py", line 173, in backward
Variable.execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\venv\lib\site-packages\torch\autograd\function.py", line 253, in apply
return user_fn(self, *args)
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\venv\lib\site-packages\torch\utils\checkpoint.py", line 146, in backward
torch.autograd.backward(outputs_with_grad, args_with_grad)
File "C:\SUPER_SD_2.0\stable-diffusion-webui-master\venv\lib\site-packages\torch\autograd_init.py", line 173, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0; 12.00 GiB total capacity; 10.19 GiB already allocated; 0 bytes free; 10.75 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Steps: 2%|▊ | 17/1000 [00:21<20:17, 1.24s/it, loss=0.0705, lr=5e-6]
Starting Dreambooth training...
VRAM cleared.
Allocated: 0.0GB
Reserved: 8.2GB

Trying to parse: [{'instance_prompt': 'Luvelia', 'class_prompt': 'person', 'instance_data_dir': 'C:\Users\Hector\Downloads\New folder\New folder', 'class_data_dir': ''}]
Unable to load concepts as JSON, trying as file.
Loaded model.
Allocated: 0.0GB
Reserved: 8.2GB

The config attributes {'set_alpha_to_one': False, 'skip_prk_steps': True, 'steps_offset': 1} were passed to DDPMScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
Scheduler Loaded
Allocated: 0.2GB
Reserved: 8.2GB

Total target lifetime optimization steps = 2000
CPU: False Adam: True, Prec: fp16, Prior: False, Grad: True, TextTr: True
Allocated: 3.8GB
Reserved: 8.2GB

Steps: 5%|██▋ | 52/1000 [00:20<06:26, 2.45it/s, loss=0.0851, lr=5e-6] Loaded pipeline for preview...
Allocated: 7.1GB
Reserved: 11.2GB

Saving checkpoint at step 1052.
Successfully trained model for a total of 1052 steps, converting to ckpt.
Pipeline cleared...
Allocated: 5.7GB
Reserved: 7.1GB

Training complete??
Steps: 5%|██▌ | 53/1000 [00:36<1:18:38, 4.98s/it, loss=0.0851, lr=5e-6] Loaded pipeline for preview...
Allocated: 7.1GB
Reserved: 11.2GB

Saving checkpoint at step 1053.
Successfully trained model for a total of 1053 steps, converting to ckpt.
Pipeline cleared...
Allocated: 5.7GB
Reserved: 7.1GB

Training complete??
Steps: 5%|██▋ | 54/1000 [01:03<3:04:55, 11.73s/it, loss=0.0851, lr=5e-6] Loaded pipeline for preview...
Allocated: 7.1GB
Reserved: 11.2GB

Saving checkpoint at step 1054.

[YakuzaSuske]

after pressing cancel it gets stuck on a loop, i had to close the cmd to stop it

[ShinkoNet]

Doesn't work.

Exception importing 8bit adam: cannot import name 'CUDASetup' from partially initialized module 'bitsandbytes.cextension' (most likely due to a circular import)

However, the same DLLs work with geocine's version because that force installs cu116

So I went to webui's main folder and did this:

.\venv\Scripts\activate
pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116

It then uninstalled cu113

Found existing installation: torchvision 0.13.1+cu113
    Uninstalling torchvision-0.13.1+cu113:
      Successfully uninstalled torchvision-0.13.1+cu113
Successfully installed torch-1.12.1+cu116 torchvision-0.13.1+cu116

and now it works.

[d8ahazard]

Just added this same bit in install.py via 9e7ec85

[ShinkoNet]

Great. Now this works on windows fine for me.

[YakuzaSuske]

Now we need a good training guide, i'm getting confused and sometimes bad results, mostly because i don't know how to train this well.

[d8ahazard]

That's the plan! For completeness, I'd probably want to add a custom script that calls launch.py from venv using accelerate launch instead of python. Otherwise, yep. :D

…

On Mon, Nov 7, 2022 at 6:37 PM ShinkoNet ***@***.***> wrote: Great. Now this should work on windows fine. — Reply to this email directly, view it on GitHub <#7 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMO4NEKDF56YM2S7OTYNUTWHGODLANCNFSM6AAAAAARZNF364> . You are receiving this because you commented.Message ID: ***@***.***>

[d8ahazard]

I'm going to mark this as closed. It may be necessary to specify a custom torch cmd in webui-user.bat, which I will document briefly in the readme. But, I can get bnb to load and run fine now, so we should be good here.

[d8ahazard]

Readme has been updated with installation instructions, but the trick to update CUDA (if needed) is to add this to your webui-user.bat file:

set TORCH_COMMAND="pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116"

[roperi]

@d8ahazard

I'm a bit confused.

Automatic1111 suggest this in its webui-user.sh (I'm a linux user):

export TORCH_COMMAND="pip install torch==1.12.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113"

But you suggests this other one (for Windows users at least):

TORCH_COMMAND="pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116"

Is Automatic1111 in the wrong? Shoud he/she use cu116 too?

Also, it seems a bit arbitrary to install one cuda version or another one without checking our own systems. So shouldn't the CUDA version be the one our own system has (i.e. nvcc --version)? Or am I missing something? Thanks in advance!

EDIT:
Disregard all that. I noticed bitsandbytes has some pretty stringent system requirements which I wasn't aware:

• Hardware requirements:

1. LLM.int8(): NVIDIA Turing (RTX 20xx; T4) or Ampere GPU (RTX 30xx; A4-A100); (a GPU from 2018 or older).
2. 8-bit optimizers and quantization: NVIDIA Maxwell GPU or newer (>=GTX 9XX).

[roperi]

8-bit optimizers and quantization: NVIDIA Maxwell GPU or newer (>=GTX 9XX).

So if I'm not mistaken choosing 8-bit adam will fail if you don't have those NVIDIA specs.

[GioPetro]

Similar issues taking place with newer versions now.
Anyone has a solution for CUDA 11.7 + ? Talking about bitsandbytes on windows (11) and cuda accel.
Thanks

[SiriumLA]

Possible solution for Windows users:

• Go to ...\stable-diffusion-webui\venv\Lib\site-packages
• Delete the folder "bitsandbytes"
• Use the bitsandbytes version with the following link:
  https://pypi.org/project/bitsandbytes-windows/
• Install the version and ensure it was added to site-packages of stable diffusion
• If you use xformers ensure you use version 0.0.22 in requirements.txt (xformers==0.0.22)

[swumagic]

Bitsandbytes was not supported windows before, but my method can support windows.（yuhuang）
1 open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD
or WIN+R, CMD 。enter，cd /d J:\StableDiffusion\sdwebui
2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl

Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）

[swumagic]

OR you are Linux distribution (Ubuntu, MacOS, etc.)system ,AND CUDA Version: 11.X.

Bitsandbytes can support ubuntu.（yuhuang）
1 open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD
or WIN+R, CMD 。enter，cd /d J:\StableDiffusion\sdwebui
2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/TimDettmers/bitsandbytes/releases/download/0.41.0/bitsandbytes-0.41.0-py3-none-any.whl

Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）