etherealxx / batchlinks-webui Goto Github PK

Download several Huggingface, MEGA, and CivitAI links at once. SD webui extension. For colab.

License: MIT License

Python 97.95% JavaScript 0.54% Shell 0.32% Batchfile 1.19%

colab stable-diffusion stable-diffusion-web-ui stable-diffusion-webui-plugin steble-diffusion-webui batch-download bulk-downloader mass-downloader gradio gradio-interface

batchlinks-webui's Issues

CivitAI not working somehow

If you guys also found out about the CivitAI problem, please comment down below with the screenshot of your error message. I will find the culprit of this mess🙏

IndentationError: expected an indented block on line 1497

Tested today (Fri Mar 24 11:23:53 CET 2023)
This error prevent the module to be loaded in colab

File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/batchlinks-webui/scripts/batchlinks-downloader.py", line 1497
if everymethod == False:
IndentationError: expected an indented block

Creating a tag without downloading anything into the folder causes error

Creating a custom tag like this without specifying any downloads causes the extension to break until UI restart.
@new #000None /content/stable-diffusion-webui/models/Lora/000None

This does create the empty folder, but breaks for any subsequent downloads.

I do this to create an empty directory that I use as a makeshift filter.
Feel free to delete this if it's not part of the project scope.

Here is the stack trace:

Adding custom path failed! Reason: unexpected EOF while parsing (<string>, line 1)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/gradio/routes.py", line 337, in run_predict
    output = await app.get_blocks().process_api(
  File "/usr/local/lib/python3.9/dist-packages/gradio/blocks.py", line 1015, in process_api
    result = await self.call_function(
  File "/usr/local/lib/python3.9/dist-packages/gradio/blocks.py", line 833, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.9/dist-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.9/dist-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.9/dist-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/content/stable-diffusion-webui/extensions/batchlinks-webui/scripts/batchlinks-downloader.py", line 1692, in run
    downloadedfiles = writeall(oldfilesdict, isshell, addedcustompath)
  File "/content/stable-diffusion-webui/extensions/batchlinks-webui/scripts/batchlinks-downloader.py", line 1116, in writeall
    newdict = trackall()
  File "/content/stable-diffusion-webui/extensions/batchlinks-webui/scripts/batchlinks-downloader.py", line 1162, in trackall
    exec(f"os.makedirs({x}path, exist_ok=True)")
  File "<string>", line 1
    os.makedirs(000Nonepath, exist_ok=True)
                   ^
SyntaxError: invalid syntax

Downloads using civitai always download in the default directories

Downloads using civitai always download in the default directories regardless of any prior commands; eg:

@new #pokemon_loras /content/stable-diffusion-webui/models/Lora/Pokemon
#pokemon_loras
https://civitai.com/models/8243/misty-pokemon-lora-8-mb

This code should create the Pokemon directory and download into it.

How to change default Directory for saving files ?

I used SDless mode, i tried to change the path in your parameter but it always lead to /content/SD-Webui...
I want to change default location for saving on my google Drive (mounted) how can i do that ? pls help me !

Cannot install MEGA files.

Control the volume of the sound effect?

I really like the feature of the sound effect notification - but the default sound is quite loud ^_^ anyway, we can control that? Thanks!

some civitai models do not download (only Model Preview Images are downloaded)

Some models do not download well, only the preview image downloads.
I will give an example with two models, this is just an example but it happens to me with several,
I will give an example of one that is downloaded and another that is not downloaded, and how it looks in the Sagemaker notebook:
This model does not download it for me, it only downloads the preview image

On the other hand, this other model does download it completely

at first, I thought it was because one model was heavier than the other, so the heavier one wouldn't download properly for me. But then I tried with others, and it happened the other way around: the heavier one would download, but not the lighter one.

Then I think it had to do with the fact that some models seem to have multiple download options. Some have a download button, while others seem to offer several options (sorry for my ignorance, I'm just starting out with this, so I don't know why some models are like this on Civitai). Well, that's not it either.

So, I don't know what it is. Can you please help me? I don't know if it's an error, or if I'm doing something wrong.

I'll leave two links of two other example models, one that downloads fine for me and another that doesn't:
https://civitai.com/models/23521/anime-pastel-dream <-- This model downloads correctly
https://civitai.com/models/90642/animesh <-- this not

Traceback (most recent call last):

huggingface download finished but the downloaded files are small

https://huggingface.co/XpucT/Deliberate/blob/main/Deliberate_v2.safetensors

the file is downloaded quickly but it's not the full file.

Please, add a possibilite to download controlnet models

Would be very nice

maybe take some inspiration from here

https://github.com/aka7774/sd_filer

Model link methon was crashed.

CivitAI direct link method: slow download speed

[MEGA link]: Failed to create socket for registering for state changes. (Using Google Colab)

Hello there. Thank you so much for creating such a useful extension, it's been a life changer. Today, however, I faced a rather peculiar issue:

I don't know what this means, but it's only for MEGA links. It had no issues with Drive, CivitAi, and HuggingFace links. You wanna know what's weird about this? I had used a different Colab earlier the same day and that same link (+ one more) had worked! It downloaded them just fine! So I really don't know what or why this is happening.
Any thoughts? Could this be an issue with this particular Colab? If so, why an issue with only MEGA links?

Huggingface links : it's downloading nothing at all

I tried to download all controlnet files from the main tree... nothing.
Only way to download... file by file, is to click on a file on huggingface ([control_v11p_sd15_mlsd.pth, etc.), copy the webpage link for that file into your extension, and then it downloads that unique file... in the root of stable diffusion.
This doesn't batch download anything. If one needs to copy paste the download link of 30 files, it's pretty much useless.

When downloading a model it instead downloads a batchlink solid or a yaml

This happens with certain models I used to be able to download but now whenever I try to download them it instead does this

completely stopped working for me after AUTO1111 webui 1.6

I am on Google Colab, with /models/Lora hard linked to a Google Drive folder, not sure if that matters, but it's always worked just fine with this extension in the past.

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/gradio/routes.py", line 488, in run_predict
    output = await app.get_blocks().process_api(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1431, in process_api
    result = await self.call_function(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1103, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/dist-packages/gradio/utils.py", line 707, in wrapper
    response = f(*args, **kwargs)
  File "/content/stable-diffusion-webui/extensions/batchlinks-webui/scripts/batchlinks-downloader.py", line 1292, in run
    oldfilesdict = trackall()
  File "/content/stable-diffusion-webui/extensions/batchlinks-webui/scripts/batchlinks-downloader.py", line 1210, in trackall
    exec(f"os.makedirs({x}path, exist_ok=True)")
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/content/stable-diffusion-webui/models/Lora'

I'm not sure why it's trying to make a directory that already exists. Is there a mkdir somewhere in here without -p or without a check to see if the directory exists?

[HELP NEEDED]: FileNotFoundError raised when more than one download process is cancelled

Whenever i tried to download more than 1 link, for example

https://civitai.com/models/10028/neverending-dream
https://civitai.com/models/43331/majicmix-realistic

and i cancel it midway using the cancel button, it'll raise this error

BatchLinks Downloads starting...

https://civitai.com/models/10028/neverending-dream
[#100615 816MiB/3.9GiB(20%) CN:16 DL:165MiB ETA:19s]

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
100615|INPR|   154MiB/s|/content/stable-diffusion-webui/models/Stable-diffusion/neverendingDreamNED_bakedVae.safetensors

Status Legend:
(INPR):download in-progress.

aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.

Removed incomplete download: /content/stable-diffusion-webui/models/Stable-diffusion/neverendingDreamNED_bakedVae.safetensors.aria2
Removed incomplete download: /content/stable-diffusion-webui/models/Stable-diffusion/neverendingDreamNED_bakedVae.safetensors
Operation Cancelled

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/gradio/routes.py", line 337, in run_predict
    output = await app.get_blocks().process_api(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1018, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 956, in postprocess_data
    prediction_value = block.postprocess(prediction_value)
  File "/usr/local/lib/python3.10/dist-packages/gradio/components.py", line 2608, in postprocess
    dataframe = pd.read_csv(y)
  File "/usr/local/lib/python3.10/dist-packages/pandas/util/_decorators.py", line 211, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pandas/util/_decorators.py", line 331, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pandas/io/parsers/readers.py", line 950, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/usr/local/lib/python3.10/dist-packages/pandas/io/parsers/readers.py", line 605, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/usr/local/lib/python3.10/dist-packages/pandas/io/parsers/readers.py", line 1442, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/usr/local/lib/python3.10/dist-packages/pandas/io/parsers/readers.py", line 1735, in _make_engine
    self.handles = get_handle(
  File "/usr/local/lib/python3.10/dist-packages/pandas/io/common.py", line 856, in get_handle
    handle = open(
FileNotFoundError: [Errno 2] No such file or directory: 'p'

This error doesn't break the gradio UI though, and you can continue to use the UI like before. However this error won't show if only one link is processed on the box. For example:

https://civitai.com/models/10028/neverending-dream

I tried to find the root of the problem, but sadly i can't. Perhaps someone else can, would be very helpful and appreciated🙏

updated batchlink alternative

for someone who looking for the batch download similar as this repo. this

Vae fails to be downloaded on civitai model page link

the png got downloaded instead

Add an option for authorization with Civitai user token

Civitai restricted download for not logged in users.

⚠️If you get KeyError: 'content-disposition' click here.

CivitAI is currently down, that's why the extension can't download the model.
I'll add feature to check if civit website is currently down later.

some model in civitai cannot download

When download this model https://civitai.com/models/10028/neverending-dream, the error code is below:

https://civitai.com/models/10028/neverending-dream
Traceback (most recent call last):
  File "/user/.local/lib/python3.10/site-packages/gradio/routes.py", line 399, in run_predict
    output = await app.get_blocks().process_api(
  File "/user/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1299, in process_api
    result = await self.call_function(
  File "/user/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1022, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/user/.local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/user/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/user/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/user/stable-diffusion-webui/extensions/batchlinks-webui/scripts/batchlinks-downloader.py", line 1607, in run
    civitdown2(currentlink, currentfolder, choosedowner, currenttorename, False, civitdefault, civitpruned, civitvae)
  File "/user/stable-diffusion-webui/extensions/batchlinks-webui/scripts/batchlinks-downloader.py", line 801, in civitdown2
    data_index, data_url, data_filename = civitmodeltypechooser(model, isprunedmodel, modeldefaulttype, civitlinkandnames) #@note
ValueError: too many values to unpack (expected 3)

Feature Suggestion and Testing

A lot of embeddings and loras are uploaded to catbox.moe and related sites from users on 4ch.

Maybe an edge case, I tend to store my TIs in Discord since its quite convenient.

Since this extension is already utilizing gdown, downloading from Google Drive also?

Maybe add this as a toggleable option since supporting every site will lead to unexpected things.

error in path with @new

This is an edition of a bug that wasn't, but since I can't delete it, I edit with this other one.

Many of the errors that I receive when downloading the files is for various reasons but it does not clarify it.
Because the final file already exists. (for example the model already exists)
For not being able to download the source file.
it could be corrected by noting that it already exists, or by renaming it to a version like "duplicate".

Thank you
Good job !

Got a codec error in download process

like this.

Rename failed - webui running with --share

After downloading a model from CivitAI the rename failed.

Is this because the webui is running with --share?

I tried this in a local Mac running webui without any problem.

Running on public URL: https://206221c3xxx.gradio.live

This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces
Startup time: 14.9s (import torch: 0.9s, import gradio: 0.7s, import ldm: 0.5s, other imports: 0.6s, load scripts: 1.1s, load SD checkpoint: 2.3s, create ui: 0.8s, gradio launch: 7.9s).
BatchLinks Downloads starting...



https://civitai.com/api/download/models/46846
Downloading...
From: https://civitai.com/api/download/models/46846
To: /home/hjwang/stable-diffusion-webui/models/Stable-diffusion/46846

100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.51G/5.51G [21:49<00:00, 4.21MB/s]Rename failed.

BatchLinks Downloads finished!

additional network not working

No matter the hashtag it populates in the module lora folder and not the additional network folder. However when i use the other Civitai extension in gradio web diffusion it works fine and that LORA downloads in the additional network folder and allows for keywords to trigger it.

etherealxx / batchlinks-webui Goto Github PK

batchlinks-webui's Issues

Recommend Projects

Recommend Topics

Recommend Org