Bug with making an archive about e-hentai-downloader

[ccloli]

It might be no enough memory and your browser stop running the script, but please also tell me your browser, extension info and debug logs on console if possible.

[Zero3K]

I'm using the latest version of Firefox with the following addons:

DownThemAll
EagleGet Browser Extension
GreaseMonkey
ReloadEvery

The error being shown is "Error: Bug : can't construct the Blob".

[ccloli]

Hmm......thought I don't know what's wrong with in it, I think it might be possible to detect if it construct successfully. I'll check it tomorrow.

BTW, what's your RAM size and usage percentage now?

[Zero3K]

I tried closing a program that was using a decent amount of RAM but the issue still occurred.

[ccloli]

It seems that the error message was from JSZip. JSZip throw the error if browser cannot create a Blob object, but most browsers are supported Blob constructor. So it might be a bug of JSZip, but I'm not sure. I've tried the gallery you provided, but it worked fine. I'm not sure how to fix it, the thing I can do for that is to detect if the Zip file (Blob) generates successfully.

[iShyboy]

I've got this issue few times, but that was when i download a couple of galleries at the same time, or when the uptime of FF was too high, normally if I restart FF and try again just the gallery alone it works fine.
Oh, when the gallery has too many files (or too many errors like mime 403 etc), even if not huge in size it's high the chance of this bug, but it's difficult to reproduce. It's more easy to cut the gallery in half and download the 2nd half later.

[ccloli]

@iShyboy If you mean "can't construct the Blob", I don't know how to solve it. But from your comments, I think it mostly caused by no enough memory or Blob Storage space (800MB, from FileSaver.js), then Firefox stop running the script. The way to solve it now is do not download some galleries at the same time and use Pages Range (has been mentioned in README.md, auto-scale is in plan, but doesn't work on it).

For 403 error, I pushed 1.19 yesterday to enhanced fixing it, now it'll fetch image URL only when it's needed, instead of fetching all the URLs at the first time. If it not working, please let me know.

BTW, here is a node-webkit version, porting by my friend. It will download images to HDD, instead of storing in RAM and ziping them. If you are interested in it, check it out: https://github.com/8qwe24657913/E-Hentai-Downloader-NW.js

[iShyboy]

Right, should be this, dunno. It's hard to reproduce, the bug never happened on chrome for me, must be the blob space for FF.

[ccloli]

You reminded me. I thought it would be a bug of Firefox before, but just now I think the bug "can't construct the Blob" would probably because of no enough free memory (or Blob Storage). Then I tested it, and this bug was confirmed.

The error is thrown from JSZip when a Blob object cannot be construct. I separated it to a function arrayBuffer2Blob, and create a Array Buffer object which size is 100 MB:

var abdata = new ArrayBuffer(100*1024*1024);

Then construct it into a Blob object with arrayBuffer2Blob:

var blob = arrayBuffer2Blob(abdata);

When I constructing the 20th Blob object (my RAM is 8 GB, but it seems that the Blob Storage would probably larger than 800 MB), it threw the error Bug : can't construct the Blob., and the error message that caught in try-catch was NS_ERROR_OUT_OF_MEMORY.

So it's clear now, instead of giving a fake Blob object (Chrome), Firefox throw an error to inform that there is no enough space to create Blob object.

I tried to detect and solve it in 1.18.7, but I'm not sure it would work correctly because I couldn't reproduce it at that time. Now I'll continue working on it, and if it works, the issue would be closed.

[ccloli]

Now it can be detected if browser throw an error. When you see this dialog, that means you don't have enough memory to handle it, tried to close some tabs or programs to free your memory, then try again.

For more information, I'll write a wiki page tomorrow.

[ccloli]

The wiki page has been created here: https://github.com/ccloli/E-Hentai-Downloader/wiki/Can't-make-Zip-file-successfully