webscraperio / image-downloader Goto Github PK

how can i batch download images from a site that requires a login

Hi guys,
Total newb here, (this would be maybe 10th time I've ever opened terminal...lol)
I have downloaded/installed latest version of Python(3.12.0), I have my .csv file in the right place, it has the correct columns, and is with the "image-downloader.py" file. But I don't get any images downloaded into the created folder, and when I look at the readout from terminal this is what it says:

INFO:executed by python 2
INFO:importing data from scraped_data.csv
Traceback (most recent call last):
File "image-downloader.py", line 133, in
main(sys.argv)
File "image-downloader.py", line 125, in main
download_csv_file_images(csv_filename)
File "image-downloader.py", line 111, in download_csv_file_images
with open(filename, "r", encoding="utf8") as csvfile:
TypeError: 'encoding' is an invalid keyword argument for this function

I'm on a pretty ancient mac - maybe that's the problem?
2012 Macbook Pro / MacOS Catalina 10.15.7

Please help! :)
I will go and try on my new Mac mini and see if that helps.....

Hi there,

First of all thank you very much for this script. I really did helped me with my job.
However currently I'm having troubles to rename files through "web-scraper-order" column. If I put there instead of default (f.e.: 1608498486-1648) data desired future file name (f.e.: 3245/285/330) script can't proceed and download the images. Apparently it has something to do with UTF-8 coding and slash - "/".
Can anybody give me an advice how to adjust the script so I can do such thing?

Any advice would be appreciated!

The subject error occurs when executing the script.
How do I modify the script?

I have a problem with .webp format when I download :)

Example:

.jpg.pagespeed.ic.r1QtDCQg3m.webp

You may want to add handling for webp images - I am seeing them in my scrapes. They are often .jpg in the url, but the format that comes down and the Content Type is webp. Easy change in the code is:

   if content_type == 'image/jpeg' or content_type == 'image/jpg':
        ext = 'jpg'
    elif content_type == 'image/png':
        ext = 'png'
    elif content_type == 'image/gif':
        ext = 'gif'
    elif content_type =='image/webp':
        ext = 'webp'

That way they come down with a webp extension and can be converted.

I would put in a pull request, but don't know if you want to deal with that, so offering it here.

PS: Thank you for all this, it has really helped me!

i get following error

WARNING:Image download error. HTTP Error 403: Forbidden
INFO:downloading image

It just dies when i run it. My file is 3million lines long and 300mb? Was hoping it would work.... Any ideas
screenshot here... just dies... https://imgur.com/V6XFGmA