akianonymus / gdrive-downloader Goto Github PK

I used this command,

gdl -df pdf <url>

support for proxy

like this: --all-proxy 127.0.0.1:1080
gdl --all-proxy 127.0.0.1:1080 "https://drive.google.com/u/0/uc?id=1nDtRgzAFgz2j6BlLHIMqfCcd5jDLBcDd&export=download" -aria --aria-flags '-s 10 -j 10 -x 16' --aria-flags '--https-proxy=http://127.0.0.1:1080'

For example, this one:

gdl "https://drive.google.com/drive/folders/18NcDYyhr9BHAWZww1QAnCXCiqiPOQ1WJ"

will result in an empty folder:

======================================[ Street Epistemology Tip Sheets | Empty Folder ]=======================================

========================================== Time Elapsed: 0 minute(s) and 1 seconds. ==========================================

Hey,

Hope you're doing well. The issue Download quota exceeded comes on my shared drive whenever I'm trying to download.

I know it's self-explanatory but I'm downloading files from my own drive, just made it shareable and it can't even download the files (from a folder) once, a bug with the shared drive? Limitation from the gdrive side?

When I open those files in incognito, the same happens. But, when I try accessing those files or downloading them authenticated, it works perfectly fine.

Can we somehow integrate credentials or some tokens to make it work based on the account? Just like https://github.com/labbots/google-drive-upload?

Here's a breakdown of what I've observed:
(Sometimes) the first file downloads perfectly fine from the folder, the rest of the files start giving limit quota exceeded even though they weren't even downloaded or accessed. Strange. Maybe, they blacklist IP? But after one download, doesn't make sense.

Let me know if you want to try this yourself and I'll create a fresh URL of files that haven't been downloaded ever and you can try and reproduce it yourself.

Here's a POC from a URL I created (for the very first time):

With this limitation, one can't ever download any files in bulk (even personal). Most probably we're overlooking something.

Hi, I am downloading from a shared folder but it only finds the first 1000 items exactly and downloads them with no problem.

Is there some way to increase this limit?

Hi,

Thanks for your work! I find that it sometimes works sometimes doesn't. And if it doesn'r work, I received the following output in linux screen. Can you please help me solve the issue?

Thank you very much!

Hey can you add support for background downloading with links picked up from a file.
This will be easier to add a cron tab.

Right now I'm using-
nohup gdl -o -q https://drive.google.com/file/d/xxx &
For background downloading with nohup

Also support for logging with bit more details like file name link start time end time..avg speed etc

https://drive.google.com/u/0/uc?id=1MR7M-as-stOIog2EyhNXwnKKC9-aXXiM

https://drive.google.com/u/0/uc?id=1MR7M-as-stOIog2EyhNXwnKKC9-aXXiM&export=download&confirm=t

Hi,

Hope you're doing fine. The tool's working amazing so far.

If you feel like it, can you please look into integrating aria2(c) as a downloader in the script as well? Can add a separate argument for downloading (i.e. --use-aria2) or something like that and one can pass custom parameters or just use default one.

It by default has its own progress information, supports multiprocessing, and many other things.

I tried integrating it myself by replacing it in the gdl binary but wasn't able to load cookies from the cookie jar file generated.

aria2c -s 10 -j 10 -x 16 --file-allocation=none --load-cookies "$TMPFILE"COOKIE "https://drive.google.com/uc?export=download&id=$file_id${confirm_string:+&confirm=$confirm_string}" -o "$name" -c -l -

Let me know if you need any more details to work with, from my side.

I spent about 2 hours looking up like 8 different huge tools, that wanted me to log in, create some api keys, and never worked

AND I JUST WANTED TO DOWNLOAD A PUBLIC LINK

And this just works! Thank you for making this!

Run gdl for a urls/ids but just check if it can be downloaded.

Flag name: -dr | --dry-run

If I put these two lines in a sh script, then the second line . /home/$USER/.bashrc is not executed.
curl -Ls --compressed https://github.com/Akianonymus/gdrive-downloader/raw/master/install.sh | sh -s . /home/$USER/.bashrc
But if I run it in a terminal then it is executed.

Im trying to test out download on a single file.

I am running "gdl 1KpFwfTVU62jtpWh3vHVa7Cfu3s_jh-0S -o" after setting up the outh api ID and key

How is the folder path supposed to be mentioned, i did not see any example scripts, tried different combinations of drive url nd id.

What am i missing in my command?

Thanks for your help in advance.

gdl "https://drive.google.com/drive/folders/1dPJuKXMXLcKndcTWjH9rFCkPBNGYwD-p"

gdl: line 894: cd: - : invalid option

https://developers.googleblog.com/2022/02/making-oauth-flows-safer.html?m=1#disallowed-oob

OAuth out-of-band (OOB) is a legacy flow developed to support native clients which do not have a redirect URI like web apps to accept the credentials after a user approves an OAuth consent request. The OOB flow poses a remote phishing risk and clients must migrate to an alternative method to protect against this vulnerability. New clients will be unable to use this flow starting on Feb 28, 2022.

Key dates for compliance

Feb 28, 2022 - new OAuth usage will be blocked for the OOB flow
Sep 5, 2022 - a user-facing warning message may be displayed to non-compliant OAuth requests
Oct 3, 2022 - the OOB flow is deprecated for existing clients

Otherwise, identifying through OAuth leads into this

Plenty of other Google-related projects already worked/are working on this:

• r-lib/gargle#202
• openhab/openhab-addons#12455
• fregante/chrome-webstore-upload#59
• simonw/google-drive-to-sqlite#39
• astrada/google-drive-ocamlfuse#756
• rclone/rclone#6000

Meanwhile, I'm using an API key.

Can we have a feature to set custom user agent parameter for curl in the gdl command line parameter

With the new installation of gdl, I am facing the following error:

/home/arul/.gdrive-downloader/gdl: line 18: 2: Missing string

I have executed gdl with command: gdl <gdrive_link> -o.
It asks for a new account name. After giving the account name, it throws the above error.

.gdrive-downloader/gdl:

#!/usr/bin/env bash
LAST_UPDATE_TIME="1629023193" # added values
VALUES_LIST="REPO COMMAND_NAME INSTALL_PATH TYPE TYPE_VALUE SHELL_RC LAST_UPDATE_TIME AUTO_UPDATE_INTERVAL INSTALLATION GLOBAL_INSTALL PERM_MODE GDL_INSTALLED_$
REPO="Akianonymus/gdrive-downloader" # added values
COMMAND_NAME="gdl" # added values
INSTALL_PATH="/home/arul/.gdrive-downloader" # added values
TYPE="branch" # added values
TYPE_VALUE="master" # added values
SHELL_RC="/home/arul/.zshrc" # added values
AUTO_UPDATE_INTERVAL="432000" # added values
INSTALLATION="bash" # added values
GLOBAL_INSTALL="false" # added values
PERM_MODE="u" # added values
GDL_INSTALLED_WITH="script" # added values
LATEST_INSTALLED_SHA="17185a7f55fefdea4ed20d3fe13eba315ab02299" # added values
SELF_SOURCE="true"
_assert_regex(){
declare pattern="${1:?Error: Missing pattern}" string="${2:?Missing string}"
if [[ $string =~ $pattern ]];then
return 0
else
return 1
fi
}

Addition of a static logfile with downloaded file history with bit more details like

date and time,
file name,
file link,
Start time end time
avg speed

Tried the script on a sample public file I found here. Used the curl based install from a ubuntu:latest docker image interactively. Got the following output:

I tried without the /view portion of the URL with the same result. Likewise, attempting to repeat the download results in the tool indicating that a partial download was detected, but it still fails.

It's been erroring out on the first actual folder within said folder. I've used an API key and oauth, with those two giving me this error in the 'folder' it downloads (which it exports as a file).

 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "fileNotExportable",
    "message": "Export only supports Docs Editors files."


   }
  ],
  "code": 403,
  "message": "Export only supports Docs Editors files."
 }
}

This is what happens when I don't use an API key or oauth.

I am wondering whether it is possible to have functions like ls so that I don't have to go to webpage find the name of the file

I applaud your work in making Google Drive more usable and anonymous, but the current progress indicator is really bad. Because it continually scrolls, it can be difficult to watch the progress of the transfer without getting a migraine. One way of doing this is to use carriage returns alongside a progress bar.

Here's an example Bash function that can do this (courtesy of ChatGPT because I can't do bash):

displayProgressBar() {
    local width=50
    local progress=$1
    local percentage=$((progress * 100 / 100))
    local fill=$((progress * width / 100))
    local progressBar="["
    for ((i = 0; i < width; i++)); do
        if [ "$i" -lt "$fill" ]; then
            progressBar+="="
        else
            progressBar+=" "
        fi
    done
    progressBar+="] $percentage%"
    echo -ne "$progressBar\r"
}

Hi,
Thank u for this great repository. Im having some troubles, after multiple succeeded files downloaded, next files are showing "Incomplete download ", trying via browser everything is ok but the files with incomplete download have virus check message.

This is a file example: https://drive.google.com/file/d/1nFpz6XlUZ9yIFSQmK0FAhXslkZazxjSE/

Hope can help me to solve the issue. Thanks!

Is it possible to implement a batch downloading feature where I could supply it with a file containing a bunch of urls and it would download all of them? Currently I just create a shell script that launches a new instance for the next file but batch downloading could offer up the ability to parallel download the files as well.

Followed the guides for auth.md

On the first run, I executed something like gdl https://drive.google.com/drive/folders/{ID} -o "{CLIENT-ID}" "{CLIENT-SECRET}"

I get the following:

[ Checking credentials.. ]
[  New account name:  ]
[ Info: Account names can only contain alphabets / numbers / dashes. ]

-> 

Not sure what account names are. Any idea why the credentials are not setting?

Hey can you add support for background downloading with links picked up from a file.
This will be easier to add a cron tab.

Right now I'm using-
nohup gdl -o -q https://drive.google.com/file/d/xxx &
For background downloading with nohup

Hey can you add support for background downloading with links picked up from a file.
This will be easier to add a cron tab.

Right now I'm using-
nohup gdl -o -q https://drive.google.com/file/d/xxx &
For background downloading with nohup

Just cloned the master branch and I get this:

gdrive-downloader$ ./install.sh
[ Checking Internet Connection.. ]
./install.sh: 361: ./install.sh: cannot create /dev/null: File exists
[  ]

Any ideas?

Hi,

First of all, this is a great tool, it helped me download all the contents of the following drive . Although it downloaded everything fine, it frequently said "Download failed"/"Download incomplete", although the download was completed adequately. Just thought you should know. Maybe you can reproduce it with that link. Thanks in advance.

Hey,

Hope you're doing fine, I was recently trying to download a zip file of 89 GB. Due to some issues in the container on the VPS I've, consumes it's storage.

If I can allocate the space in the really beginning when the command runs, it'll be useful. Right now the script has it already implemented to not allocate any space, I'd say let's keep that but over-write that argument if the user specifies any.

I solved my issue by manually removing --file-allocation=none at https://github.com/Akianonymus/gdrive-downloader/blob/master/bash/gdl.bash#L245 and then passing --file-allocation=falloc.

Aria2c's supported types:

 --file-allocation=METHOD     Specify file allocation method.
                              'none' doesn't pre-allocate file space. 'prealloc'
                              pre-allocates file space before download begins.
                              This may take some time depending on the size of
                              the file.
                              If you are using newer file systems such as ext4
                              (with extents support), btrfs, xfs or NTFS
                              (MinGW build only), 'falloc' is your best
                              choice. It allocates large(few GiB) files
                              almost instantly. Don't use 'falloc' with legacy
                              file systems such as ext3 and FAT32 because it
                              takes almost same time as 'prealloc' and it
                              blocks aria2 entirely until allocation finishes.
                              'falloc' may not be available if your system
                              doesn't have posix_fallocate() function.
                              'trunc' uses ftruncate() system call or
                              platform-specific counterpart to truncate a file
                              to a specified length.

                              Possible Values: none, prealloc, trunc, falloc
                              Default: prealloc
                              Tags: #basic, #file

Hi ,

I try using gdownloader but get error= (grep: /tmp/tmp.bdByTMbCqw_15QWvLt5H1-XImx-dOXMrGFfCNum_BlnX_COOKIE: No such file or directory)

For example, all the files in this folder: https://drive.google.com/drive/folders/1_0RKPtibDHD-RXkIf3z7Exs4qeNUTRTQ

On a different folder, it can download file that is smaller than 100MB (I think whatever file that is larger than the virus scan limit of Google, will be downloaded to only 2.5kB.

I wonder if it's an issue on my side?

Hey demon, it's me ya boy (sauce meme: https://user-images.githubusercontent.com/18597330/99195267-8ab36880-27a6-11eb-86f8-e240f624a881.png)

So, I actually I was trying to download a folder and it had too much-nested folders, like 10-20 in each folder, and a total of 90 parent directories.

The thing is, it first completes a directory and then downloads its files when that completes, moves onto other which makes the script rather slow.

Gupload (from you) on the other hand creates folders first and then uploads files, is it possible to port that feature in this script maybe?

Would really appreciate using the bandwidth and will save a lot of time I think.

TL;DR: Creation of directories prior to downloading of files, so, if a user specifies 50 files, it doesn't wait on the directory to complete which let's say has 5 files and move on. It downloads all 50 in parallel.

Hello,

When the script is downloading a shared folder, if a file exists in the folder but exceeds the download quota limit, it will cause the script to get stuck.
Google Drive site returns http code 200 but the content is a page with hints "Google Drive - Quota exceeded". The script will continuously try to test whether the file is in the local directory every 0.5 seconds.

First of all, this is a great tool, helps us download folders in quite easier way.

While downloading the internet connection was disconnected. Which made the script skip some files.
Like file1 was 1gb only 200mb was downloaded.
And file2 downloading had begun.
So I launched the script again.
Now file 1 and 2 was complete.
But file 3 which was 1gb was downloaded twice. The completed size was 2gb

So in short

There should be size check or file integrity check.

A way to kill the script as killing curl makes it launch again automatically somehow

First of all, thanks for the great tool!

My problem:

1. I've been downloading a large google drive folder by creating new OAuth credentials (the only way it would let me do it consistently)
2. Made a break for a week
3. Run again, now gdl would report:

{
"error": "invalid_grant",
"error_description": "Token has been expired or revoked."
}

4. Turned out google revokes refresh tokens for the "test" apps (not in "production") after 7 days. More info here
5. I've deleted all the info in .gdl.conf (saved client id & secret), rerun gdl URL --oauth (entered client id & secret again) and got the new refresh token for another 7 days

Proposal:

It would be great if I can run something like gdl --oauth-refetch-refresh-token to get refresh token again without messing with config files and entering credentials again!

Hey,

Awesome script as always, I was trying to download files from my shared google drive (not personal). The link generated in the same way works completely fine for personal drive's folders/files. Whenever I try to download files from shared drive, it gives me a error saying invalid URL/ID.

POCs:

File from shared drive:

File from own personal drive:

Just like your guploader script allow's to upload files on shared drive, can we also have the same feature implemented in this one? Or am I doing something wrong? If not implemented, I'd really appreciate getting it to work 🙏

Here's a folder (on gdrive) to help you debug: https://drive.google.com/drive/folders/16oz2YXXqoEIBhzNU3ScyGQ7U_pkfltdG?usp=sharing

One of the files from the above drive^: https://drive.google.com/file/d/1spBOrsuGcALciS7gDHw-ie-Os21K2CoY/view?usp=sharing

Debug output:

root@b0x:~# gdl https://drive.google.com/file/d/1spBOrsuGcALciS7gDHw-ie-Os21K2CoY/view?usp=sharing -D
+ PS4='-> '
-> return 0
-> _check_internet
-> : justify 'Checking Internet Connection..' -
-> _timeout 10 curl -Is google.com
-> declare timeout=10
-> shift
-> _clear_line 1
-> :
-> trap 'abnormal_exit="1"; exit' INT TERM
-> trap _cleanup EXIT
--> printf '%(%s)T\n' -1
-> START=1602423445
-> _process_arguments
-> export DEBUG LOG_FILE_ID VERBOSE API_KEY API_URL API_VERSION FOLDERNAME SKIP_SUBDIRS NO_OF_PARALLEL_JOBS PARALLEL_DOWNLOAD SKIP_INTERNET_CHECK COLUMNS CURL_SPEED TMPFILE CURL_PROGRESS EXTRA_LOG RETRY QUIET
-> export -f _bytes_to_human _count _fetch _json_value _print_center _print_center _newline _clear_line _download_file _download_file_main _download_folder _log_in_file
-> cd .
-> unset Aseen
-> declare -A Aseen
-> for id in "${ID_INPUT_ARRAY[@]}"
-> [[ -n '' ]]
-> Aseen[$id]=x
-> _check_id 1spBOrsuGcALciS7gDHw-ie-Os21K2CoY
-> [[ 1 == 0 ]]
-> : justify 'Validating URL/ID..' -
-> declare id=1spBOrsuGcALciS7gDHw-ie-Os21K2CoY json
-> unset NAME SIZE
--> _fetch 'https://www.googleapis.com/drive/v3/files/1spBOrsuGcALciS7gDHw-ie-Os21K2CoY?alt=json&fields=name,size,mimeType&key=AIzaSyD2dHsZJ9b4OXuy5B_owiL8W18NaNOM8tk'
--> curl -e https://drive.google.com --compressed 'https://www.googleapis.com/drive/v3/files/1spBOrsuGcALciS7gDHw-ie-Os21K2CoY?alt=json&fields=name,size,mimeType&key=AIzaSyD2dHsZJ9b4OXuy5B_owiL8W18NaNOM8tk' -s
--> _clear_line 1
--> :
-> json='{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "notFound",
    "message": "File not found: 1spBOrsuGcALciS7gDHw-ie-Os21K2CoY.",
    "locationType": "parameter",
    "location": "fileId"
   }
  ],
  "code": 404,
  "message": "File not found: 1spBOrsuGcALciS7gDHw-ie-Os21K2CoY."
 }
}'
-> _json_value code 1 1
-> _clear_line 1
-> :
-> _print_center justify 'Invalid URL/ID' =
-> [[ 3 == 3 ]]
-> printf '%s\n' 'Invalid URL/ID'
Invalid URL/ID
-> _newline '\n'
-> :
-> return 1
-> continue
-> return 0
--> printf '%(%s)T\n' -1
-> END=1602423445
-> DIFF=0
-> _print_center normal ' Time Elapsed: 0 minute(s) and 0 seconds. ' =
-> [[ 3 == 3 ]]
-> printf '%s\n' ' Time Elapsed: 0 minute(s) and 0 seconds. '
 Time Elapsed: 0 minute(s) and 0 seconds.
-> _cleanup
-> return 0

Hi, I'm trying to get this working on a Mac M1 running Mac OS Ventura (13.3.1). I've got Homebrew and used it to install both this script and bash 5.2.15.

The script exits after the "Enter Client ID" step, with the error "chmod: --: No such file or directory". I wasn't familiar with the double dash -- syntax, maybe it's not supported by the chmod provided by MacOS?