hr / github-clone

:octocat: ⬇️ ⠀git clone repo subdirectories

GitHub clone

Git clone (download) any sub-directories of any GitHub repository (at any reference) without having to clone the entire repository, with only its GitHub URL. Uses the GitHub API to recursively clone the sub-directories tree and files.

Motivation

I often find myself wanting to only download a certain directory, path or package of an especially big repo that I'm currently viewing (without even cloning the entire repo at depth 1) and to do so by simply copy & pasting the GitHub URL so that's why. Probably more instances where this might come in handy ;)

Rate limit

The GitHub API imposes a rate limiting of up to 60 requests per hour applies but can be increased to up to 5000 requests per hour using an OAuth token (to get one see https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line).

GitHub clone makes an initial request to fetch repo metadata and then, a request for every subfolder in the repo. The requests to download the files within the folders are not counted against the rate limit so in most cases (i.e. the folder/repo you're trying to clone has less than 60 subfolders) the rate limit should not be a problem.

Private repositories

To clone private repositories you need to supply an OAuth token for an account with access to the private repository (to get one see https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line).

Installation

Available on PyPi https://pypi.org/project/github-clone/.

Install the script via pip:

pip install github-clone

or via pipsi:

pipsi install github-clone

Uses Python 3.3+

Usage

GitHub clone (git.io/ghclone)

Usage:
  ghclone <url> [-t | --token=<token>]
  ghclone (-h | --help)
  ghclone (-v | --version)

Examples:
  ghclone https://github.com/HR/Crypter/tree/master/app
  ghclone https://github.com/HR/Crypter/tree/dev/app
  ghclone https://github.com/HR/Crypter/tree/v3.1.0/build
  ghclone https://github.com/HR/Crypter/tree/cbee54dd720bb8aaa3a2111fcec667ca5f700510/build
  ghclone https://github.com/HR/Picturesque/tree/master/app/src -t li50d67757gm20556d53f08126215725a698560b

Options:
  -h --help           Show this screen.
  -v --version        Show version.
  -t --token=<token>  Set a GitHub OAuth token (see https://developer.github.com/v3/#rate-limiting).

License

Copyright (C) 2019-2021 Habib Rehman (https://git.io/HR)

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

github-clone's Issues

raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 503 Server Error: Server Temporarily Unavailable for url: http://grcm39.rest.ensembl.org/variation/Mus_musculus?pops=1

Hi,
I running a script, it presents this problem:

File "E:\python\lib\site-packages\requests\models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 503 Server Error: Server Temporarily Unavailable for url: http://grcm39.rest.ensembl.org/variation/Mus_musculus?pops=1

OS: win10, x64.
Thanks

Failed to fetch metadata for icons

Hi,
I'm trying to clone a subdirectory as linked below via ghclone command:
https://github.com/all-contributors/all-contributors/tree/master/website/static/img/icons

This is the command I gave in git bash inside my local git repo project directory:
MINGW64 ~src/site/images/icons
$ ghclone https://github.com/all-contributors/all-contributors/tree/master/website/static/img/icons
Cloning into 'icons'...
Failed to fetch metadata for icons

I'm using windows 10 with Python 3.11. This is the installed directory of ghclone in my machine:
C:\Windows\system32>pip install github-clone
Requirement already satisfied: github-clone in c:\users\abdul\appdata\local\programs\python\python311\lib\site-packages (1.2.0)
Requirement already satisfied: requests>=2.20.0 in c:\users\abdul\appdata\local\programs\python\python311\lib\site-packages (from github-clone) (2.28.1)
Requirement already satisfied: docopt>=0.6.2 in c:\users\abdul\appdata\local\programs\python\python311\lib\site-packages (from github-clone) (0.6.2)
Requirement already satisfied: charset-normalizer<3,>=2 in c:\users\abdul\appdata\local\programs\python\python311\lib\site-packages (from requests>=2.20.0->github-clone) (2.1.1)
Requirement already satisfied: idna<4,>=2.5 in c:\users\abdul\appdata\local\programs\python\python311\lib\site-packages (from requests>=2.20.0->github-clone) (3.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\abdul\appdata\local\programs\python\python311\lib\site-packages (from requests>=2.20.0->github-clone) (1.26.13)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\abdul\appdata\local\programs\python\python311\lib\site-packages (from requests>=2.20.0->github-clone) (2022.12.7)

Can you please help? I cannot figure out what I'm missing.
Thanks in advance for your assistance.

Failed to fetch metadata

I get error subdirectories of nothing but realtime-colors subdirectory

Failed to fetch metadata realtime-colors

error: raise HTTPError(http_error_msg, response=self)

When I trying to execute the example code ,it presents a problem：

$ ghclone https://github.com/HR/Crypter/tree/master/app
Cloning into 'app'...
Traceback (most recent call last):
  File "g:\python-3.7.1\lib\site-packages\ghclone\ghclone.py", line 80, in clone
    r.raise_for_status()
  File "g:\python-3.7.1\lib\site-packages\requests\models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://api.github.com/repos/HR/Crypter/contents%5Capp?ref=master

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "G:\python-3.7.1\Scripts\ghclone-script.py", line 11, in <module>
    load_entry_point('ghclone==1.1.0', 'console_scripts', 'ghclone')()
  File "g:\python-3.7.1\lib\site-packages\ghclone\ghclone.py", line 143, in main
    clone(api_req_url, rel_url, path, ref)
  File "g:\python-3.7.1\lib\site-packages\ghclone\ghclone.py", line 82, in clone
    exit_with_m('Failed to fetch metadata for ' + path, e)
TypeError: exit_with_m() takes from 0 to 1 positional arguments but 2 were given

In addition to this, when i pip the package ,it report a waring , I don't know if that will make a difference？

 WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=15)")': /packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz

But it should have been installed successfully, as it show the result code:

 Installing collected packages: docopt, ghclone
  Running setup.py install for docopt: started
    Running setup.py install for docopt: finished with status 'done'
  Running setup.py install for ghclone: started
    Running setup.py install for ghclone: finished with status 'done'
Successfully installed docopt-0.6.2 ghclone-1.1.0

My envirement:

$ git --version
git version 2.21.0.windows.1

$ ghclone --version
1.1.0

Usage?

I just tried to clone adafruit:

ghclone https://github.com/adafruit -t mytoken
Invalid GitHub URI

Am I doing something wrong?

Farzam

Farzam
Dsa

Failed to fetch metadata for subdirectory

D:\>ghclone https://github.com/alshedivat/al-folio/tree/master/assets/img
Cloning into 'img'...
Failed to fetch metadata for img

Ran into this error.

Issue with cloning subdirectories which contain large number of files

Github truncates subdirectories which have many files such as https://github.com/iamcal/emoji-data/tree/master/img-apple-64 . As you can see, it says Sorry, we had to truncate this directory to 1,000 files. 2,524 entries were omitted from the list.

When you want to ghclone such subdirectory, it clones only the 1000 files and not all of the files. Is there a workaround?

Thank you.

Farzam

Farzamporkazem

'ghclone' is not recognized as an internal or external command on Windows 10

> C:\Users\sul00\Desktop>pip install git+git://github.com/HR/github-clone#egg=ghclone
> Collecting ghclone
>   Cloning git://github.com/HR/github-clone to c:\users\sul00\appdata\local\temp\pip-install-8oq_uty5\ghclone
> Collecting requests>=2.20.0
>   Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
>      |████████████████████████████████| 61 kB 964 kB/s
> Collecting docopt>=0.6.2
>   Downloading docopt-0.6.2.tar.gz (25 kB)
> Collecting certifi>=2017.4.17
>   Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
>      |████████████████████████████████| 147 kB 1.3 MB/s
> Collecting idna<3,>=2.5
>   Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
>      |████████████████████████████████| 58 kB 692 kB/s
> Collecting chardet<5,>=3.0.2
>   Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
>      |████████████████████████████████| 178 kB 1.6 MB/s
> Collecting urllib3<1.27,>=1.21.1
>   Downloading urllib3-1.26.3-py2.py3-none-any.whl (137 kB)
>      |████████████████████████████████| 137 kB 1.3 MB/s
> Using legacy 'setup.py install' for ghclone, since package 'wheel' is not installed.
> Using legacy 'setup.py install' for docopt, since package 'wheel' is not installed.
> Installing collected packages: certifi, idna, chardet, urllib3, requests, docopt, ghclone
>   WARNING: The script chardetect.exe is installed in 'C:\Users\sul00\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\Scripts' which is not on PATH.
>   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
>     Running setup.py install for docopt ... done
>     Running setup.py install for ghclone ... done
> Successfully installed certifi-2020.12.5 chardet-4.0.0 docopt-0.6.2 ghclone-1.2.0 idna-2.10 requests-2.25.1 urllib3-1.26.3
> WARNING: You are using pip version 20.2.3; however, version 21.0.1 is available.
> You should consider upgrading via the 'C:\Users\sul00\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\python.exe -m pip install --upgrade pip' command.
> 
> C:\Users\sul00\Desktop>ghclone https://github.com/HR/github-clone/tree/master/ghclone
> 'ghclone' is not recognized as an internal or external command,
> operable program or batch file.

Github clone