hexagonit / hexagonit.recipe.download

zc.buildout recipe for downloading and extracting packages

Download recipe for zc.buildout

The recipe downloads packages from the net and extracts them on the filesystem. It is based on the gocept.download recipe with a few additional features.

• Github page: http://github.com/hexagonit/hexagonit.recipe.download
• Clone URL: git://github.com/hexagonit/hexagonit.recipe.download.git
• Issue tracker: http://github.com/hexagonit/hexagonit.recipe.download/issues
• Travis build:

Supported Python versions: 2.6, 2.7, 3.2, 3.3, 3.4, 3.5

Supported zc.buildout versions: 1.x, 2.x

hexagonit.recipe.download's Issues

it is not possible to use recipe without cache

h.r.download is setting default cache directory, even if buildout:download-cache does not exists. In environments where destination files are changing often it does not redownloads changed files from network. It requires to delete those files in buildout:directory/downloads and reinstall part using h.r.download

Wanted behaviour would be possibility to pass parameter to h.r.download to not use cache at all, and only use temporary files. It would be nice if such parameter could be set globally in buildout: namespace, with falling back to part namespace.

Of course, current behaviour of automatic caching is perfect default for most of world.

NOT-ISSUE: Checking for python 3.10.x support

Hi,
Just wanted to check when this recipe will support python3.10.x.
If there is any workaround to make it work with python 3.10.x, please share

Thanks in advance.

Issues with setuptools 0.8 and buildout 2.2.0

I'm using gp.recipe.node which uses hexagonit.recipe.download to download the node binaries. Somehow, the extracted files are corrupt. Using a VM without upgrading setuptools with buildout v1 causes no problems, but upgrading to buildout 2.2.0 with setuptools 0.8 causes it.

For reference and steps to reproduce:
gawel/gp.recipe.node#12

non-unique download cache files

if two or more downloads are specified with the same basename, the cache does not distinguish between the downloaded content.

for example with such urls used by github:

url = https://github.com/hexagonit/hexagonit.recipe.download/tarball/master
url = http://github.com/webpy/webpy/tarball/master

solution:

the recipe should use the hash_name=True argument in zc.buildout's Download constructor
thx, bernd

ignore-existing

ignore-existing is set true, the recipe still attempts to copy source to target.

Allow to specify user agent

Recently jqueryui.com started to reject all requests from Python's urllib.urlretrieve:

Installing jquery-ui.
Downloading http://jqueryui.com/resources/download/jquery-ui-1.11.2.zip
While:
  Installing jquery-ui.
Error: Error downloading extends for URL http://jqueryui.com/resources/download/jquery-ui-1.11.2.zip: ('http error', 403, 'Forbidden', <httplib.HTTPMessage instance at 0x2b24654b3758>)

Maybe changing user agent would solve it. So it would be great to have possibility to change user agent.

allow specifying destination filename if download-only = true

it would be nice if there was a 'filename' option to use when "download-only = true".

Many times the original URL has no information about the intention of the downloaded file, so it would be nice if it was possible to specify the file name in the destination. This should have no impact on the file-name in the download cache directory and should only be effective if "download-only=true"

not SSL/sni friendly

while download on a site requiring sni negociation:

Error: Error downloading extends for URL https://cloud.github.com/downloads/wycats/handlebars.js/handlebars-1.0.0.beta.6.js: (SSLError(1, '_ssl.c:510: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure'),)

py2.7/zc.buildout1.7/setuptools7/ubuntu14.4/libssl1.0.1f-1ubuntu2.8

if download-only is set, we could have a "target" parameter set in the option dict, and set "filename" parameter if not already set

Everything is in the title. If you are ready, I can implement the first, the second, or both, and submit a PR.

We are little by little phasing out our own slapos.recipe.download recipe in favor of this one so that we can share development, and this is the two last drawbacks from switching everything.

NameError: global name 'log' is not defined

While trying to work with https://launchpad.net/bzr-windows-installers I've encountered the following problem inside hexagonit:

C:\work\Bazaar\bzr-windows-installers>py26 build.py --standalone-only installers
Initialising C:\work\Bazaar\bzr-windows-installers\build-win32 directory
Running c:\Python26\python.exe bootstrap.py ...
Creating directory 'C:\work\Bazaar\bzr-windows-installers\build-win32\bin'.
Creating directory 'C:\work\Bazaar\bzr-windows-installers\build-win32\parts'.
Creating directory 'C:\work\Bazaar\bzr-windows-installers\build-win32\eggs'.
Creating directory 'C:\work\Bazaar\bzr-windows-installers\build-win32\develop-eggs'.
Generated script 'C:\work\Bazaar\bzr-windows-installers\build-win32\bin\buildout'.
Generating buildout configuration for bzr-2.2 ...
Generating buildout-bzr-2.2.cfg in .
Running bin\buildout -c buildout-bzr-2.2.cfg ...
Getting distribution for 'hexagonit.recipe.download==1.2.2'.
Got hexagonit.recipe.download 1.2.2.
c:\work\bazaar\bzr-windows-installers\build-win32\eggs\hexagonit.recipe.download-1.2.2-py2.6.egg\hexagonit\recipe\download__ini
t__.py:5: DeprecationWarning: the md5 module is deprecated; use hashlib instead
import md5
Getting distribution for 'gf.recipe.bzr==1.0rc8'.
Got gf.recipe.bzr 1.0rc8.
Installing svn-lib.
svn-lib: Creating download directory: C:\work\Bazaar\bzr-windows-installers\build-win32\downloads
While:
Installing svn-lib.

An internal error occured due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
File "c:\work\bazaar\bzr-windows-installers\build-win32\eggs\zc.buildout-1.4.3-py2.6.egg\zc\buildout\buildout.py", line 1660,
in main
getattr(buildout, command)(args)
File "c:\work\bazaar\bzr-windows-installers\build-win32\eggs\zc.buildout-1.4.3-py2.6.egg\zc\buildout\buildout.py", line 532, i
n install
installed_files = self[part].call(recipe.install)
File "c:\work\bazaar\bzr-windows-installers\build-win32\eggs\zc.buildout-1.4.3-py2.6.egg\zc\buildout\buildout.py", line 1204,
in call
return f()
File "c:\work\bazaar\bzr-windows-installers\build-win32\eggs\hexagonit.recipe.download-1.2.2-py2.6.egg\hexagonit\recipe\downlo
ad__init.py", line 125, in install
base = self.calculate_base(extract_dir)
File "c:\work\bazaar\bzr-windows-installers\build-win32\eggs\hexagonit.recipe.download-1.2.2-py2.6.egg\hexagonit\recipe\downlo
ad__init__.py", line 65, in calculate_base
log.error('Unable to strip top level directory because there are more '
NameError: global name 'log' is not defined

See https://bugs.launchpad.net/bzr-windows-installers/+bug/561240

"mode" support

Hello,
What would you think about a "mode" support in h.r.download? It would just chmod the output file if "download-only" is specified.

The new ignore-existing make this recipe useless in some cases

In older versions of the recipe we where able to overwrite existing files with new ones downloaded by the recipe itself.

Newer version now fail if one or more files are will be overwritten by the recipe.
The new versions introduced this ignore-existing option, but in the case above this is still useless as we can only

• let buildout fail
• ignore existings files

In facts we are no more able to overwrite files.

.gz support

Now hexagonit.recipe.download does not recognize .gz archives, for example, I have this section:

[geo-lite-country]
recipe = hexagonit.recipe.download
url = http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
destination = ${buildout:parts-directory}/geoip
hash-name = false

And running buildout I get this error:

Installing geo-lite-country.
Downloading http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
geo-lite-country: Unable to extract the package /home/sirex/.buildout/downloads/GeoIP.dat.gz. Unknown format.
While:
  Installing geo-lite-country.
Error: Package extraction error

<Remove this issues>

ignore-existing=true doesn't work on Windows

When dest exists, shutil.move calls shutil.copytree, calls os.makedirs, call os.mkdir(dest), which throws an error. This is on Windows, but it's in keeping with the docs for move, copytree and makedirs. The problem shows up on Windows because on other platforms shutil.move usually manages to use os.rename to implement the move. When src and dest are on different filesystems you'd probably get the same issue.

if download-only = true, the destination filename should match the original filename

The version 1.4.0 caused filenames in the cache to be named as hash values by default.

This is fine, but when using:

download-only = true

The filename in the destination ends up being the hash name, which is not usually what was intended. In this cases there is very little chance of a name collision (as is the case in the download cache where names should be hashed by default) so naming the downloaded file in the destination as a hash does not seem like a good idea.