mayeulc / hb-downloader Goto Github PK

A few useful features would be to:

• Add an interactive mode (not sure what counts as interactive, but extend the command line)
• Allow listing games, books, and other media
  • Allow listing games, books, and audio
  • Allow listing bundles and purchases
• Allow downloading individual media, or categories
  • Allow downloading categories
  • Allow downloading individual games, ebooks or others based on their machine name
  • Allow downloading individual bundles or purchases based on their machine name or the key
• Allow listing steam keys, and exporting them
• Compare steam keys with games registered in an account to show unredeemed ones

I have a few more ideas down the line with what we could do with that... :)

Would be cool if you could work directly in a calibre library instead of downloading to a folder.

I mean, you select a folder that is a calibre library.
Skip downloading books that are already downloaded in that library.
Get metadata from Humblebundle?
Add multiple formats to the same entry.

Hello MayeulC

i faced the situation that i wanted to run hb-downloader without cd-ing in to the hb directory.

By default i got this error:

oceanic@batch:[~]$: /opt/hb-downloader/hb-downloader.py
Humble Bundle Downloader v0.5.0
This program is not affiliated nor endorsed by Humble Bundle, Inc.
For any suggestion or bug report, please create an issue at:
https://github.com/MayeulC/hb-downloader/issues

Traceback (most recent call last):
  File "/opt/hb-downloader/hb-downloader.py", line 22, in <module>
    Configuration.load_configuration("hb-downloader-settings.yaml")
  File "/opt/hb-downloader/configuration.py", line 46, in load_configuration
    with open(config_file, "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'hb-downloader-settings.yaml'

I edited ./hb-downloader/hb-downloader.py

Line 22:

old: Configuration.load_configuration("hb-downloader-settings.yaml")
new: Configuration.load_configuration(workdir + "/hb-downloader-settings.yaml")

and in the header section i added:

import os
workdir = (os.path.dirname(os.path.abspath(__file__)))

not sure if the is the way to do it correctly or not first try understanding python

hope this is useful to someone

thank you

I'd appreciate an option to group downloaded files first by platform as in the example tree below.

.
|--------Audio
|      |
|      |--Product01
|
|--------Windows
       |
       |--Product02

For automatic installation and requirement management a setup.py would be a nice enhancement for people not that familiar with python or that can't (or just won't) RTFM. ¯_(ツ)_/¯

Original issue/proposal

Originally asked by @tobias-p in #4.

if possible, could you alter the downloader, so that everything is downloaded by bundle?

For example, I have single games bought in hb where it is ok that the downloader creates a folder called like the game, but another game within a bundle is also downloaded in just a folder called like the game itself.

For example:

bought bioshock in hb store: then download to "/destination/bioshock/..." is ok
got bioshock within a bundle: then download should go to "/destination/bundlename/bioshock/..."

this would make it easier to check out, which bundles are alreday downloaded.

Proposed implementation

The major issue I see with this if using a naive implementation, is that it would take two times as much disk space, and bandwidth to download duplicate games.
I would rather create a new folder, "by-purchase", or something like this, containing one subfolder for each purchase, and symlinks to the actual files.

Proposed organization, that just adds a "by-purchase" folder:

.
├── by-purchase
│   ├── Humble indie bundle X
│   │   ├── product01 -> ../../product01
│   │   └── product02 -> ../../product02
│   └── Humble store product 01
│       └── product01 -> ../../product01
├── product01
│   ├── audio
│   ├── linux
│   ├── mac
│   └── windows
└── product02
    ├── ebook
    ├── linux
    └── windows

See alternative trees at the end of the post.

Now, one question: symlinks, or hardlinks?

• hardlinks remain valid when moving original files around (both the file and the link is an "original file")
• changing content from a hardlink changes the original as well
• symlinks can span multiple filesystems
• symlinks are usually better understood

I have an issue with symlinks, though. On windows, as far as I know, only ntfs supports them (maybe vfat as well), and before the creators update, you had to be root/administrator to create one. Maybe we can try to do so, and fallback to .lnk if that doesn't work.

Alternative organizations:

• group by bundle/store purchase/monthly by creating one such subfolder in by-purchase
  • Alternatively, create three top-level directories by-bundle, by-store-purchase, by-monthly
• to avoid cluttering the top level directory, create another folder, by-product that has the current contents.

Configuration:

I would make this optional (though probably enabled by default), behind a configuration switch. When disabled, she structure should remain the current one.
This might not be easily backwards-compatible if the above feature (storing products in a by-products folder) is implemented. If so, I would probably always leave the downloads in by-product.

It would be nice to download games from the humble trove as well. Unfortunately, that requires an active subscription to develop, as far as I know.

Can this script download from the tribe? If not, it would be a very useful feature.

Whether we perform API calls asynchronously or in parallel, that would be of tremendous help speeding up the script, especially the initial purchase information, that's needed by hb-downloader list among others. As the workload only makes API calls, making them asynchronously could easily speed things up 50 times, depending on the number of purchases. That's a conservative estimate, and it could speed things up a lot more on slower connections.

The other elephant in the room is hashing and downloading, that can be done in parallel (download new files while checking already downloaded ones). This could be really helpful on slower machines. Next steps could be multithreading hashing, and performing multiple download simultaneously.

So, here is the roadmap:

• Simultaneous humble API calls
• Download while hashing
• Perform multiple hashes in parallel

I am not sure what would be the best way to perform this. However, it would be nice if:

• The same library could be used for all of the above, to reduce complexity and dependencies
• We used a built-in python module, not to depend on anything else
• Any extra dependency we brought in was common enough to be found in most distributions' packages

A couple references:

• http://skipperkongen.dk/2016/09/09/easy-parallel-http-requests-with-python-and-asyncio/
• https://github.com/requests/requests-threads
• https://github.com/kennethreitz/grequests
• https://github.com/ross/requests-futures
• https://pythonhosted.org/joblib/parallel.html

I discovered that in addition to the familiar "games, ebooks, audio" platforms, there is now a "video" platform. By default, hb-downloader does not list or download anything from the video platform. A simple change like below to hb-downloader-settings.yaml does successfully download the videos when a plain command of hb-downloader.py download is ran, though, so it seems support should be easy to add?

download-platforms:
  audio: True
  ebook: True
  windows: True
  mac: True
  linux: True
  android: True
  asmjs: False
  video: True

Latest git

[2018/04/18 03:45:10] [    Config] write_md5=True
[2018/04/18 03:45:10] [    Config] read_md5=True
[2018/04/18 03:45:10] [    Config] force_md5=False
[2018/04/18 03:45:10] [    Config] ignore_md5=False
[2018/04/18 03:45:10] [    Config] debug=True
[2018/04/18 03:45:10] [    Config] download_location=/home/parker/backup
[2018/04/18 03:45:10] [    Config] cookie_filename=cookies.txt
[2018/04/18 03:45:10] [    Config] chunksize=8192000
[2018/04/18 03:45:10] [    Config] resume_downloads=True
[2018/04/18 03:45:10] [    Config] Platform audio=False
[2018/04/18 03:45:10] [    Config] Platform ebook=False
[2018/04/18 03:45:10] [    Config] Platform windows=False
[2018/04/18 03:45:10] [    Config] Platform mac=False
[2018/04/18 03:45:10] [    Config] Platform linux=True
[2018/04/18 03:45:10] [    Config] Platform android=True
[2018/04/18 03:45:10] [    Config] Platform asmjs=False
Enter your Authy token: <authycode>
Traceback (most recent call last):
  File "./hb-downloader.py", line 41, in <module>
    hapi.login(ConfigData.username, ConfigData.password, ConfigData.authy_token)
  File "/home/parker/build/hb-downloader/humble_api/humble_api.py", line 137, in login
    error_msg, request=request, response=response, authy_required=authy_required)
humble_api.exceptions.humble_captcha_exception.HumbleCaptchaException: Are you sure you're not a robot? Please try again.

At some point, the downloads start to fail, and the script downloads every file with a 0B size.

I actually have had a pretty good idea of the cause for a while now: this is probably due to the links being generated by humblebundle.com expiring.

There are two ways this could be fixed: generate links just before downloading instead of ahead-of-time, or regenerate them when there is an issue.
The first solution probably adds some latency, but that's probably not a big problem. The second one is more hacky, and probably more complicated to implement.

There is another, probably better way: generate small batches of links, and investigate the timeout.

-- Assuming links are indeed the issue here.