malloydelacroix / downloaderforreddit Goto Github PK

So, i've tried pretty much every version of the software. The program keeps crashing for me when i try to edit my settings.
Prior 2.1.0 it used to freeze then give me the option to end the process... now after the "save fix" it only crashes instantly.
I've tried deleting the save file, re-import the save and everything i could think of but nothing seems to work.

The only thing i can think of is reinstalling the .net framework? would that help? thanks.

Using the last version but when I try to save settings, the program exists completely and doesn't save any settings!!!
Also tried the previous version and the same happens.
Is there a way to edit the settings, like a config file?? If yes, where is located?? Couldn't find it...

In the latest provided release (zipped dfr 1.1.0 for windows).
Don't know if it is a false positive or not.

Steps to reproduce
1- open a user
2- right click "restrict by date" option
3- choose any 2 reset options in menu
4- crash

Version- 2.3.0 -master
commit- 49a38be

python- 3.7.0-windows stable

Error Log:
https://imgur.com/a/hqaPLX2

A window to allow us to enable which sites we want videos to be downloaded from and which we want disabled since some sites do not work or if we do not want certain sites to download.
The new video downloader does not work 100% as if I try download a user and if for example they have p0rnhub link it does not download thus going into an infinite page download.But what if I want to disable that site all together? Removing said site from supported_video_sites .txt does not work as it just say cannot find extractor but it will still try to download it anyway.
something like:

-----------------------------------|III|-----------------------------
     sites enabled                 |III|          availables sites           
-----------------------------------|III| ----------------------------
Youtube                            |III|  P0rnhub
Vimeo                              |III|  AMC
etc                                |III|  etc

Using my two wallpaper/s groups as before, the app says both are valid, then sits at 50% seemingly indefinitely. Logs below.

{"levelname": "ERROR", "name": "DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "load_pickled_state", "lineno": 87, "message": "Failed to load from save file", "asctime": "03/16/2018 09:11:25 AM", "exc_info": "Traceback (most recent call last):\n File "shelve.py", line 111, in getitem\nKeyError: 'user_list_models'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "DownloaderForReddit\Persistence\ObjectStateHandler.py", line 62, in load_pickled_state\n File "shelve.py", line 113, in getitem\n File "dbm\dumb.py", line 148, in getitem\nKeyError: b'user_list_models'", "save_file_location": "C:\Users\Craig\AppData\Roaming\SomeGuySoftware\DownloaderForReddit\save_file"}
{"levelname": "ERROR", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "load_state", "lineno": 1329, "message": "Load state error: No save file found", "asctime": "03/16/2018 09:11:25 AM", "exc_info": "Traceback (most recent call last):\n File "DownloaderForReddit\GUI\DownloaderForRedditGUI.py", line 1312, in load_state\nTypeError: 'NoneType' object is not subscriptable"}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "init", "lineno": 47, "message": "Settings dialog opened", "asctime": "03/16/2018 09:11:31 AM"}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "set_imgur_client", "lineno": 201, "message": "Imgur client_id and client_secret set", "asctime": "03/16/2018 09:11:36 AM", "valid_client_id": true, "valid_client_secret": true}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "save_settings", "lineno": 307, "message": "Settings saved", "asctime": "03/16/2018 09:11:46 AM", "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "update_user_settings", "lineno": 1056, "message": "User settings updated", "asctime": "03/16/2018 09:11:46 AM", "updated_users": 0, "total_users": 0}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "update_subreddit_settings", "lineno": 1070, "message": "Subreddit settings updated", "asctime": "03/16/2018 09:11:46 AM", "updated_subreddits": 0, "total_subreddits": 0}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "run_subreddit", "lineno": 474, "message": "Subreddit download initiated", "asctime": "03/16/2018 09:12:14 AM", "list_size": 2, "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "start_reddit_extractor_thread", "lineno": 536, "message": "Downloader thread started", "asctime": "03/16/2018 09:12:14 AM"}
{"levelname": "CRITICAL", "name": "DownloaderForReddit.main", "filename": "main.py", "module": "main", "funcName": "log_unhandled_exception", "lineno": 86, "message": "Unhandled exception", "asctime": "03/16/2018 09:13:02 AM", "exc_info": "Traceback (most recent call last):\n File "lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen\n File "lib\site-packages\urllib3\connectionpool.py", line 346, in _make_request\n File "lib\site-packages\urllib3\connectionpool.py", line 850, in validate_conn\n File "lib\site-packages\urllib3\connection.py", line 326, in connect\n File "lib\site-packages\urllib3\util\ssl.py", line 329, in ssl_wrap_socket\n File "ssl.py", line 401, in wrap_socket\n File "ssl.py", line 808, in init\n File "ssl.py", line 1061, in do_handshake\n File "ssl.py", line 683, in do_handshake\nConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "lib\site-packages\requests\adapters.py", line 440, in send\n File "lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen\n File "lib\site-packages\urllib3\util\retry.py", line 357, in increment\n File "lib\site-packages\urllib3\packages\six.py", line 685, in reraise\n File "lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen\n File "lib\site-packages\urllib3\connectionpool.py", line 346, in _make_request\n File "lib\site-packages\urllib3\connectionpool.py", line 850, in validate_conn\n File "lib\site-packages\urllib3\connection.py", line 326, in connect\n File "lib\site-packages\urllib3\util\ssl.py", line 329, in ssl_wrap_socket\n File "ssl.py", line 401, in wrap_socket\n File "ssl.py", line 808, in init\n File "ssl.py", line 1061, in do_handshake\n File "ssl.py", line 683, in do_handshake\nurllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "DownloaderForReddit\Core\DownloadRunner.py", line 390, in run_extraction\n File "DownloaderForReddit\Core\DownloadRunner.py", line 414, in extract\n File "DownloaderForReddit\Extractors\Extractor.py", line 52, in run\n File "DownloaderForReddit\Extractors\Extractor.py", line 62, in extract\n File "DownloaderForReddit\Extractors\ImgurExtractor.py", line 56, in init\n File "lib\site-packages\imgurpython\client.py", line 85, in init\n File "lib\site-packages\imgurpython\client.py", line 94, in get_credits\n File "lib\site-packages\imgurpython\client.py", line 131, in make_request\n File "lib\site-packages\requests\api.py", line 72, in get\n File "lib\site-packages\requests\api.py", line 58, in request\n File "lib\site-packages\requests\sessions.py", line 508, in request\n File "lib\site-packages\requests\sessions.py", line 618, in send\n File "lib\site-packages\requests\adapters.py", line 490, in send\nrequests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))"}
{"levelname": "INFO", "name": "DownloaderForReddit.Persistence.SettingsManager", "filename": "SettingsManager.py", "module": "SettingsManager", "funcName": "check_first_run", "lineno": 60, "message": "First run of new version", "asctime": "03/17/2018 08:42:16 AM", "new_version": "v2.2.1", "old_version": "v2.2.0"}
{"levelname": "ERROR", "name": "DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "load_pickled_state", "lineno": 87, "message": "Failed to load from save file", "asctime": "03/17/2018 08:42:17 AM", "exc_info": "Traceback (most recent call last):\n File "shelve.py", line 111, in getitem\nKeyError: 'user_list_models'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "DownloaderForReddit\Persistence\ObjectStateHandler.py", line 62, in load_pickled_state\n File "shelve.py", line 113, in getitem\n File "dbm\dumb.py", line 148, in getitem\nKeyError: b'user_list_models'", "save_file_location": "C:\Users\Craig\AppData\Roaming\SomeGuySoftware\DownloaderForReddit\save_file"}
{"levelname": "ERROR", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "load_state", "lineno": 1329, "message": "Load state error: No save file found", "asctime": "03/17/2018 08:42:17 AM", "exc_info": "Traceback (most recent call last):\n File "DownloaderForReddit\GUI\DownloaderForRedditGUI.py", line 1312, in load_state\nTypeError: 'NoneType' object is not subscriptable"}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "init", "lineno": 47, "message": "Settings dialog opened", "asctime": "03/17/2018 08:42:23 AM"}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "set_imgur_client", "lineno": 201, "message": "Imgur client_id and client_secret set", "asctime": "03/17/2018 08:42:28 AM", "valid_client_id": true, "valid_client_secret": true}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "save_settings", "lineno": 307, "message": "Settings saved", "asctime": "03/17/2018 08:42:32 AM", "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "update_user_settings", "lineno": 1056, "message": "User settings updated", "asctime": "03/17/2018 08:42:32 AM", "updated_users": 0, "total_users": 0}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "update_subreddit_settings", "lineno": 1070, "message": "Subreddit settings updated", "asctime": "03/17/2018 08:42:32 AM", "updated_subreddits": 0, "total_subreddits": 0}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "run_subreddit", "lineno": 474, "message": "Subreddit download initiated", "asctime": "03/17/2018 08:43:18 AM", "list_size": 2, "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "start_reddit_extractor_thread", "lineno": 536, "message": "Downloader thread started", "asctime": "03/17/2018 08:43:30 AM"}
{"levelname": "ERROR", "name": "DownloaderForReddit.Extractors.Extractor", "filename": "Extractor.py", "module": "Extractor", "funcName": "extract", "lineno": 82, "message": "Failed to extract content: Unknown error", "asctime": "03/17/2018 09:10:46 AM", "exc_info": "Traceback (most recent call last):\n File "lib\site-packages\urllib3\connection.py", line 141, in _new_conn\n File "lib\site-packages\urllib3\util\connection.py", line 83, in create_connection\n File "lib\site-packages\urllib3\util\connection.py", line 73, in create_connection\nTimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen\n File "lib\site-packages\urllib3\connectionpool.py", line 346, in _make_request\n File "lib\site-packages\urllib3\connectionpool.py", line 850, in _validate_conn\n File "lib\site-packages\urllib3\connection.py", line 284, in connect\n File "lib\site-packages\urllib3\connection.py", line 150, in _new_conn\nurllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x032383F0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "lib\site-packages\requests\adapters.py", line 440, in send\n File "lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen\n File "lib\site-packages\urllib3\util\retry.py", line 388, in increment\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.imgur.com', port=443): Max retries exceeded with url: /3/credits (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x032383F0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond',))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "DownloaderForReddit\Extractors\Extractor.py", line 62, in extract\n File "DownloaderForReddit\Extractors\ImgurExtractor.py", line 56, in init\n File "lib\site-packages\imgurpython\client.py", line 85, in init\n File "lib\site-packages\imgurpython\client.py", line 94, in get_credits\n File "lib\site-packages\imgurpython\client.py", line 131, in make_request\n File "lib\site-packages\requests\api.py", line 72, in get\n File "lib\site-packages\requests\api.py", line 58, in request\n File "lib\site-packages\requests\sessions.py", line 508, in request\n File "lib\site-packages\requests\sessions.py", line 618, in send\n File "lib\site-packages\requests\adapters.py", line 508, in send\nrequests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.imgur.com', port=443): Max retries exceeded with url: /3/credits (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x032383F0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond',))", "url": "https://i.imgur.com/wjk5QuI.jpg", "reddit_object": {"name": "wallpaper", "object_type": "SUBREDDIT", "version": "v2.2.1", "save_path": "U:/Downloads/Reddit Downloader/", "post_limit": 1000, "avoid_duplicates": true, "download_videos": true, "download_images": true, "nsfw_filter": "INCLUDE", "added_on": 1521240191.041695, "do_not_edit": false, "new_submission_count": 925, "saved_submission_count": 0, "previous_download_count": 4076, "date_limit": 1521235400.0, "custom_date_limit": null, "content_count": 4076, "failed_extract_count": 1, "saved_content_count": 0}}
{"levelname": "INFO", "name": "DownloaderForReddit.Core.DownloadRunner", "filename": "DownloadRunner.py", "module": "DownloadRunner", "funcName": "stop_download", "lineno": 347, "message": "Downloader stopped", "asctime": "03/17/2018 09:11:18 AM", "run_time": "28 mins, 0 secs"}
{"levelname": "ERROR", "name": "DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "save_pickled_state", "lineno": 129, "message": "Unable to save to save_file", "asctime": "03/17/2018 09:13:10 AM", "exc_info": "Traceback (most recent call last):\n File "DownloaderForReddit\Persistence\ObjectStateHandler.py", line 122, in save_pickled_state\n File "shelve.py", line 124, in setitem\nTypeError: can't pickle Content objects", "save_file_location": "C:\Users\Craig\AppData\Roaming\SomeGuySoftware\DownloaderForReddit\save_file"}
{"levelname": "ERROR", "name": "DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "save_pickled_state", "lineno": 129, "message": "Unable to save to save_file", "asctime": "03/17/2018 09:13:14 AM", "exc_info": "Traceback (most recent call last):\n File "DownloaderForReddit\Persistence\ObjectStateHandler.py", line 122, in save_pickled_state\n File "shelve.py", line 124, in setitem\nTypeError: can't pickle Content objects", "save_file_location": "C:\Users\Craig\AppData\Roaming\SomeGuySoftware\DownloaderForReddit\save_file"}

I have a subreddit list with two groups, "wallpaper" and "wallpapers". Although it has download a combined few thousand files, it keeps crashing whenever I start the download. Log file below ...

{"levelname": "INFO", "name": "DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "load_pickled_state", "lineno": 83, "message": "Object lists loaded from save file", "asctime": "03/03/2018 05:39:15 PM", "total_users": 0, "total_subreddits": 32, "updated_users": 0, "updated_subreddits": 0}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "load_state", "lineno": 1317, "message": "Save_file successfully loaded into gui", "asctime": "03/03/2018 05:39:15 PM"}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "run_subreddit", "lineno": 474, "message": "Subreddit download initiated", "asctime": "03/03/2018 05:39:18 PM", "list_size": 2, "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "start_reddit_extractor_thread", "lineno": 536, "message": "Downloader thread started", "asctime": "03/03/2018 05:39:18 PM"}
{"levelname": "CRITICAL", "name": "DownloaderForReddit.main", "filename": "main.py", "module": "main", "funcName": "log_unhandled_exception", "lineno": 86, "message": "Unhandled exception", "asctime": "03/03/2018 05:39:43 PM", "exc_info": "Traceback (most recent call last):\n File "DownloaderForReddit\Core\DownloadRunner.py", line 385, in extract\n File "DownloaderForReddit\Core\RedditObjects.py", line 195, in load_unfinished_downloads\nTypeError: init() missing 1 required positional argument: 'display_only'"}
{"levelname": "INFO", "name": "DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "load_pickled_state", "lineno": 83, "message": "Object lists loaded from save file", "asctime": "03/03/2018 05:41:26 PM", "total_users": 0, "total_subreddits": 32, "updated_users": 0, "updated_subreddits": 0}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "load_state", "lineno": 1317, "message": "Save_file successfully loaded into gui", "asctime": "03/03/2018 05:41:26 PM"}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "display_imgur_client_information", "lineno": 1180, "message": "Imgur client info calculated", "asctime": "03/03/2018 05:41:32 PM", "remaining_app_credits": 5567, "remaining_user_credits": 500}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "run_subreddit", "lineno": 474, "message": "Subreddit download initiated", "asctime": "03/03/2018 05:41:57 PM", "list_size": 2, "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "start_reddit_extractor_thread", "lineno": 536, "message": "Downloader thread started", "asctime": "03/03/2018 05:41:57 PM"}
{"levelname": "CRITICAL", "name": "DownloaderForReddit.main", "filename": "main.py", "module": "main", "funcName": "log_unhandled_exception", "lineno": 86, "message": "Unhandled exception", "asctime": "03/03/2018 05:42:23 PM", "exc_info": "Traceback (most recent call last):\n File "DownloaderForReddit\Core\DownloadRunner.py", line 385, in extract\n File "DownloaderForReddit\Core\RedditObjects.py", line 195, in load_unfinished_downloads\nTypeError: init() missing 1 required positional argument: 'display_only'"}

I know this is on your todo list. I've put some thought into this too and I'm wondering how I can help. I don't really have time right now, but maybe if we can split it up into small chunks it'll be easier to do.

To ORM or not to ORM?

Should we use the stdlib sqlite3 package or an ORM?

• https://docs.python.org/3/library/sqlite3.html
• Peewee ORM http://docs.peewee-orm.com/en/latest/
• PonyORM https://ponyorm.com/
• SQLAlchemy https://www.sqlalchemy.org/

Initial migration path

1. Hard migrations – we have a one time migration. One and done. This would mean everything has to get converted to sqlite
2. Gradual migration – migrate one model at a time. RedditObjects, Redditors, Redditor preferences, Subreddits, Subreddit preferences, system preferences, etc.

Maintenance

Deleting old data? Vacuums? Schema migrations?

I generally notice with these new commits the general performance slowdowns when downloading, takes much longer to extract than usual.Would like to see some speedups in the future.Perhaps SQL-lite backend and more efficient processing on the part of the extractors. from commit 55e8789 to 3ec6f1e is a noticable difference

Just trying to save settings without changing anything and it crashes lmao. Also why is the default save path C:\Users\<user>/Downloads/. Windows 7 btw

Could it be possible to determine location where to save save_file.dat?

Some stuff I don't want to keep on Windows partition as Windows gets clean installed few times a year. So it's better to not have files there - easier & faster & safer.

I noticed this just now, with nothing going on credits go down on their own

https://imgur.com/a/CTyT6VP

ver- 2.3.0-master
commit- 9f1f509
python- 3.7.0-windows stable

Upgrading from ver: 2.3.0-master commit: 55e8789
to
commit: 3ec6f1e
App will no load due to missing attribute error
commit: e63a253 seems to be the suspect

Log:
https://imgur.com/a/8giX6vi

rolling back to last working commit

I'm not quite sure does that work, it seems that every time DFR dl's a bunch of duplicates... :/

Then again I had Reddit Data Ext before DFR and I didn't delete old content...

I'm using a tool to find exact duplicates after DL with DFR to sort out old duplicates compared to new.

Hmm I'm not sure ~ perhaps RDE has few moe settings to fine tune. That client either doesn't 'find' other clients data as something of previous.

Noticed about a week ago that this error comes up when it hits a gfy link.

Happens in both Win10 and Linux.

version: 2.3.1?-master
commit: 842be61
python: 3.7.0-stable windows

https://imgur.com/a/z7pCizh

I changed the version.py like last time and it didn't work

The imgurpython library is no longer being maintained. The imgur extractor currently makes heavy use of this library and will need to be updated to access the imgur api without the library.

The library is still working at the moment, but there has been an update in imgur's api documentation, so it is likely only a matter of time before it becomes non-operational.

idk best way to title this but here the issue,app I know is designed for any reason it cannot download an imgur imgage because of credits,over capacity or etc what happens(or should happen) is that link get saved for next download run.So when you download again it should get the image it could not get for some reason last time.In this version I believe that is not properly working .Imgur images that fail to download do not download next run, but prob is that it triggers a crash as described in issue #21 and #27 .Crash triggers right away making the user undownloadable. Only way to fix it is to remove user then readd user and redownload that user..

Steps to reproduce
1- start with very low imgur credits
2- add a user (that has imgur images more than your credits)
3- open user
4- download
5- crash

Version- 2.3.0 -master
commit- 49a38be

python- 3.7.0-windows stable

It caches tons of Reddit subreddits. urls are similar as on actual Reddit domain - like pholder.com/r/feminism AND user pages like: pholder.com/u/feminist

^ just an example ^ could it be possible for this app to check pholder_dot_com/* if it has /r/ or /u/ content cached (pholder_dot_com can go years back.. :D) and possibly DL if usual places are empty?

Or provide links of non-downloaded but available content on DFR's user page? The log can become overwhelming to read and copy paste links..

Just ideas!

I like the app but it 's getting annoying when my hourly user credits get burned each time I download users with avoid duplicates on.Thought the purpose for it was to remember what content it downloaded and save it in a database this way next time I download it knows better to make a call for it.Literally when my hourly credits reset I am left with just 50 credits (from 500) just to download 6 imgur pics.I notice credits burn and nothing is downloaded.

I'm trying all I can... I just cant get the app to read/save imported save file :/

1.) click a user
2.) click download this user
3.) after process completes i click save
4.) click help> view imgur credits
5.) crash

edit: also crashes while trying view credit count DURING downloading

Happens sometimes randomly, happens more if you have more than 1 download thread set.
Steps:
1-click download
2-while downloading crash sometimes occurs randomly

Version: 2.3.0-master commit: 4a321df
Python: 3.7.0-windows

Screenshot:
https://imgur.com/a/w1S4bC2

App is downloading Gifv instead of .webm/.mp4, gifv are not loadable they just contain metadata of a sort.

Some mp4 / webm files comes as .gifv files to local...?

What to do? Try to re-DL?

Some code assumes you're running from the DownloaderForReddit/ directory, but the tests are then in ../Tests. There's lots of other weird things that prevent the normal ways of getting python -m unittest from being able to discover and run the tests.

I can get tests to start running if I'm in the project root and do PYTHONPATH=DownloaderForReddit/ python -m unittest but then I'm in the wrong directory for open(supported_sites_file, 'r') to work.

Hi,

I don't have any logs right now, but this one should be easy enough to replicate.
If you crawl a subreddit, and domain the program is trying to download an image from has expired/doesn't exist, the program will crash.

The same thing seems to happen if the domain exist, but is offline (doesn't respond).
Would it be possible to make the program just skip these entries after a few retries, instead of completely shutting down?

Credits not refreshing at the designated time unless you reload the app.

Ver 2.3.0-master
PY 3.7.0-windows stable
Edit: typo with python version

#jpg being appended to all pictures (jpgs) that is on i.redd.it

Ver: 2.3.0-master commit: 4a321df
Python: 3.7.0-windows

not related to other crash in issue #21
happened during downloading
log: https://imgur.com/a/N8pKTFB
version: 2.3.1?-master
commit: 3ec6f1e
python: 3.7.0-windows stable

Quick question.. about Twitter

Does anyone know a an app similar to this but for Twitter?

In my local branch, I updated the requirements and things seem to be working. In trying to figure out how to contribute that back, I wasn't sure which style of requirements.txt the project is using to match.

https://github.com/MalloyDelacroix/DownloaderForReddit/blob/master/requirements.txt

1. List everything -- it looks like it's listing everything because I see prawcore and six, but requests's requirements and some others are missing
2. Only list the top level requirements -- Then delete deps of deps like prawcore and six
3. Only list the top level requirements and deps of deps if there's a problem -- I would expect to see comments explaining why something had to be kept back

I've used pip-tools and pipenv and I prefer pip-tools. The current versions I get from that are:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
beautifulsoup4==4.6.3
certifi==2018.8.24        # via requests
chardet==3.0.4            # via requests
idna==2.7                 # via requests
imgurpython==1.1.7
praw==6.0.0
prawcore==1.0.0           # via praw
pyqt5-sip==4.19.12        # via pyqt5
pyqt5==5.11.2
python-json-logger==0.1.9
requests==2.19.1
update-checker==0.16      # via praw
urllib3==1.23             # via requests
youtube-dl==2018.8.28

I have been receiving errors after updating from DFR 2.22 to 2.3x - I have tried 2.31 and 2.32. OS Win10. To find the tidiest way to duplicate this problem here's what I did:

• deleted %appdata%\Roaming\SomeGuySoftware\DownloaderForReddit
• started DFR
• asked it to import save file from an old copy I had on my desktop
• confirmed overwrite

After overwrite this error immediately appears, and it also appears if the app is closed and reopened:

If the config directory is purged completely and a new config is created, the app works fine.

See attached log:
DownloaderForReddit.log

I am having a lot of trouble saving the user list.
If I add users then immediately save, it's all good.
If I add users and then start the download, then one of 2 things happens:

1. the final download to complete is a success (an image on a host supported by the app) -> then all is good, the user list and progress can be saved.

2. if the final download to process is a failure (unsupported type) -> unable to save, any users added if not saved before starting the download are lost. The modification date on the save file is not changed.

When scenario 2 happens, the "download" button remains stuck, clicking it to stop does nothing. Attempting to save via the file menu gives a failure message.

The biggest issue with this is not only are users added to the list discarded if not saved before starting the download, but the same content is downloaded over and over again until a scenario # 1 happens and a save can be done properly.

the log shows this when attempting to save, also URL to screen capture of pop up error.

{"levelname": "ERROR", "name": "DownloaderForReddit.DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "save_pickled_state", "lineno": 129, "message": "Unable to save to save_file", "asctime": "12/07/2018 04:32:02 PM", "exc_info": "Traceback (most recent call last):\n File \"DownloaderForReddit\\Persistence\\ObjectStateHandler.py\", line 121, in save_pickled_state\n File \"shelve.py\", line 124, in __setitem__\nTypeError: can't pickle Content objects", "save_file_location": "C:\\Users\\Administrator\\AppData\\Roaming\\SomeGuySoftware\\DownloaderForReddit\\save_file"}

url

DownloaderForReddit works great when downloading from a handful of subreddits at a time but there doesn't seem to be any support for multireddits, when i tried to work around this and add each subreddit manually the program started to crash when a list became greater than ~85 subreddits long. Being able to download relevant images from multiple subreddits in one batch could certainly be quite handy.

Hi -

Windows 7 (DFR version unkown, ZIP named 2.0.2 but the Help>About reports 2.0)

Also, I can't remove lists as the app will crash (Windows appcrash dialog).

Let me know if you need additional information.

Thanks!

Hi,

I was trying to use your GUI based downloader, and i wanted to download the top posts of all time from the subreddit www.reddit.com/r/learnprogramming

I added the subreddit into the subreddit list. Checked download subreddit. I have attached a screenshot of my settings as well.

After all that, i get the error : Failed to extract post: Url domain not supported.

Could you please help me out, let me know what im doing wrong. They are almost all text posts with some posts containing links. No images, no videos etc. Thank you so much for your help and for creating this program.

"There are no user lists available. To add a user, please add a user list"

I've added 2 user lists but its not letting me add a subreddit.

• It's possible to open more than one instance of the program. (make this an option?)
• Clicking [download] places a * in the titlebar as if settings need to be saved again?
• Lists menu - "Rmove Subreddit List"
• Some child windows appear in random locations (annoying on multimonitor). maybe centre to parent?

Import usernames/subreddits from a text file?

A list that I'm following is on at text file, I'd like to import that to the app...
Manually wayyy slow by adding one at a time :/

I try to execute the app but I get an error message: "failed to execute script main". How to solve?

Windows 10.

Downloader suddenly crashes during validation phase. The log reports a Critical level entry caused by: prawcore.exceptions.InvalidToken: received 401 HTTP response.

Thanks for the update. Unfortunately, I can't get it to work at all now.

I completely removed the app and installed it again, including roaming data.
I have added one list "wallpaper" and added the groups "wallpaper" and "wallpapers".
It says both are valid, gets to 50% then crashes. Checking my download folder, nothing is there, not even the subfolders.

Here is my log, thanks.

{"levelname": "ERROR", "name": "DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "load_pickled_state", "lineno": 87, "message": "Failed to load from save file", "asctime": "03/16/2018 09:11:25 AM", "exc_info": "Traceback (most recent call last):\n File "shelve.py", line 111, in getitem\nKeyError: 'user_list_models'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "DownloaderForReddit\Persistence\ObjectStateHandler.py", line 62, in load_pickled_state\n File "shelve.py", line 113, in getitem\n File "dbm\dumb.py", line 148, in getitem\nKeyError: b'user_list_models'", "save_file_location": "C:\Users\Craig\AppData\Roaming\SomeGuySoftware\DownloaderForReddit\save_file"}
{"levelname": "ERROR", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "load_state", "lineno": 1329, "message": "Load state error: No save file found", "asctime": "03/16/2018 09:11:25 AM", "exc_info": "Traceback (most recent call last):\n File "DownloaderForReddit\GUI\DownloaderForRedditGUI.py", line 1312, in load_state\nTypeError: 'NoneType' object is not subscriptable"}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "init", "lineno": 47, "message": "Settings dialog opened", "asctime": "03/16/2018 09:11:31 AM"}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "set_imgur_client", "lineno": 201, "message": "Imgur client_id and client_secret set", "asctime": "03/16/2018 09:11:36 AM", "valid_client_id": true, "valid_client_secret": true}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "save_settings", "lineno": 307, "message": "Settings saved", "asctime": "03/16/2018 09:11:46 AM", "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "update_user_settings", "lineno": 1056, "message": "User settings updated", "asctime": "03/16/2018 09:11:46 AM", "updated_users": 0, "total_users": 0}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "update_subreddit_settings", "lineno": 1070, "message": "Subreddit settings updated", "asctime": "03/16/2018 09:11:46 AM", "updated_subreddits": 0, "total_subreddits": 0}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "run_subreddit", "lineno": 474, "message": "Subreddit download initiated", "asctime": "03/16/2018 09:12:14 AM", "list_size": 2, "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "start_reddit_extractor_thread", "lineno": 536, "message": "Downloader thread started", "asctime": "03/16/2018 09:12:14 AM"}
{"levelname": "CRITICAL", "name": "DownloaderForReddit.main", "filename": "main.py", "module": "main", "funcName": "log_unhandled_exception", "lineno": 86, "message": "Unhandled exception", "asctime": "03/16/2018 09:13:02 AM", "exc_info": "Traceback (most recent call last):\n File "lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen\n File "lib\site-packages\urllib3\connectionpool.py", line 346, in _make_request\n File "lib\site-packages\urllib3\connectionpool.py", line 850, in validate_conn\n File "lib\site-packages\urllib3\connection.py", line 326, in connect\n File "lib\site-packages\urllib3\util\ssl.py", line 329, in ssl_wrap_socket\n File "ssl.py", line 401, in wrap_socket\n File "ssl.py", line 808, in init\n File "ssl.py", line 1061, in do_handshake\n File "ssl.py", line 683, in do_handshake\nConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "lib\site-packages\requests\adapters.py", line 440, in send\n File "lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen\n File "lib\site-packages\urllib3\util\retry.py", line 357, in increment\n File "lib\site-packages\urllib3\packages\six.py", line 685, in reraise\n File "lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen\n File "lib\site-packages\urllib3\connectionpool.py", line 346, in _make_request\n File "lib\site-packages\urllib3\connectionpool.py", line 850, in validate_conn\n File "lib\site-packages\urllib3\connection.py", line 326, in connect\n File "lib\site-packages\urllib3\util\ssl.py", line 329, in ssl_wrap_socket\n File "ssl.py", line 401, in wrap_socket\n File "ssl.py", line 808, in init\n File "ssl.py", line 1061, in do_handshake\n File "ssl.py", line 683, in do_handshake\nurllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "DownloaderForReddit\Core\DownloadRunner.py", line 390, in run_extraction\n File "DownloaderForReddit\Core\DownloadRunner.py", line 414, in extract\n File "DownloaderForReddit\Extractors\Extractor.py", line 52, in run\n File "DownloaderForReddit\Extractors\Extractor.py", line 62, in extract\n File "DownloaderForReddit\Extractors\ImgurExtractor.py", line 56, in init\n File "lib\site-packages\imgurpython\client.py", line 85, in init\n File "lib\site-packages\imgurpython\client.py", line 94, in get_credits\n File "lib\site-packages\imgurpython\client.py", line 131, in make_request\n File "lib\site-packages\requests\api.py", line 72, in get\n File "lib\site-packages\requests\api.py", line 58, in request\n File "lib\site-packages\requests\sessions.py", line 508, in request\n File "lib\site-packages\requests\sessions.py", line 618, in send\n File "lib\site-packages\requests\adapters.py", line 490, in send\nrequests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))"}

I'm not quite sure it's the right place but I've been looking for a way to extract the video and images direct link rather than the content in itself. Is there a way to do that?

And offers and update... (if clicked).

download videos checkbox being ignored in user settings page with "do not overwrite override these settings" checked.
It still tries to download videos even if I set the user not to

Ver: 2.3.0-master commit: 4a321df
Python: 3.7.0-windows

Mouse - right click -> paste username/subredditname into adding field - latest version right click option >>> ### PASTE <<<< missing.

Reddit splits video/audio and this app couldnt grab audio from links, is there anyway to grab audio and merge it with video file on this app ?

I have a few suggestions for this great program.

• Automatically save changes to lists (crashes wiped out my lists the first few times before I realised I had to manually save)
• Option to Backup/restore lists
• Selecting "import save file" then cancelling results in an error
• Include a pause/resume button
• Option to exclude subreddit from downloading without removing it from list
• Option to rename subreddits (if a typo is made when entering. have to remove and readd at the moment)
• Allow adding multiple subreddits to lists by pasting from the clipboard instead of one at a time.

Maybe in the future you could make a version which works with 4chan or similar.

More of an enhancement idea than an issue.

Is there a way or plans to schedule a download at regularly set intervals? I find that some users are deleting posts after they have been up for less than a day and I would like to capture it if I am not at the computer.