emily3403 / isisdl Goto Github PK

View Code? Open in Web Editor NEW

32.0 32.0 4.0 20.67 MB

A downloader for ISIS of TU-Berlin

License: GNU General Public License v3.0

Shell 0.65% Python 99.17% Batchfile 0.14% Roff 0.05%

isisdl's People

Contributors

Stargazers

Watchers

Forkers

lhoenig thehllm

isisdl's Issues

isisdl-config crashes when cron is not installed

Issue: isisdl-config attempts to access /usr/bin/crontab and crashes if it doesn't exist.

Proposed solution:

isisdl-config should access crontab only if (and after) the user requested a cron-job to be created/deleted.
isisdl-config should show an error message if crontab doesn't exist, prompting the user to install it before rerunning isidl-config and the exit gracefully. (only if user requested a cron-job to be created)
Bonus: For maximum user comfort, include a systemd-timer-option for systems without cron.

OS: Archlinux (Linux Kernel Ver. 5.15.12-arch1-1)

Traceback:

Traceback (most recent call last):
  File "/home/<user>/.local/bin/isisdl-config", line 8, in <module>
    sys.exit(main())
  File "/home/<user>/.local/lib/python3.10/site-packages/isisdl/bin/config.py", line 248, in main
    cron_prompt()
  File "/home/<user>/.local/lib/python3.10/site-packages/isisdl/bin/config.py", line 160, in cron_prompt
    with CronTab(user=True) as cron:
  File "/home/<user>/.local/lib/python3.10/site-packages/crontab.py", line 228, in __init__
    self.read(tabfile)
  File "/home/<user>/.local/lib/python3.10/site-packages/crontab.py", line 291, in read
    (out, err) = open_pipe(self.cron_command, l='', **self.user_opt).communicate()
  File "/home/<user>/.local/lib/python3.10/site-packages/crontab.py", line 190, in open_pipe
    return sp.Popen(args, stdout=sp.PIPE, stderr=sp.PIPE, env=env)
  File "/usr/lib/python3.10/subprocess.py", line 966, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1842, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/crontab'
Exception ignored in atexit callback: <function OnKill.exit at 0x7fb23afc2830>
Traceback (most recent call last):
  File "/home/<user>/.local/lib/python3.10/site-packages/isisdl/backend/utils.py", line 235, in exit
    import isisdl.backend.request_helper as request_helper
  File "/home/<user>/.local/lib/python3.10/site-packages/isisdl/backend/request_helper.py", line 9, in <module>
    from concurrent.futures import ThreadPoolExecutor
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/usr/lib/python3.10/concurrent/futures/__init__.py", line 49, in __getattr__
    from .thread import ThreadPoolExecutor as te
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 37, in <module>
    threading._register_atexit(_python_exit)
  File "/usr/lib/python3.10/threading.py", line 1497, in _register_atexit
    raise RuntimeError("can't register atexit after shutdown")
RuntimeError: can't register atexit after shutdown

Feature Request: add argument to change encoder and preset used by ffmpeg

Being able to run for example:

isisdl --compress --encoder=hevc_nvenc --preset=fast

would make it easy to make use of hardware acceleration with out isisdl having to probe for supported encoders etc.

Feature Request: Propagation of choice implications to posterior prompts in config wizard

Fist of all: Thanks for developing this super useful tool!

Feature request:
Currently selecting for example cleaned file paths e.g. 'Course one-a' -> 'CourseOneA' doesn't influence posterior config wizard pages.
It would be awesome if that could be implemented.

Filename conflict not detected

Hi,
I'm enrolled in https://isis.tu-berlin.de/course/view.php?id=31139 (Logik, should be publicly enrollable). There is a file called "freiwillige Hausaufgabe 01 Datei", which has the filename ha01.pdf and another file called 1. Hausarbeit Datei which has the same filename ha01.pdf.

According to the wiki of isisdl, this should be detected right? Unfortunately it isn't and zhere is only one file called ha01.pdf in the download folder of isisdl. With every download isisdl seemingly picks one by random and re-downloads it (probably due to race conditions it changes).

Is there a way to fix that? If you're not able to reproduce it based on my information, I'm happy to provide more logs.

> isisdl -v 
isisdl version 1.3.14

Running on Linux
This is the compiled version

working directory: /home/thomas/Documents/uni/isisdl
python executable: /home/thomas/Documents/uni/isisdl/Logik/isisdl

database_version = 2
has_ffmpeg = True
forbidden_chars = {'/', '\x00'}
fstype = 'ext4'

> uname -a
Linux thomas-manjaro 6.0.8-1-MANJARO #1 SMP PREEMPT_DYNAMIC Thu Nov 10 20:52:34 UTC 2022 x86_64 GNU/Linux

# Part of isisdl logs
Name: 'ha01.pdf'
Course: 31139
Size: 121.11 KiB
Time: 2022-10-23 22:16:09
Is downloaded: True
Path: /home/thomas/Documents/uni/isisdl/Logik/ha01.pdf

# ...... other files

Name: 'ha01.pdf'
Course: 31139
Size: 145.83 KiB
Time: 2022-11-25 11:58:14
Is downloaded: False
Path: /home/thomas/Documents/Uni/isisdl/Logik/ha01.pdf

Course titles with slashes are interpreted as subdirectories

For course titles with slashes '/', subdirectories will be created: for example for the course with title MI1 (WS22/23), a directory MI1 (WS22 will be created containing a directory 23), where the course content will be stored.

Slashes should be replaced with a nonsemantic char, for example an underscore or space.

--init: text for safe mode for filenames is missing actual question

`isisdl --compress` doesn't find non-existent video

As the title suggests, launching isisdl --compress ends up crashing, because a non-existent video doesn't get found.
I've checked isis and found that the video is available for manual downloading there, but isisdl didn't download it or removed it somehow.

Isisdl output:


Processing ...

Total time: 00:35:06                                  
Total videos: 37 / 312                                
Total time / GB: 950.63s                              
                                                      
Total size before:     73.27 GiB                      
Total size now:        73.27 GiB                      
Total size remaining:  71.31 GiB                      
Total size skipped:     2.18 GiB                      
                                                      
Global efficiency: -89.29%                            
                                                      
Currently processing:                                 
Lecture3Video.mp4                                     
                                                      
Percent done: 99.88%                                  
Finished in:  00:00:00                                
Time elapsed: 00:03:58                                
                                                      
Original  file size:  64.49 MiB                       
Current   file size:  52.90 MiB                       
Estimated file size:  48.92 MiB                       
                                                      
Estimated efficiency: -24.13%                         
Compression score:      1.74                          

I have encountered the following Exception. I'm sorry this happened 😔

Traceback (most recent call last):
  File "/home/matteo/.local/lib/python3.10/site-packages/isisdl/compress.py", line 484, in compress
    compress_status.done_thing(ret_code == 0)
  File "/home/matteo/.local/lib/python3.10/site-packages/isisdl/compress.py", line 223, in done_thing
    new_file_size = os.stat(make_temp_filename(self.cur_file) if was_successful else self.cur_file.path).st_size
FileNotFoundError: [Errno 2] No such file or directory: '/home/matteo/Isis/SoSe2021ML2/Videos/Lecture3Video.mp4'

I have logged this error to the file
/home/matteo/Isis/.errors/1653577444.txt

Bug: Login not working

Feature Request: Archive old file versions

Hey, an idea for a new feature: it happens from time to time that courses update already uploaded files. When running isisdl, the new file is downloaded and the old version is overwritten as far as I can see. Would it be possible for isisdl to pick up on the fact that an older version of the file has already been downloaded and move it something like <basename-wo-ext>-<datetime>.<ext> instead of overwriting it?

This behavior could then also be made configurable.

MacOS Support

Currently isisdl is not working for MacOS. The Problem is somewhere in the underlying socket implementation. Since I don't own a Mac this problem is kinda hard for me to debug. So if anyone with Mac would like to help it would be greatly appreciated.

Feature Request: Find duplicates on modern file systems

This tool is great, thanks for your hard work! It saved me from a lot of anger when ISIS was down. I have one more suggestion:

Sometimes you want to download an old ISIS course because it contains useful information and may already have material that the current course does not. (Future exercises, to work ahead, or videos from online teaching that have now been replaced with in-person tutoring)
Of course content such as exercises or current information can change, which is why it's handy to have both courses.

However, duplicates can often be found, e.g. in videos. To save disk space you could benefit from the copy-on-write feature of modern file systems like BTRFS, XFS or APFS. Maybe you can use https://github.com/jbruchon/jdupes for this.

Would be cool to have the feature in isisdl, so that files are not downloaded multiple times but only cloned. For example, if an ISIS course from the summer semester 2022 gets a new video, which is already included in the 2021 course, you should not have to download it separately but instead use a copy on write feature or perhaps a symlink on older file systems.

Maybe this is just a edge case and not worth programming, but I know some people who access old ISIS courses because lecturers refuse to provide material in advance for some irrational reason.

Incompatibility with `packaging` 22.0

They removed LegacyVersion, which is used in isisdl (utils.py).

Repeated calling of `isisdl --compress` just updating

System:
OS: Arch Linux
isisdl --version yields:

isisdl version 1.3.6


Build info:

is_static = False
current_database_version = 2
has_ffmpeg = True
forbidden_chars = {'\x00', '/'}
fstype = 'btrfs'

Current behaviour:
Launching isisdl --compress (repeatedly) yields the following.


There is a new version of isisdl available: 1.3.7 (current: 1.3.6).
According to your update policy I will auto-install it.

Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: isisdl in ./.local/lib/python3.10/site-packages (1.3.7)
Collecting isisdl
  Using cached isisdl-1.3.7-py3-none-any.whl (67 kB)
  Using cached isisdl-1.3.6-py3-none-any.whl (67 kB)
Requirement already satisfied: cryptography>=35 in /usr/lib/python3.10/site-packages (from isisdl) (37.0.0)
Requirement already satisfied: requests>=2 in /usr/lib/python3.10/site-packages (from isisdl) (2.27.1)
Requirement already satisfied: pyyaml>=6 in ./.local/lib/python3.10/site-packages (from isisdl) (6.0)
Requirement already satisfied: packaging>=21 in ./.local/lib/python3.10/site-packages (from isisdl) (21.3)
Requirement already satisfied: colorama>=0.4 in /usr/lib/python3.10/site-packages (from isisdl) (0.4.4)
Requirement already satisfied: distro>=1 in /usr/lib/python3.10/site-packages (from isisdl) (1.7.0)
Requirement already satisfied: psutil>=5 in /usr/lib/python3.10/site-packages (from isisdl) (5.9.0)
Requirement already satisfied: pyinotify>=0.9 in ./.local/lib/python3.10/site-packages (from isisdl) (0.9.6)
Requirement already satisfied: cffi>=1.12 in /usr/lib/python3.10/site-packages (from cryptography>=35->isisdl) (1.15.0)
Requirement already satisfied: pycparser in /usr/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=35->isisdl) (2.21)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/lib/python3.10/site-packages (from packaging>=21->isisdl) (3.0.9)
Requirement already satisfied: chardet>=3.0.2 in /usr/lib/python3.10/site-packages (from requests>=2->isisdl) (4.0.0)
Requirement already satisfied: idna>=2.5 in /usr/lib/python3.10/site-packages (from requests>=2->isisdl) (3.3)
Requirement already satisfied: urllib3>=1.21.1 in /usr/lib/python3.10/site-packages (from requests>=2->isisdl) (1.26.9)

Successfully updated!

Expected behaviour:
Launching isisdl --compress compresses the videos that have been produced by isisdl.