fnep / mtv_dl Goto Github PK

The file extension {ext} should automatically be added to the --target.

If the user specifies "--target=test.mp4" the movie will be downloaded and then overwritten with the subtitles and then overwritten with the nfo-file, because all three files write to test.mp4 (see log below).
So the user has to specify {ext} in the target, otherwise the output will be useless. But if the user has to specify it, why not add it automatically?

Just adding {ext} to each --target will be not backwards compatible to older configurations.

Proposal 1: Automatically add {ext} to --target in the script, if {ext} is not existing in --target.
Proposal 2: Give a warning if {ext} is not in --target and subtitles or nfo are generated.

python3 mtv_dl.py download -o --target=test.mp4 hash=2f2bad3ed7c                                        
2020-07-16T14:52:08  INFO     Saved show 'Der erste April (26)' (ARD, 'Meister Eder und sein Pumuckl', 2020-07-05 04:40:00, 2f2bad3ed7c) to PosixPath('test.mp4').                                
2020-07-16T14:52:09  INFO     Saved subtitles 'Der erste April (26)' (ARD, 'Meister Eder und sein Pumuckl', 2020-07-05 04:40:00, 2f2bad3ed7c) to PosixPath('test.mp4').                           
                     INFO     Saved nfo 'Der erste April (26)' (ARD, 'Meister Eder und sein Pumuckl', 2020-07-05 04:40:00, 2f2bad3ed7c) to PosixPath('test.mp4').

The subtitles for some videos prevents VLC and Kodi from playing the file.
If the the subtitle file .srt is deleted, the file plays fine.

Example: Hash 58887acac00, resolution --high

Output of VLC with subtitle:

[00007f16e8c9fe20] main demux error: option sub-original-fps does not exist
[00007f16e8001170] ts demux error: libdvbpsi error (PSI decoder): TS discontinuity (received 2, expected 11) for PID 0
[00007f16e8001170] ts demux error: libdvbpsi error (PSI decoder): TS discontinuity (received 0, expected 10) for PID 4096
[00007f16e8001170] ts demux error: libdvbpsi error (PSI decoder): TS discontinuity (received 3, expected 14) for PID 0
[00007f16e8001170] ts demux error: libdvbpsi error (PSI decoder): TS discontinuity (received 0, expected 12) for PID 4096
[00007f16e8001170] ts demux error: libdvbpsi error (PSI decoder): TS duplicate (received 0, expected 1) for PID 17
[00007f16e8001170] ts demux error: libdvbpsi error (PSI decoder): TS discontinuity (received 4, expected 15) for PID 0
[00007f16e8001170] ts demux error: libdvbpsi error (PSI decoder): TS discontinuity (received 0, expected 12) for PID 4096
[00007f16e8001170] ts demux error: libdvbpsi error (PSI decoder): TS duplicate (received 0, expected 1) for PID 17
[...]

Problem: How to capture each new "Tatort" aired on Sunday 20:15, but not all the repetitions during the week?

Solution: Have additional command line filter parameters for the start field, like weekday, hour, minute.

My current workaround would be to define age-1h and just call the script once on Sunday at 20:30. (But this does not play well with one filter set file).

How to exactly specify the parameters in the option --target on the commandline?
All variants I tried still resulted in using the placeholder-name but not replaced with the content.

python3 mtv_dl.py download --oblivious --target={{dir}}/{{channel}}/{{topic}}/{{start}}_{{title}}{{ext}} hash=2d5b18d8d7c
Downloading 'Wählen Sie Ihren Lieblingsfilm!' (3Sat, '3satZuschauerpreis', 2019-11-23 13:00:00, 2d5b18d8d7c) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:00 
2020-07-14T10:53:18  INFO     Saved show 'Wählen Sie Ihren Lieblingsfilm!' (3Sat, '3satZuschauerpreis', 2019-11-23 13:00:00, 2d5b18d8d7c) to PosixPath('{dir}/{channel}/{topic}/{start}_{title}{ext}').    

From the manual:

-t, --target=<path>                   Directory to put the downloaded files in. May contain
                                        the parameters {{dir}} (from the option --dir),
                                        {{filename}} (from server filename) and {{ext}} (file
                                        name extension including the dot), and all fields from
                                        the listing plus {{date}} and {{time}} (the single parts
                                        of {{start}}).
                                        [default: {{dir}}/{{channel}}/{{topic}}/{{start}} {{title}}{{ext}}]

Hallo fnep,
vielen Dank für dieses tolle Paket. Sicherlich ist das kein Issue, sondern eher ein Bedienfehler.
Ich habe meine Probleme etwas zu downloaden.
Dazu nutze ich die folgende Zeile mit dem Ziel die Videos in den aktuellen Pfad zu kopieren. Ich hatte auch schon eine absolute Pfadangabe nach dem Muster 'E:\Videos' probiert. Aber mit derselben Fehlermeldung.
mtv_dl.exe download topic='Elefant, Tiger' channel='MDR' -h --target='./{title}'

Daraufhin bekomme ich dann die folgenden Fehlermeldung

Ich würde mich freuen, wenn du mir auf die Sprünge helfen kannst.

Vielen Dank und Grüße,
Marcus

Hello again,

I have a weird issue with two separate computers:

1. Raspberry Pi 3B+ running Python 3.7.1
2. Macbook Pro with Ubuntu Budgie running Python 3.6.7

When I start a command like
mtv_dl list topic="Lenas Ranch"
nothing happens until i hit ctrl+C and then it starts executing.
see here pasted from my terminal:

mtv_dl download -v topic="Lenas Ranch"
^C2019-01-31T22:05:18+0100 DEBUG Opening database from 'http://verteiler2.mediathekview.de/Filmliste-akt.xz'.

Now I assume I am missing something very obvious, but I don't know what it is.

Guten Abend

Vielen Dank für das einfache und schlanke Tool!
Ich habe Probleme neuere Sendungen vom SRF zu finden und dann herunterzuladen.
Beispiele wären:

channel=SRF topic=#SRFglobal duration+20m
channel=SRF topic=Deville duration+20m
channel=SRF topic=Einstein duration+45m

Kann es sein, dass Geoblocking oder so aktiviert ist und somit nicht die vollständige Datenbank heruntergeladen wird oder verwende ich das Tool einfach falsch. Für Sendungen im ZDF oder ARD funktioniert es bisher einwandfrei.
Vielen Dank für jede Hilfe.

Gruss schui95

Downloading the same movie twice with option '--oblivious" into two different locations result in sql-unique-error:

CRITICAL <class 'sqlite3.IntegrityError'>: UNIQUE constraint failed: downloaded.hash                                                                                                  
                                File "mtv_dl.py", line 1175, in <module>                                                                                                                                   
                                  main()                                                                                                                                                                   
                                File "mtv_dl.py", line 1161, in main                                                                                                                                       
                                  showlist.add_to_downloaded(item)                                                                                                                                         
                                File "mtv_dl.py", line 575, in add_to_downloaded                                                                                                                           
                                  """, show)            

The exclude-future option (respective not setting the --include-future option) hides all show of the current day, so also the shows in the past of the current day.

Steps to reproduce: (assume current time is 21:00):
mtv_dl.py list age-20h
will not show a single show.
mtv_dl.py list --include-future age-20h
will show a lot of shows in the past of the current day.

Proposal: Consider also the hour of the day, and not just the date, when filtering the shows.
Change
where.append("date(show.start) < date('now')")
to
where.append("datetime(show.start) < datetime('now')")

Observation:
If the download failes due to not writable target, the movie is marked as downloaded in the history.

Expectation:
If the anything of the download process failes, do not mark as downloaded in the history.

Looks like the TV stations removed the series/episode information from the title fields and now store it in some other fields (example https://www.zdf.de/serien/solo-fuer-weiss) . However I didn't see those other fields in the data provided by the MediathekView bundle.

So I guess we need to wait until MediathekView has that sorted out in their export...

Many users may run mtv_dl on a host with limited resources, such as typical home-server, NAS or microcomputer platforms. On these, free memory is even today often scarce. mtv_dl, when parsing the filmlist-JSON, may use up to ~2.5gb of memory with the current size of the filmlist. This frequently leads to MemoryErrors when trying to download shows.

As the items from the json are iteratively inserted into the SQLite database, it is easy to use json streaming instead of loading the whole file into memory.

I wrote a small proof of concept using ijson, which works, albeit at the expense of not having a progress bar anymore. Please see the attached memory profiler runs. As soon as i have wrestled with poetry to get it to add the dependency I’ll open a pull-request … we can discuss what to do with the progressbar there.

Hi there,
i just wanted to know if it would be possible to add an option or if it would be generally a good idea to sanitize the filesnames generated by mtv_dl.

For example when i download the current Mord auf Shetland Episodes there is a | added to the filename which makes problems when you try to transfer the file via scp or SMB.
Same problem when there is a : in the filename etc.

a search like
mtv_dl list topic="Die Sendung mit der Maus" duration=30m
does not produce any results, even though it is stated that it should work:

'=' Pattern is a regular expression case insensitive search within the field value.
Available for the fields 'description', 'start', 'duration', 'age', 'region',
'size', 'channel', 'topic', 'title', 'hash' and 'url'.

Searching without duration shows that there should be search results.

Btw, great work! I am very excited about having found this!

List count parameter -c ambiguously

 python3 mtv_dl.py list -c 5
-c is specified ambiguously 2 times

Today I encountered this error

2023-04-15T10:27:27  DEBUG    Downloading subtitles for 'Michel in der Suppenschüssel' (ARD, 'Astrid Lindgren', 2023-03-26 16:40:00, 0e2f6a7e798) from 'https://api.ardmediathek.de/player-service/subtitle/ebutt/urn:ard:subtitle:9696fedef144f1d8'. Downloading 'Michel in der Suppenschüssel' (ARD, 'Astrid Lindgren', 2023-03-26 16:40:00, 0e2f6a7e798) 
2023-04-15T10:27:30  ERROR    Download of 'Michel in der Suppenschüssel' (ARD, 'Astrid Lindgren', 2023-03-26 16:40:00, 0e2f6a7e798) failed: [Errno 22] Invalid argument: 'C:\\PUBLIC\\.tmpbwqlmlek\\urn:ard:subtitle:9696fedef144f1d8'

Colons : are not supported in Windows filenames.

Using mtv_dl==0.22.0.

Having a opening bracket in the title search results in critical error.

$mtv_dl list title="("

CRITICAL <class 'sqlite3.OperationalError'>: user-defined function raised exception
File "/home/blomi/bin/mtv_dl", line 8, in
sys.exit(main())
^^^^^^

(And thanks for this great tool!)

line 41:
-o, --oblivious Download even if the show alredy is marked as downloaded.
-->
-o, --oblivious Download even if the show already is marked as downloaded.

line 87:
age-1mm (age is older then 1 month)
-->
age-1mm (age is younger then 1 month)

Would be great if a .nfo file for each downloaded movie is created.
This should be compatible for Kodi https://kodi.wiki/view/NFO_files#Metadata_nfo

When using pytest to test #61 I get this warning:

.venv/lib/python3.9/site-packages/ijson/compat.py:48
  /home/kuehnelc/git/mtv_dl/.venv/lib/python3.9/site-packages/ijson/compat.py:48: DeprecationWarning: 
  ijson works by reading bytes, but a string reader has been given instead. This
  probably, but not necessarily, means a file-like object has been opened in text
  mode ('t') rather than binary mode ('b').
  
  An automatic conversion is being performed on the fly to continue, but on the
  other hand this creates unnecessary encoding/decoding operations that decrease
  the efficiency of the system. In the future this automatic conversion will be
  removed, and users will receive errors instead of this warning. To avoid this
  problem make sure file-like objects are opened in binary mode instead of text
  mode.
  
    warnings.warn(_str_vs_bytes_warning, DeprecationWarning)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

Is the list shown with command "list" anyhow sorted?

Proposal 1 (sufficient for me): Have a default sorting by age.
Proposal 2 (more general): Add an additional command line parameter to define the column to sort by.

And by the way: Many thanks for this cool script and even more for the extremely fast reactions.

The script does not respect the option "logfile" in a configfile.
Assumed cause: the logging_handler is defined, before the configfile is read.

    # ISO8601 logger
    if arguments['--logfile']:
        logging_handler = logging.FileHandler(Path(arguments['--logfile']).expanduser())
        logging_handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)-8s %(message)s",
                                                       "%Y-%m-%dT%H:%M:%S%z"))
    else:
        logging_handler = RichHandler(console=console)
        logging_handler.setFormatter(logging.Formatter(datefmt="%Y-%m-%dT%H:%M:%S%z "))
        logging_handler._log_render.show_path = False

    logger.addHandler(logging_handler)
    sys.excepthook = lambda _c, _e, _t: logger.critical('%s: %s\n%s', _c, _e, ''.join(traceback.format_tb(_t)))

    # config handling
    arguments = load_config(arguments)

You already added the possibility to run mtv_dl inside a Docker container. What is missing in my opinion is the published image on Docker Hub for even easier use. Right now, you have to clone the repository in order to build the image. I would prefer to simply pull the image off the Docker Hub repository and begin to use it.

I would suggest using an automatic build pipeline which can be connected to this Git repository. You can define that the "latest" image and an image containing the version tags is built whenever a new release is published. Additionally I would suggest building a "dev" image on each commit to the main-branch. This way newest features are available asap.

Thx for building this great tool 👍 ! It fits my use case much better than the Mediathek!

After downloading a video, I would like mtv_dl to trigger a local script. In my case I want the script to do 2 things: Send me an email about the new download and trigger Jellyfin to update the media library.

The script should be called with configurable arguments, similar to the path configuration, in the config file:

post-download: /some/path/myscript "{title}" "{path}"

Design proposal, in case you would like me to implement this:

• Somewhere after the download, if the post-download option is set:
  • subprocess.call the script with shell=True
  • If successful: print that script was called successfully, hide the script output.
  • On failure: print the script output to the console and continue. mtv_dl should continue it's work even if the post-download script fails.

What do you think?

tqdm automatically adjusts the size of the progressbar to match the terminal width. However if the description desc is too long (which it commonly is), it will overflow the line and cause the terminal to scroll.

Downloading 'NEO MAGAZIN ROYALE - History I - DAS steckt wirklich dahinter!' [ZDF, 'NEO MAGAZIN ROYALE', 2020-02-28 05:1
Downloading 'NEO MAGAZIN ROYALE - History I - DAS steckt wirklich dahinter!' [ZDF, 'NEO MAGAZIN ROYALE', 2020-02-28 05:1
Downloading 'NEO MAGAZIN ROYALE - History I - DAS steckt wirklich dahinter!' [ZDF, 'NEO MAGAZIN ROYALE', 2020-02-28 05:1
Downloading 'NEO MAGAZIN ROYALE - History I - DAS steckt wirklich dahinter!' [ZDF, 'NEO MAGAZIN ROYALE', 2020-02-28 05:1
Downloading 'NEO MAGAZIN ROYALE - History I - DAS steckt wirklich dahinter!' [ZDF, 'NEO MAGAZIN ROYALE', 2020-02-28 05:1
Downloading 'NEO MAGAZIN ROYALE - History I - DAS steckt wirklich dahinter!' [ZDF, 'NEO MAGAZIN ROYALE', 2020-02-28 05:1
Downloading 'NEO MAGAZIN ROYALE - History I - DAS steckt wirklich dahinter!' [ZDF, 'NEO MAGAZIN ROYALE', 2020-02-28 05:1
Downloading 'NEO MAGAZIN ROYALE - History I - DAS steckt wirklich dahinter!' [ZDF, 'NEO MAGAZIN ROYALE', 2020-02-28 05:1
Downloading 'NEO MAGAZIN ROYALE - History I - DAS steckt wirklich dahinter!' [ZDF, 'NEO MAGAZIN ROYALE', 2020-02-28 05:1
5:00+08:00, 8d45aebd]:   4%| | 32.2M/854M [02:10<53:39, 255kB/s]

It would be great if the description could be limited automatically, so that the progress output matches the terminal width.

Guten Abend

Seitdem die Progressbar mit der neuen Bibliothek (Paket rich) implementiert wurde, erscheint sporadisch diese Fehlermeldung bei einem beliebigen Download:

2020-05-17T22:05:56 CRITICAL <class 'ZeroDivisionError'>: float division by zero File "mtv_dl.py", line 1168, in <module> main() File "mtv_dl.py", line 1151, in main downloader.download(quality_preference, # type: ignore File "mtv_dl.py", line 1005, in download subtitles_xml_path = list(self._download_files(temp_path, [self.show['url_subtitles']]))[0] File "mtv_dl.py", line 814, in _download_files yield destination_file_path File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "mtv_dl.py", line 246, in progress_bar yield progress File "/home/schui/.local/lib/python3.8/site-packages/rich/progress.py", line 420, in __exit__ self.stop() File "/home/schui/.local/lib/python3.8/site-packages/rich/progress.py", line 402, in stop self.refresh() File "/home/schui/.local/lib/python3.8/site-packages/rich/progress.py", line 550, in refresh self.console.print(self._live_render) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 720, in print extend(render(renderable, render_options)) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 495, in render yield from self._render(renderable, options or self.options) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 471, in _render for render_output in iter_render: File "/home/schui/.local/lib/python3.8/site-packages/rich/live_render.py", line 35, in __console__ lines = console.render_lines(self.renderable, options, style, pad=False) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 523, in render_lines lines = list( File "/home/schui/.local/lib/python3.8/site-packages/rich/segment.py", line 110, in split_and_crop_lines for segment in segments: File "/home/schui/.local/lib/python3.8/site-packages/rich/segment.py", line 65, in <genexpr> return ( File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 495, in render yield from self._render(renderable, options or self.options) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 475, in _render yield from self.render(render_output, render_options) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 495, in render yield from self._render(renderable, options or self.options) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 471, in _render for render_output in iter_render: File "/home/schui/.local/lib/python3.8/site-packages/rich/table.py", line 324, in __console__ yield from self._render(console, options, widths) File "/home/schui/.local/lib/python3.8/site-packages/rich/table.py", line 488, in _render lines = console.render_lines( File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 523, in render_lines lines = list( File "/home/schui/.local/lib/python3.8/site-packages/rich/segment.py", line 110, in split_and_crop_lines for segment in segments: File "/home/schui/.local/lib/python3.8/site-packages/rich/segment.py", line 65, in <genexpr> return ( File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 495, in render yield from self._render(renderable, options or self.options) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 471, in _render for render_output in iter_render: File "/home/schui/.local/lib/python3.8/site-packages/rich/padding.py", line 83, in __console__ lines = console.render_lines(self.renderable, child_options, style=style) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 523, in render_lines lines = list( File "/home/schui/.local/lib/python3.8/site-packages/rich/segment.py", line 110, in split_and_crop_lines for segment in segments: File "/home/schui/.local/lib/python3.8/site-packages/rich/segment.py", line 65, in <genexpr> return ( File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 495, in render yield from self._render(renderable, options or self.options) File "/home/schui/.local/lib/python3.8/site-packages/rich/console.py", line 471, in _render for render_output in iter_render: File "/home/schui/.local/lib/python3.8/site-packages/rich/bar.py", line 65, in __console__ complete_halves = int(width * 2 * completed / self.total)

Aktuelles Beispielt wäre: 2020-05-17T22:39:26 INFO Saved show 'Extra 3 vom 13.05.2020 mit Christian Ehring' (ARD, 'extra 3', 2020-05-13 21:10:00, 39190c75c9c)
Danke für den Fix.
Gruss schui95

The script should not start downloading of the database when the command is history.

Near line 1164:
logger.info('Marked %s from %s as downloaded.', downloader.label)
-->
logger.info('Marked %s as downloaded.', downloader.label)

Hi there,
ive just seen that some months ago you switch to sqlite for the history db.
Before i update (i think i use the version from end of last year) i would like to know if my old history will be migrated?
Else i would have to edit my shows.txt to only download new shows after the current date.

tl;dr: my problem was that I had 2 .History files at that time, one in cwd and one in {dir} and that mtv_dl deleted the files from the database in cwd and not in {dir}, because I used mtv_dl history --remove and not mtv_dl history --remove --config=./.mtv_dl.yml.

I have just stumbled upon this wonderful piece of software and I am testing it right now.
While playing with settings and filters, I had do re-download a tv show, which wasn't possible.

[jan@two-arch bin]$ ./mtv_dl download --config=./.mtv_dl.yml --sets=./mtv_tatort.txt
2021-10-21T09:07:46  DEBUG    Opening Filmliste database PosixPath('/home/jan/Videos/Tatort/.Filmliste.0.18.1.sqlite').                                                      
                     DEBUG    Opening History database PosixPath('/home/jan/Videos/Tatort/.History.sqlite').                                                                 
                     DEBUG    Database age is 0:17:00.                                                                                                                       
                     DEBUG    Applying filter: channel=ard, topic=tatort, title!=audiodeskription, age-1d, duration+80m (limit: None)                                        
2021-10-21T09:07:47  DEBUG    Skipping 'Tatort: Altes Eisen' (ARD, 'Tatort: Altes Eisen', 2021-10-20 20:00:00, c80523597cd) (already loaded on 2021-10-21 06:52:14)          
                     DEBUG    Skipping 'Absturz' (ARD, 'Tatort', 2021-10-20 20:10:00, 642cb991142) (already loaded on 2021-10-21 06:53:56)                                   

I then removed both shows from the history:

[jan@two-arch bin]$ ./mtv_dl history --remove=642cb991142
2021-10-21T09:08:58  INFO     Removed 642cb991142 from history.                                                                                                              
[jan@two-arch bin]$ ./mtv_dl history --remove=c80523597cd
2021-10-21T09:09:09  INFO     Removed c80523597cd from history.                                                                                                              

but the result was the same:

[jan@two-arch bin]$ ./mtv_dl download --config=./.mtv_dl.yml --sets=./mtv_tatort.txt
2021-10-21T09:09:14  DEBUG    Opening Filmliste database PosixPath('/home/jan/Videos/Tatort/.Filmliste.0.18.1.sqlite').                                                      
                     DEBUG    Opening History database PosixPath('/home/jan/Videos/Tatort/.History.sqlite').                                                                 
                     DEBUG    Database age is 0:19:00.                                                                                                                       
                     DEBUG    Applying filter: channel=ard, topic=tatort, title!=audiodeskription, age-1d, duration+80m (limit: None)                                        
2021-10-21T09:09:15  DEBUG    Skipping 'Tatort: Altes Eisen' (ARD, 'Tatort: Altes Eisen', 2021-10-20 20:00:00, c80523597cd) (already loaded on 2021-10-21 06:52:14)          
                     DEBUG    Skipping 'Absturz' (ARD, 'Tatort', 2021-10-20 20:10:00, 642cb991142) (already loaded on 2021-10-21 06:53:56)                                   

Then I tried to reset the history:

[jan@two-arch bin]$ ./mtv_dl history --reset

but still the same:

[jan@two-arch bin]$ ./mtv_dl download --config=./.mtv_dl.yml --sets=./mtv_tatort.txt
2021-10-21T09:09:52  DEBUG    Opening Filmliste database PosixPath('/home/jan/Videos/Tatort/.Filmliste.0.18.1.sqlite').                                                      
                     DEBUG    Opening History database PosixPath('/home/jan/Videos/Tatort/.History.sqlite').                                                                 
                     DEBUG    Database age is 0:19:00.                                                                                                                       
                     DEBUG    Applying filter: channel=ard, topic=tatort, title!=audiodeskription, age-1d, duration+80m (limit: None)                                        
                     DEBUG    Skipping 'Tatort: Altes Eisen' (ARD, 'Tatort: Altes Eisen', 2021-10-20 20:00:00, c80523597cd) (already loaded on 2021-10-21 06:52:14)          
                     DEBUG    Skipping 'Absturz' (ARD, 'Tatort', 2021-10-20 20:10:00, 642cb991142) (already loaded on 2021-10-21 06:53:56)                                   

The database itself when used manually is working as expected:

[jan@two-arch bin]$ sqlite3 .History.sqlite 
SQLite version 3.36.0 2021-06-18 18:36:39
...
sqlite> .headers on
sqlite> select * from downloaded;
hash|channel|description|region|size|title|topic|website|start|duration|downloaded
c80523597cd8b5e45644c85e64f43f99aecd8e74|ARD|Bei ihren Ermittlungen ...|DE|1135|Tatort: Altes Eisen|Tatort: Altes Eisen|https://www...337|2021-10-21 07:17:28
642cb99114281235c64bf72ef3105097f37d825a|ARD|Eigentlich ...|DE|1262|Absturz|Tatort|https://www...MGY|2021-10-20 20:10:00|5312|2021-10-21 07:19:10
sqlite> select hash from downloaded where hash like 'c80523597cd%';
hash
c80523597cd8b5e45644c85e64f43f99aecd8e74
sqlite> delete from downloaded where hash='c80523597cd8b5e45644c85e64f43f99aecd8e74';
sqlite> select * from downloaded;
hash|channel|description|region|size|title|topic|website|start|duration|downloaded
642cb99114281235c64bf72ef3105097f37d825a|ARD|Eigentlich ...|DE|1262|Absturz|Tatort|https://www...MGY|2021-10-20 20:10:00|5312|2021-10-21 07:19:10

Having a typo in the filter parametername (e.g. uppercase instead of lower case)
python3 mtv_dl.py list Title='extra 3'
leads to a confusing error message:
ERROR Invalid operator '=' for 'Title'.
But not the operator = is invalid, the whole parametername is invalid.

There does not seem to be an option to download subtitles, although doing a JSON dump of a certain program hash does reveal a working URL for the subtitles.

Hi Frank,

since last few days I get an overflowError:

$ mtv_dl -v list channel=ZDF topic=Heute
2022-02-18T18:23:24  DEBUG    Opening Filmliste database
                              PosixPath('/smb/video/MediathekView/.Filmliste.0.18.1.sqlite').
                     DEBUG    Opening History database PosixPath('/smb/video/MediathekView/.History.sqlite').
                     DEBUG    Initializing Filmliste database in
                              PosixPath('/smb/video/MediathekView/.Filmliste.0.18.1.sqlite').
                     DEBUG    Opening database from 'https://liste.mediathekview.de/Filmliste-akt.xz'.
Downloading database ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:00
2022-02-18T18:23:29  DEBUG    Loading database items.
Reading database items ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.1% 0:00:48
2022-02-18T18:25:21  CRITICAL <class 'OverflowError'>: timestamp out of range for platform time_t
                                File "/home/pi/.local/bin/mtv_dl", line 10, in <module>
                                  sys.exit(main())
                                File "/home/pi/.local/lib/python3.7/site-packages/mtv_dl.py", line 1239, in main
                                  history=cw_dir / HISTORY_DATABASE_FILE)
                                File "/home/pi/.local/lib/python3.7/site-packages/mtv_dl.py", line 434, in __init__
                                  self.initialize_filmliste()
                                File "/home/pi/.local/lib/python3.7/site-packages/mtv_dl.py", line 387, in
                              initialize_filmliste
                                  """, self._get_shows())
                                File "/home/pi/.local/lib/python3.7/site-packages/mtv_dl.py", line 540, in _get_shows
                                  start = datetime.fromtimestamp(int(show['start']), tz=utc_zone).replace(tzinfo=None)

The code is running on Raspian Linux 10 32bit, mtv_dl installed via pip3.

Could you please have a look?
Thx
Maik

start = datetime.fromtimestamp(int(show['start']), tz=utc_zone).replace(tzinfo=None) fails with an OSError: [Errno 22] Invalid argument for negative start dates.

I added some logging

                                try:
                                    start = datetime.fromtimestamp(int(show['start']), tz=utc_zone).replace(tzinfo=None)
                                except OSError:
                                    logger.error("Error parsing show %s: start=%s", title, show['start'])
                                    continue

which showed lots of shows with negative dates

                    ERROR    Error parsing show Europlast: start=-127635900
                     ERROR    Error parsing show Bundespostminister Stücklen eröffnet automatische Vermittlungsstelle für Auslands-Ferngespräche: start=-127473420
                     ERROR    Error parsing show Nerz und Brillanten unterm Weihnachtsbaum: start=-127549260
                     ERROR    Error parsing show Museen ohne Mittel? Von 35 Millionen Lottogewinn gingen 5 Millionen an staatliche Museen.: start=-127645680

The question probably is: Why are there negative values in the first place?

EDIT: Because the Filmliste is originally created by Java, it probably means

A negative number indicates the number of milliseconds before January 1, 1970, 00:00:00 GMT.
https://docs.oracle.com/javase/7/docs/api/java/sql/Date.html

Being honest i just wanted to say Thanks! because your tool saved me a lot of time downloading shows for my Kids.

Hi,

first of all thanks for the work put into this. I recently moved mtv to my pi4 and it works really nice.;)

Not sure if this is inteded behaviour. But taking the 'famous' 'Tatort' as an example:

The show is being aired on multiple channels. At different times they show different (or same) episodes.
If i set a filter title!='Audiodeskription' title!='klare Sprache' title!='Trailer' topic='Tatort' duration+70m i see a lot of duplicates with different hashes because of the channel-hash (even the start-time is identical).

Is this inteded? Are there shows with identical topic,title,size and time but different content?
Or could the channel-hash be removed to prevent redownloading?
Or is there another possibility?

regards,
strowi

Hi!
I just installed mtv_dl on an Alpine Linux VM.
I was able to do a couple of list commands, and then all of a sudden, this error message popped up:

2020-11-05T10:34:15  CRITICAL <class 'sqlite3.OperationalError'>: unable to open database file

                                File "/home/matti/.local/bin/mtv_dl", line 8, in <module>

                                  sys.exit(main())
                                File "/home/matti/.local/lib/python3.8/site-packages/mtv_dl.py", line 1202, in main
                                  showlist = Database(filmliste=cw_dir / FILMLISTE_DATABASE_FILE,
                                File "/home/matti/.local/lib/python3.8/site-packages/mtv_dl.py", line 414, in __init__
                                  self.connection = sqlite3.connect(filmliste_path.absolute().as_posix(),

Rebooting didn't change anything, also there is enough space on the drive.

Hi,
please add two naming options for downloaded files:

1. Option for Year, Month and Day in Filename, eg:

--- mtv_dl.py   2022-09-20 18:18:03.000000000 +0200
+++ mtv_dl.py   2022-09-20 18:39:00.826512389 +0200
@@ -900,6 +900,10 @@
                                                          ext=file_extension,
                                                          date=self.show['start'].date().isoformat(),
                                                          time=self.show['start'].strftime('%H-%M'),
+                                                         year=self.show['start'].date().year,
+                                                         monthwozero=self.show['start'].date().month,
+                                                         month=self.show['start'].strftime('%m'),
+                                                         day=self.show['start'].strftime('%d'),
                                                          **escaped_show_details))
 
         destination_file_path.parent.mkdir(parents=True, exist_ok=True)

2. Add the possibility to replace spaces/blanks with underscores.

Background: For Jellyfin/Emby I need to have a special Naming of the Files with Season and Episode to get recognized correctly. Currently I am doing this with some manual effort and was not able to adapt the python code in a way that it is working.

E.g. the output file shall look like: Die_Sendung_mit_der_Maus_S2022-E9-25_Die_Sendung_vom_25.09.2022.mp4

Thanks and best regards
Ben

From the command line, no problem:

/usr/local/bin/mtv_dl -v download topic='terra' channel='zdf' age-30d duration+30m --high --dir=/mnt/NAS/dvds/recordings
2019-10-15T21:04:47+0200 DEBUG    Using cache from PosixPath('/mnt/NAS/dvds/recordings/.Filmliste-akt.xz.cache').
2019-10-15T21:04:48+0200 DEBUG    Database age is 0:41:00.
2019-10-15T21:04:48+0200 DEBUG    Applying filter: topic=terra, channel=zdf, age-30d, duration+30m
2019-10-15T21:04:49+0200 DEBUG    Skipping 'Im Bann der Astrologie - Faszination Universum mit Harald Lesch' [ZDF, 'Terra X', 2019-09-29 19:30:00+02:00, e7da8281] (already loaded on 2019-10-01 09:11:00)
2019-10-15T21:04:49+0200 DEBUG    Downloading 'Unser kosmisches Schicksal - Faszination Universum mit Harald Lesch' [ZDF, 'Terra X', 2019-10-13 19:30:00+02:00, d944a6b8] from 'https://rodlzdf-a.akamaihd.net/none/zdf/19/10/191013_fu_schicksal_tex/3/191013_fu_schicksal_tex_3328k_p36v14.mp4'.

However, if I run the same command from within a sudo Python3.7 script using subprocess, I get this:

sudo python3.7 /mnt/NAS/dvds/recordings/pvr.py
2019-10-15 21:03:58.100566
running: /usr/local/bin/mtv_dl -v download topic='terra' channel='zdf' age-30d duration+30m --high --dir=/mnt/NAS/dvds/recordings
out: ''
err: '2019-10-15T21:03:58+0200 DEBUG    Using cache from PosixPath('/mnt/NAS/dvds/recordings/.Filmliste-akt.xz.cache').
2019-10-15T21:03:59+0200 DEBUG    Database age is 0:40:00.
2019-10-15T21:03:59+0200 DEBUG    Applying filter: topic='terra', channel='zdf', age-30d, duration+30m
'
exit: 0
2019-10-15 21:04:00.606802

This is the Python code I use to run an external command

def run(cmd):
    print(datetime.datetime.now())
    print("running:", " ".join(cmd))
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = proc.communicate()
    proc.stdout.close()
    proc.stderr.close()
    print("out: '{}'".format(stdout.decode("utf8")))
    print("err: '{}'".format(stderr.decode("utf8")))
    print("exit: {}".format(proc.returncode))
    print(datetime.datetime.now())
    print("-------------------")

run(["/usr/local/bin/mtv_dl", "-v", "download",  "topic='terra'",  "channel='zdf'",
                     "age-30d", "duration+30m", "--high", "--dir=/mnt/NAS/dvds/recordings"])

Do you have any idea what could cause this?

Hi,

i had not run the tool for a while, and now fetched the ˋlatestˋ container image.

After start the program crashes with this error:

2023-06-10T12:32:29+0000 CRITICAL <class 'sqlite3.OperationalError'>: table main.show has 16 columns but 18 values were supplied
  File "<string>", line 1, in <module>
  File "/mtv_dl.py", line 1462, in main
    showlist.initialize_if_old(refresh_after=int(arguments["--refresh-after"]))
  File "/mtv_dl.py", line 618, in initialize_if_old
    self.initialize_filmliste()
  File "/mtv_dl.py", line 380, in initialize_filmliste
    cursor.executemany(

I also tried the ˋmasterˋ image version. Same error was reported.
It looks like the new columns are missing:

# sqlite3 .Filmliste.0.0.0.sqlite 
SQLite version 3.34.1 2021-01-20 14:10:07
Enter ".help" for usage hints.
sqlite> .schema main.show
CREATE TABLE show (
                    hash TEXT,
                    channel TEXT,
                    description TEXT,
                    region TEXT,
                    size INTEGER,
                    title TEXT,
                    topic TEXT,
                    website TEXT,
                    new BOOLEAN,
                    url_http TEXT,
                    url_http_hd TEXT,
                    url_http_small TEXT,
                    url_subtitles TEXT,
                    start TIMESTAMP,
                    duration TIMEDELTA,
                    age TIMEDELTA
                );

First off: An amazing tool you have build there, @fnep ! I'm using it daily (well, my cron scheduler is 😉).

I've noticed that some TV series e.g. from ZDF are now using (S01E01) like episode markers in the title field. So we now have some data that we could use to guess TV series and slightly change the behavior of mtv_dl. It would be awesome if mtv_dl would use that information so that the Series appears nicely in the media centers (Emby, Kodi, Jellyfin, ...).

What I had in mind was:

• add a switch --series for the download command to enable special treatment of TV series.
  • If that flag is set, do a few things differently, details below
  • In the beginning I would propose to let the user enable the series mode, maybe later automatically trigger it based on some heuristics.
• Extract the series/episode from the title, using a regex \(S(\d+)/E(\d+)\) for the first case:
  • (S01/E01), sample query: mtv_dl -r23 dump topic="Nachtschicht" title=S01
  • (123), sample query: mtv_dl -r23 dump topic="Leopard, Seebär", here always assume season=1
• in the NFO file,
  • use<episodedetails> instead of <movie>
  • add the additional details with<season>XXX</season><episode>XXX</episode>
• optional: change default name pattern (or let the user do that):
  • write all files into the same folder called <topic>
  • use naming schema for file names: <topic> - SxxExx - <title>.<extension>
• optional: support episode and season also in the filters
• optional: create a root NFO file containing some general series information (not sure where to get the details from)

I didn't check if that series metadata would also be available somewhere in the database so that we don't need to guess it from the title...

If you're interested in such a feature, I could take a look in a couple of weeks and propose a pull request...

The download runs smooth and the downloads are shown with pretty title and plot in Kodi with the .nfo file! Very cool!

Unfortunately the skins in Kodi don't use the parameter in the .nfo file but instead the file modification time to show and sort the files by date.
Therefore following improvement: Have an additional parameter --set-file-mod-time. If this parameter is set, after downloading the file modification time is set to the aired time.

See PR #32 below.

I installed mtv_dl in a virtual_env
It was working fine on another host with ubuntu but here it doesnt work.
I do see:
(mtv) iUser@TP-W500-iUser:~/winshares/media/video/serien/Babylon Berlin/Season 3$ mtv_dl history
2023-10-03T00:10:37 CRITICAL <class 'sqlite3.OperationalError'>: no such table: main.show
File "/home/iUser/mtv/bin/mtv_dl", line 8, in
sys.exit(main())
File "/home/iUser/mtv/lib/python3.10/site-packages/mtv_dl.py", line 1461, in main
showlist = Database(filmliste=cw_dir / FILMLISTE_DATABASE_FILE, history=cw_dir / HISTORY_DATABASE_FILE)
File "/home/iUser/mtv/lib/python3.10/site-packages/mtv_dl.py", line 471, in init
self.initialize_filmliste()
File "/home/iUser/mtv/lib/python3.10/site-packages/mtv_dl.py", line 377, in initialize_filmliste
cursor.execute("DELETE FROM main.show")

Hi there,
i have just stumbled upon the problem that my mtv_dl install stopped downloading since the 11. June.
After digging a bit around i found auf that several of the URL in FILMLISTE_URLS seem to be down (verteiler2 to verteiler6). I tried it with my homeconnection and a VPS on both i get an unknown host message when i try to ping them.

Hi !

thank you very much for this great peace of software. Did you ever think about to make a docker container for this ?!

Hi,

i just ran into this issue on a clean pi4 with v0.23.0:

~> mtv_dl list -r title='Letzte Spur'
Downloading database 
....
━━━━━━━━━━━━━━━━━━━━━━━╺ 99.6% 0:00:02Reading database items ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:00
2023-06-02T21:15:05  INFO     Initializing History database in PosixPath('/home/osmc/mtv/.Filmliste.0.23.0.sqlite').
2023-06-02T21:15:05  CRITICAL <class 'sqlite3.OperationalError'>: near "(": syntax error
                                File "/usr/local/bin/mtv_dl", line 8, in <module>
                                  sys.exit(main())
                                File "/usr/local/lib/python3.9/dist-packages/mtv_dl.py", line 1461, in main
                                  showlist = Database(filmliste=cw_dir / FILMLISTE_DATABASE_FILE, history=cw_dir / HISTORY_DATABASE_FILE)
                                File "/usr/local/lib/python3.9/dist-packages/mtv_dl.py", line 473, in __init__
                                  self.initialize_history()
                                File "/usr/local/lib/python3.9/dist-packages/mtv_dl.py", line 420, in initialize_history
                                  cursor.execute(

Deleting the sqlite-files, downgrading to v0.22.0 and running the command again works. And after that i can upgrade to v0.23.0 which shows an additional info

2023-06-02T21:40:10 INFO Upgrading history database schema, adding columns for season and episode

and then it succeeds.

Looks to me as if sth. changed in the initialisation of the db?

So I just started moving a few services from different raspberryPis onto a single 10W TDP NUC, to have lower power usage, better peak performance, and a single point of failure. Yeah me.

I plugged the external drive with the library and history file etc. into the new system, ran the usual download script and mtv_dl wanted to redownload everything. The hashes differed everywhere, which shouldn't be.

  cd6670d4e3c │ ARD      │ Extra 3 vom 25.08.2022 im Ersten                        │ extra 3                                                │ 429  │ 2022-08-25T22:50:00+02:00 │ 44m 3s     │ None │ DE          │ 2022-08-26T01:37:44+02:00  
  90b260f6adc │ ARD      │ Extra 3 vom 25.08.2022 im Ersten                        │ extra 3                                                │ 429  │ 2022-08-25T22:50:00+02:00 │ 44m 3s     │ None │ DE          │ 2022-09-12T16:58:53+02:00  

I apparently had different locale and TZ settings. After setting the locale from C to German UTF-8 and the TZ from London to Berlin, and rebooting, (and probably redownloading the mediathek database file) mtv_dl had the correct hashes and picked up the history.

I don't know which part (I feel timezone, even though at first glance the code looks agnostic to it and the formatted start times are the same as well), but somehow the hash value is not completely agnostic of some environment settings.

I guess to replicate:

• Check hash /usr/local/bin/mtv_dl dump title="FIFA und die WM 2022 in Katar"
• Change locale and TZ, reboot
• Check hash again (database needs to be redownloaded, probably), notice difference

The download of subtitles failed. In most cases it's fine, but for the following movie the subtitles file .srt is created but empty.

python3 mtv_dl.py download --oblivious --target="{start} {title}{ext}" hash=2f2bad3ed7c
Downloading 'Der erste April (26)' (ARD, 'Meister Eder und sein Pumuckl', 2020-07-05 04:40:00, 2f2bad3ed7c) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:00 
2020-07-14T20:07:55  INFO     Saved show 'Der erste April (26)' (ARD, 'Meister Eder und sein Pumuckl', 2020-07-05 04:40:00, 2f2bad3ed7c) to PosixPath('2020-07-05 04_40_00 Der erste April (26).mp4').                                                                                                                                                                  
Downloading 'Der erste April (26)' (ARD, 'Meister Eder und sein Pumuckl', 2020-07-05 04:40:00, 2f2bad3ed7c) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:00 
2020-07-14T20:07:56  INFO     Saved subtitles 'Der erste April (26)' (ARD, 'Meister Eder und sein Pumuckl', 2020-07-05 04:40:00, 2f2bad3ed7c) to PosixPath('2020-07-05 04_40_00 Der erste April (26).srt').    

The file 2020-07-05 04_40_00 Der erste April (26).srt is created but empty.

The original subtitles file in the mediathek has content:
https://classic.ardmediathek.de/subtitle/411383

Hello once again,
I now have the problem that the file is not moved to the new location on the mounted NFS drive with

[Errno 18] Invalid cross-device link
This seems to be due to the usage of Path.rename in _move_to_user_target (line 719) which apparently does not work across different filesystems.
Is it possible to use some other method for moving the file (like shutil.move) in case Path.rename fails?
Being a non-software developer, I can only guess what needs to be done and my solution would not be as robust as anything you could come up with.
Thanks for making this great tool, I really hope I can use it for my use cases :)

mtv_dl.py download hash=9bafd1df3bb
The movie has a invalid subtitle url "/static/avportal/untertitel_mediathek_preview/24429544.xml"
This causes the script to abort with a critical error.

Unfortunately this prevents downloading of additional movies of the same query.

Proposal: Change the logic, that in case of a failure of download, the other movies in the same query are still downloaded.