thewicklowwolf / lidatube

Find and download missing Lidarr albums via yt-dlp.

LidaTube is a tool for finding and fetching missing Lidarr albums via yt-dlp.

Run using docker-compose

services:
  lidatube:
    image: thewicklowwolf/lidatube:latest
    container_name: lidatube
    volumes:
      - /path/to/config:/lidatube/config
      - /data/media/lidatube:/lidatube/downloads
      - /etc/localtime:/etc/localtime:ro
    ports:
      - 5000:5000
    restart: unless-stopped

Configuration via environment variables

Certain values can be set via environment variables:

• lidarr_address: The URL for Lidarr. Defaults to http://192.168.1.2:8686.
• lidarr_api_key: The API key for Lidarr. Defaults to ``.
• lidarr_api_timeout: Timeout duration for Lidarr API calls. Defaults to 120.
• thread_limit: Max number of threads to use. Defaults to 1.
• sleep_interval: Interval to sleep. Defaults to 0.
• fallback_to_top_result: Whether to use the top result if no match is found. Defaults to False.
• library_scan_on_completion: Whether to scan Lidarr Library on completion. Defaults to True.
• sync_schedule: Schedule times to run (comma seperated values in 24hr). Defaults to ``
• minimum_match_ratio: Minimum percentage for a match. Defaults to 90
• secondary_search: Method for secondary search (YTS or YTDLP). Defaults to YTS.
• preferred_codec: Preferred codec (mp3). Defaults to mp3.
• attempt_lidarr_import: Attempt to import each song directly into Lidarr. Defaults to False.

Cookies (optional)

To utilize a cookies file with yt-dlp, follow these steps:

• Generate Cookies File: Open your web browser and use a suitable extension (e.g. cookies.txt for Firefox) to extract cookies for a user on YT.

• Save Cookies File: Save the obtained cookies into a file named cookies.txt and put it into the config folder.

---

https://hub.docker.com/r/thewicklowwolf/lidatube

lidatube's Issues

Sort Order

I believe the following code:

 params = {"apikey": self.lidarrApiKey, "pageSize": self.lidarrMaxTags, "sortKey": "artists.sortname", "sortDir": "asc"}

Should be as follows:

 params = {"apikey": self.lidarrApiKey, "pageSize": self.lidarrMaxTags, "sortKey": "artists.sortname", "sortDirection": "ascending"}

If I am reading the API documentation correctly...
https://lidarr.audio/docs/api/#/Missing/get_api_v1_wanted_missing

A couple RFEs for new options

Posted on your reddit thread where you announced these as well, but posting here as well.

2 improvements I would like to see -

1. an option to search the lidarr library for a certain artist/album, instead of just "get missing list"

2. On the download side, be able to just select certain tracks to download. I have some albums that have 1 or 2 missing tracks and using lidatube to backfill just those tracks would be great :)

youtube login support

i would like to be able to login to my youtube account. this would like me download from youtube premium for the music.
goes with #4

Search filters

Hi first of all congratulations for this project ,really very useful ,it would be even more interesting if in the future there would be the possibility to set filters ,for example I noticed that sometimes live tracks are downloaded.

Add Support for Multiselect via Holding Shift

Support for holding shift and clicking two checkboxes to check/uncheck large list sections at once. This can help when resuming an interrupted download session.

[FEATURE REQUEST] Make the downloads directory configurable

Make the downloads directory an option inside the configuration file, it can be useful especially when not running inside docker.

Error Downloading - not supported between instances

Running docker on Unraid

Settings connected between Lidarr instance can successfully search for matches, but get met with the below error for every match song

Error Downloading: '<' not supported between instances of 'int' and 'str'

Lidarr import tuning parameter

2024-07-12 19:32:00.1|Debug|CloseAlbumMatchSpecification|Album match is not close enough: 0.42105263157894735 vs 0.2 [album, year, country]. Skipping [/lidatube/downloads/AHEE/Crystal Beam (2022)]
2024-07-12 19:32:00.1|Debug|ImportDecisionMaker|Album rejected for the following reasons: [Permanent] Album match is not close enough: 57.9% vs 80% [album, year, country]

I verified a couple of import with the right 3 parameters album, year, country, but the ratio match keep beeing low for lidarr import.

What do i need to increase the success import on album that are really the good ones ?

[Feature Request] - Support for "Other versions", Run on schedule, etc

Prologue

Howdy! First of all, I'm extremely happy to stumble onto the project, and wanted to thank the owner and any other contributors for setting this up. Had this idea recently to use YouTube music as a fallback for missing stuff on lidarr and was super relieved to find that it already existed in a working state (previously was doing this manually via MeTube). Also happy to see the project is primarily Python, so I can actually help contribute since that's my bread and butter.

Feature Request

So as we all know, music data across platforms so rarely agrees with one another, in this case if you look at an artist like P.O.S. the album "Chill, Dummy" is separate from the album "Chill, Dummy (Instrumental)" where lidarr is concerned (such is the same for many artists I'm sure). Ironically on Lidarr a different album for the same artist represents the instrumental version as a different release within the same album, rather than a totally separate album, but I digress.

In the case of the normal and instrumental versions of "Chill, Dummy", I was unable to download the instrumental through LidaTube since it presents on YouTube Music as a "Other version" and despite the name searched via LidaTube being the instrumental version it just corrects to the non-instrumental. So I had to manually link that instrumental album in MeTube to grab it. You can see the original album here and the instrumental "other version" here.

I think we should add the capability to pull in data for "Other versions" and look for closer matches in "Other versions" when searching for wholesale albums. I would absolutely be willing to commit some time to a PR for such a feature and will reference this issue in the PR if I get around to it soon! 😄

Additional Features

I see that some features previously requested have been marked "out of scope" for the project, which is understandable and I don't wish to step on anyone's toes here, but one feature in particular caught my eye, the ability to search individual songs if someone has part of an album but missing a song or two. I definitely understand not letting the project scope expand out of control and keeping things grounded and straightforward without too much code debt, but as I said I'm very experienced with Python (and JS) and would be willing to commit additional time to getting this feature implemented as well. I think it could be an excellent addition, but I just wanted to get clarification on that, if I were to work on such a feature and submit a competent PR to implement it, would the PR be reviewed and accepted if adequate, or would it still be out of scope? No pressure at all if the issue just isn't desired, just wanted to ask first before I commit the time to implementing that 😁

Also, I'm not sure if this has already been implemented, but I think adding some kind of daemon mode or lidarr hook that automatically grabs from YouTube when attempts from lidarr have been exhausted would be fantastic, or at least the ability to toggle the behavior of running a LidaTube search on a routine schedule. Personally I plan to use LidaTube to grab anything missing in my collection, then enable upgrades in Lidarr that way the YouTube version is always the fallback solution for music but better versions i.e. FLAC can be ingested if and when they become available.

Unraid Template

Lastly, I think we should try to get an Unraid template listed in Unraid community apps. Would make a wonderful addition, since as of now there are no community apps that support this level of automation for grabbing music from YouTube as a fallback solution for lidarr that implement music metadata as competently as LidaTube does (well done on that!). As far as I can tell MeTube is the closest, but it has many drawbacks, mainly that you have to manually link every album you want, and the metadata is hit or miss.

Write Metadata to Tracks Using Beets (or similar)

Upon running on a large media collection, the songs that were downloaded had the incorrect album art embedded in the tracks (they were the YouTube thumbnails instead). The album art in my music players would be correct, but upon pressing on a song, the album art displayed would be the YouTube thumbnail.

This is because (to my understanding) Lidarr does not write album art images to music files, or something similar. Upon importing all new songs to Lidarr (and scanning and replacing all metadata within Lidarr), the incorrect album art images persisted within the track files.

Regardless of the root cause, I had to run my entire library through MusicBrainz Picard in order to write the correct album art images to each track downloaded from LidaTube - which took many, many hours. This process could be potentially automated by having LidaTube run each downloaded file into Beets (or similar) upon completing a download.

More info on what exactly this does...

More information on what exactly this does and how it integrates with Lidarr.

Any way to set preferred codec?

I love this!! I have been using my own script to grab from yt_dlp and it is nice, but this GUI is super slick.

The one thing that I wished I could do is set the preferred code to FLAC instead of MP3. I see it in the python code. Is there a chance you could expose that via the settings or by an env var?

Thanks!!

Not downloading anything.

It looks like its trying to download from youtube but nothing in the download folder and it goes really fast maybe a second or two then says download complete.

services:
lidatube:
image: thewicklowwolf/lidatube:latest
container_name: lidatube
environment:
- PUID=1000
- PGID=1000
- TZ=America/Chicago
volumes:
- /opt/lidarr2/config:/lidatube/config:rw
- /mnt/wd4tb/downloads/complete/lidatube:/lidatube/downloads:rw
- /etc/localtime:/etc/localtime:ro
ports:
- 5055:5000
restart: unless-stopped

I was able to pull this info from lidatube logs from portainer

Not matching album for: Frank Zappa - Whisky a Go Go, 1968
Links found for all tracks of: Frank Zappa - Whisky a Go Go, 1968
Searching for Album: U2 - Atomic City (Tiësto remix)
Searching for whole Album: U2 - Atomic City (Tiësto remix)
Not matching album for: U2 - Atomic City (Tiësto remix)
Links found for all tracks of: U2 - Atomic City (Tiësto remix)
Searching for Album: Within Temptation - Worlds Collide Tour (Live in Amsterdam)
Searching for whole Album: Within Temptation - Worlds Collide Tour (Live in Amsterdam)
Not matching album for: Within Temptation - Worlds Collide Tour (Live in Amsterdam)
Links found for all tracks of: Within Temptation - Worlds Collide Tour (Live in Amsterdam)
Searching for Album: Dark Funeral - Shadows Over Transylvania
Searching for whole Album: Dark Funeral - Shadows Over Transylvania
Not matching album for: Dark Funeral - Shadows Over Transylvania
Links found for all tracks of: Dark Funeral - Shadows Over Transylvania
Searching for Album: Bob Dylan - Smoking Down the Track- I and I - 1984-1999
Searching for whole Album: Bob Dylan - Smoking Down the Track- I and I - 1984-1999
Not matching album for: Bob Dylan - Smoking Down the Track- I and I - 1984-1999
Links found for all tracks of: Bob Dylan - Smoking Down the Track- I and I - 1984-1999
Searching for Album: Orange Goblin - The Fire At The Centre Of The Earth Is Mine
Searching for whole Album: Orange Goblin - The Fire At The Centre Of The Earth Is Mine
Not matching album for: Orange Goblin - The Fire At The Centre Of The Earth Is Mine
Links found for all tracks of: Orange Goblin - The Fire At The Centre Of The Earth Is Mine
Searching for Album: Boris - ANOTHER NOISE
Searching for whole Album: Boris - ANOTHER NOISE
Not matching album for: Boris - ANOTHER NOISE
Links found for all tracks of: Boris - ANOTHER NOISE
Searching for Album: The Japanese House - -)
Searching for whole Album: The Japanese House - -)
Not matching album for: The Japanese House - -)
Links found for all tracks of: The Japanese House - -)
Searching for Album: David Gilmour - Between Two Points
Searching for whole Album: David Gilmour - Between Two Points
Not matching album for: David Gilmour - Between Two Points
Links found for all tracks of: David Gilmour - Between Two Points
Searching for Album: Melvins - Twins of Evil
Searching for whole Album: Melvins - Twins of Evil
Not matching album for: Melvins - Twins of Evil
Links found for all tracks of: Melvins - Twins of Evil
Searching for Album: Fleshgod Apocalypse - Bloodclock
Searching for whole Album: Fleshgod Apocalypse - Bloodclock
Not matching album for: Fleshgod Apocalypse - Bloodclock
Links found for all tracks of: Fleshgod Apocalypse - Bloodclock
Searching for Album: Sam Feldt - Mi Amor
Searching for whole Album: Sam Feldt - Mi Amor
Not matching album for: Sam Feldt - Mi Amor
Links found for all tracks of: Sam Feldt - Mi Amor
Searching for Album: The Decemberists - As It Ever Was, So It Will Be Again
Searching for whole Album: The Decemberists - As It Ever Was, So It Will Be Again
Not matching album for: The Decemberists - As It Ever Was, So It Will Be Again
Links found for all tracks of: The Decemberists - As It Ever Was, So It Will Be Again
Searching for Album: R.E.M. - R.E.M.’s Top Forty Playlist (according to Berry, Buck, Mills and Stipe)
Searching for whole Album: R.E.M. - R.E.M.’s Top Forty Playlist (according to Berry, Buck, Mills and Stipe)
Not matching album for: R.E.M. - R.E.M.’s Top Forty Playlist (according to Berry, Buck, Mills and Stipe)
Links found for all tracks of: R.E.M. - R.E.M.’s Top Forty Playlist (according to Berry, Buck, Mills and Stipe)
Searching for Album: The Saints - Break Away
Searching for whole Album: The Saints - Break Away
Not matching album for: The Saints - Break Away
Links found for all tracks of: The Saints - Break Away
Searching for Album: Van Halen - Top of the World (live at the West End Marketplace, Dallas, Tx, 12+4+1991)
Searching for whole Album: Van Halen - Top of the World (live at the West End Marketplace, Dallas, Tx, 12+4+1991)
Not matching album for: Van Halen - Top of the World (live at the West End Marketplace, Dallas, Tx, 12+4+1991)
Links found for all tracks of: Van Halen - Top of the World (live at the West End Marketplace, Dallas, Tx, 12+4+1991)
Searching for Album: IDLES - Spotify Singles
Searching for whole Album: IDLES - Spotify Singles
Not matching album for: IDLES - Spotify Singles
Links found for all tracks of: IDLES - Spotify Singles
Searching for Album: Toro y Moi - Tuesday
Searching for whole Album: Toro y Moi - Tuesday
Not matching album for: Toro y Moi - Tuesday
Links found for all tracks of: Toro y Moi - Tuesday
Searching for Album: Pearl Jam - 2024-05-25- BottleRock, Napa Valley Expo, Napa, CA, USA
Searching for whole Album: Pearl Jam - 2024-05-25- BottleRock, Napa Valley Expo, Napa, CA, USA
Not matching album for: Pearl Jam - 2024-05-25- BottleRock, Napa Valley Expo, Napa, CA, USA
Links found for all tracks of: Pearl Jam - 2024-05-25- BottleRock, Napa Valley Expo, Napa, CA, USA
Searching for Album: Jethro Tull - Boston Tea Party, Boston 1970
Searching for whole Album: Jethro Tull - Boston Tea Party, Boston 1970
Not matching album for: Jethro Tull - Boston Tea Party, Boston 1970
Links found for all tracks of: Jethro Tull - Boston Tea Party, Boston 1970
Searching for Album: Johnny Cash - Spotlight + Well Alright
Searching for whole Album: Johnny Cash - Spotlight + Well Alright
Not matching album for: Johnny Cash - Spotlight + Well Alright
Links found for all tracks of: Johnny Cash - Spotlight + Well Alright
Searching for Album: Santana - Earthquake Relief Concert 1989
Searching for whole Album: Santana - Earthquake Relief Concert 1989
Not matching album for: Santana - Earthquake Relief Concert 1989
Links found for all tracks of: Santana - Earthquake Relief Concert 1989
Searching for Album: Testament - Into the Pit (2024 Remaster)
Searching for whole Album: Testament - Into the Pit (2024 Remaster)
Not matching album for: Testament - Into the Pit (2024 Remaster)
Links found for all tracks of: Testament - Into the Pit (2024 Remaster)
Searching for Album: The Offspring - Make It All Right
Searching for whole Album: The Offspring - Make It All Right
Not matching album for: The Offspring - Make It All Right
Links found for all tracks of: The Offspring - Make It All Right
Searching for Album: The Stooges - London Session 1972
Searching for whole Album: The Stooges - London Session 1972
Not matching album for: The Stooges - London Session 1972
Links found for all tracks of: The Stooges - London Session 1972
Downloading Finished
Lidarr library scan started

Tag Artifacts with Semver

Hello,

Trying out your app and I really like it! It would be very helpful if you tagged releases according to semver standards, specifically to be compatible with dependency automation tools like renovate or flux. Additionally, if you could include these tags in the [dockerhub images] published from here, that would be great!

Sync schedule not accepting 24 hour time

First of all, thank you, this tool is great. I appreciate your work.

I am trying to set up the sync schedule and I noticed:

08:35, 09:00, 10, 1130

becomes:

0,10

Whereas entering something like:

9,10,11,12

Appears to save properly in the sync command, but does not appear to do anything on the hour like I would expect.

I also tested this to see if it was expecting cronjob format and it didn't work as expected then either:

<minute> <hour> <day_of_month> <month> <day_of_week>

[FEATURE REQUEST] Trigger Lidarr "Refresh & Scan" on download finish

Trigger Lidarr "Refresh & Scan" when the downloading process is finished, this is especially useful when the download directory is the same as Lidarr's root/media directory.

Lidatube broken due to "poToken Experiments" Youtube rollout

Youtube is rolling out "poToken experiments" which breaks the version of yt-dlp that Lidatube is using, see Lidatube logs below:

WARNING: [youtube] Webpage contains broken formats **(poToken experiment detected)**. Ignoring initial player response
[youtube] NeoKe1LTWW0: Downloading ios player API JSON
[youtube] NeoKe1LTWW0: Downloading player 1f8742dc
[youtube] NeoKe1LTWW0: Downloading web player API JSON
WARNING: [youtube] NeoKe1LTWW0: nsig extraction failed: Some formats may be missing
         n = d6Qu9CfNhcKhglD48 ; player = https://www.youtube.com/s/player/1f8742dc/player_ias.vflset/en_US/base.js
WARNING: [youtube] NeoKe1LTWW0: nsig extraction failed: Some formats may be missing
         n = C0igT1mQ9xY3VB75K ; player = https://www.youtube.com/s/player/1f8742dc/player_ias.vflset/en_US/base.js
[youtube] NeoKe1LTWW0: Downloading m3u8 information
ERROR: [youtube] NeoKe1LTWW0: Requested format is not available. Use --list-formats for a list of available formats
Error downloading song: https://www.youtube.com/watch?v=NeoKe1LTWW0. Error message: ERROR: [youtube] NeoKe1LTWW0: **Requested format is not available.** Use --list-formats for a list of available formats

yt-dlp version 2024.07.16 has a fix for this:

https://github.com/yt-dlp/yt-dlp/releases/tag/2024.07.16

Avoid poToken experiment player responses

Suggest updating Lidatube to the latest YT-DLP to fix this issue.

Prevent IP Blocking by YouTube

Ran this on a large collection upon first install, and was hit with a block. I now receive the "Your IP is likely being blocked by YouTube" errors, with a number of 500 after letting it run through my collection.

Apart from using sleep scheduling (which I didn't), perhaps consider allowing changing between IPV4 and IPV6 addresses, which I believe are flags that can be passed via yt-dlp. Swapping between public IPV4 and IPV6 addresses is one way to get around rate-limited (or similar) bans.

I believe this would be as simple as the introduction of an environment variable, such as "force_ip_type", with the configuration options of "ipv4" and "ipv6". The "ipv4" option would pass the "--force-ipv4" option to yt-dlp, and the "ipv6" option would pass the "--force-ipv6" option. I believe that these options can be shortened to "-4" and "-6" respectively.

Another potential option may be openvpn integration if you wanted to be overkill.

Lidarr API sortKey does not work with external Postgres backend

Reference Issue in lidar

Trying out lidarr requests:

curl --location 'https://{lidarr}/api/v1/wanted/missing?includeArtist=true&apikey={key}&sortKey=artists.sortname&sortDirection=ascending'

Exception data:\n    Severity: ERROR\n    SqlState: 42703\n    MessageText: column \"Sortname\" does not exist\n    Hint: Perhaps you meant to reference the column \"Artists.SortName\".\n    Position: 536\n    File: parse_relation.c\n    Line: 3656\n    Routine: errorMissingColumn"

curl --location 'https://{lidarr}/api/v1/wanted/missing?includeArtist=true&apikey={key}&sortKey=artists.sortName&sortDirection=ascending'

Exception data:\n    Severity: ERROR\n    SqlState: 42803\n    MessageText: column \"Artists.SortName\" must appear in the GROUP BY clause or be used in an aggregate function\n    Position: 536\n    File: parse_agg.c\n    Line: 1433\n    Routine: check_ungrouped_columns_walker"

Would it be possible to disable sorting until the above lidarr issue is resolved?

Permissions and Post-Processing Errors

Hi there. Found this in a Reddit thread and it's a super cool project. Running into some problems getting it going, though.

First thing is the general user (1000:1000) doesn't seem to have permissions for read/write in the downloads folder within the container itself. I was able to get around this using docker exec -u root -it lidatube sh and then running chmod a+rwx downloads. Download error itself seems to have gone away after that.

Still having issues with post-processing. The files download as .temp and never fully merge:

ERROR: Postprocessing: Error opening output files: Invalid argument
Error downloading song: https://www.youtube.com/watch?v=P2C2IhmP4gk. Error message: ERROR: Postprocessing: Error opening output files: Invalid argument

Feature Request: Set folder structure and files names for downloads

Hi, I was wondering if it would be possible to set the file names and folder structure/names for downloaded files. That way, it would be possible to use the Usenet blackhole feature of Lidarr to import from the LidaTube downloads folder. Right now, Lidarr cannot see each album, as each album ends up nested in a folder with the artist name. Ideally, I'd like LidaTube to name each downloaded album as Artist - Album (year)/01 - track 1.mp3 etc.

youtube premium Music

i would like to beable to download from youtube premium Music for albums.
goes with #3

Clean up empty directories

It would be nice if LidaTube would remove any empty directories from the LidaTube Download Directory. Either on a schedule and/or by adding a button to the yt_dlp side of the UI to "Clean Up".

After I import the files into Lidarr I have to manually clean this up. Not a big deal, and I have partially automated it with a Node-Red flow, but it would be nice to have it in the same UI.

This project is great btw!!!

Lidatube downloading songs not found in Lidarr

I ran this tool recently on the first 250 results and was pleased with what was found. However, when I go to import into Lidarr, many of the tracks downloaded are not found in Lidarr and cannot be imported.

Do you know why that might be?

attempt_lidarr_import

Could you explain what attempt_lidarr_import does please? Is this different than mapping the Lidatube download folder to the root Lidarr music folder? I'd like to stay away from mapping it directly as from my first run theres already some that Lidarr is saying it doesn't like, so I'm curious if this env var is basically a "Hey Lidarr, import the ones here that you like and ignore the bad ones" or am I misunderstanding it?