Podsync not updating new content about podsync

[mmarfil]

Hello 👋—Just came here to report the very same issue. None of my feeds have updated today, and some of them are not even playing the videos.

[mxpv]

Can you share concrete example?

[iHack3x2]

It did update in between my original comment, but for the most part it is still behind by like a day.

Democracy Now:
https://podsync.net/rg5A4gk08
https://www.youtube.com/user/democracynow

Secular Talk:
https://podsync.net/3SfOioUz-
https://www.youtube.com/user/SecularTalk

Although these two happened to grab something within the last 5 hours.

BlackPenRedPen:
https://podsync.net/yiXwwoUzV
https://www.youtube.com/user/blackpenredpen

VICE News: (Missing the latest video, which was uploaded about 4 hours ago from this comment.)
https://podsync.net/4H20Kck08
https://www.youtube.com/user/vicenews/videos

[dizziness1138]

Same here. No updates for three days.

[mmarfil]

Yeah, just a little update. I unsubscribed from all feeds, and re-subscribed with new links. It worked fine for about 24h but then it stopped updating again.

Besides, I'm having all sorts of connectivity issues. Sometimes videos stop working (~15 min in or so), others it won't even load.

[huskies62]

Has this being resolved? My feed doesn't work well. If I try to download one at a time and keep trying I usually end up getting them all but it can take days

[amcgregor]

I must report "seemingly ordinary operation" (I have more trouble with the NPR Hourly News actual podcast…) however attempting to download episodes occasionally takes a few tries. Usually two, which indicates to me possibly "lazy" server-side (podsync-side) collection of video content. Tap download, get the error popup, tap retry, it usually works. Sometimes it takes a few minutes, if the episode is particularly large.

[dizziness1138]

Similar @amcgregor, I run the RSS feed into Zapier to save a copy to S3. Those frequent timeouts cause the Zap to fail.

[iHack3x2]

@amcgregor hmm, I'm not sure I just thought it was because of the demands it puts on the server. According to the FAQ:

Does feeds auto-update as new videos show up on YouTube? Or do they just represent a snapshot of whenever you generated them?
Feeds are cached for 10 minutes to reduce server loads.

So I thought it meant every 10 minutes it is updated, but perhaps that isn't the case anymore. Maybe Podsync has become quite popular and they increased the time. I just wish the communication was better and they would say what is going on. The way the guy was talking on Patreon, the reason this now open source is because he needs help maintaining it all.

[amcgregor]

@iHack3x2 "Cached for 10 minutes" means, typically, the data being discussed will "persist" or "live" for 10 minutes after first generation or request without needing to be re-generated or re-requested. Not that something happens every 10 minutes. If the cache does not contain a hit for the requested item, it will be regenerated or scheduled for re-request the next time the operation is attempted. Edited to add: notably, if you request the same feed within 10 minutes of your last request for that feed, no additional work would be performed by Podsync's servers. They'd just hand you back the exact same podcast feed, from cache.

This aligns with my experience of requesting a video to download, it failing, tapping retry, then it working. Between the first attempt and hitting retry, the background job to pull in the video completed, making it available on the second try.

[sch1308]

even after retrying the download I still get the error 500 message. im using podcast addict.

[michellichand]

My feeds don't even update. I've had channels release videos 2-3 days ago and they don't show up in PodcastAddict. I usually have to re-create the feeds to get them to update, which is very time consuming.

[humphreyyy]

I'm seeing the same issues. Feeds not updating, episodes not downloading, error 500s. It's just completely broken. I'm using the official podcast app in iOS13 but it's the same story on other apps

[michellichand]

Considering the creator hasn't said anything in months, should we be looking for an alternative software?

…

On Mon, Sep 30, 2019, 12:23 humphreyyy ***@***.***> wrote: I'm seeing the same issues. Feeds not updating, episodes not downloading, error 500s. It's just completely broken. I'm using the official podcast app in iOS13 but it's the same story on other apps — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#34?email_source=notifications&email_token=AHEE64GB7VQTPJLSKDY3M5LQMJGZ5A5CNFSM4IN4LG3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD76ZQBA#issuecomment-536713220>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHEE64DGWO62UQFMV2VLKWTQMJGZ5ANCNFSM4IN4LG3A> .

[humphreyyy]

I've tried, but haven't really found anything that works the same way :(

[iHack3x2]

@michellichand I'm treating it more as an archive, I no longer expect the latest videos. My only issue is feed not syncing in a timely manner. I'm still a pateron and hoping that the issues will get resolved, so far it's my best option because no other service that I know of is going to load 600 videos.

[michellichand]

Also still a patreon, if only because when I dropped my pledge, my feeds got considerably slower.

…

On Mon, Sep 30, 2019 at 4:56 PM iHack3x2 ***@***.***> wrote: @michellichand <https://github.com/michellichand> I'm treating it more as an archive, I no longer expect the latest videos. My only issue is feed not syncing in a timely manner. I'm still a pateron and hoping that the issues will get resolved, so far it's my best option because no other service that I know of is going to load 600 videos. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34?email_source=notifications&email_token=AHEE64CG4GQEEHYJNSXEVNTQMKG2HA5CNFSM4IN4LG3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD77ORQA#issuecomment-536799424>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHEE64DKD42OBFKDLWQPTNTQMKG2HANCNFSM4IN4LG3A> .

-- *Michel Lichand* [phone] [email protected]

[amcgregor]

I felt bad, given the sheer volume of media I consume in a given day, so examined the open source code here and implemented my own "podsync server" (not actually running this code) as a shell script automating youtube-dl calls wrapped in GNU's parallel tool to improve performance. My own usage will no longer be adversely impacting other users of Podsync. ;)

I had to apply custom patches to youtube-dl: they're described in the example linked below. Predominantly these were to avoid re-requesting the same video detail page over and over and over again, only to be rejected for date range restrictions, and preventing re-request of every page of video results when literally only the first video on the first page will have been new.

Here is everything you need. (Includes youtube-dl automation shell script, notes on youtube-dl patches/tweaks/hacks, Python script to generate a Podcast RSS feed from a directory of videos downloaded this way, and the template for that RSS feed. Current major defects are:

• Video upload times are just… gone. Haven't figured out a sane way to handle that, or how podsync manages it, yet.

• Playlist Podcast titles should actually be the description. If you sync multiple playlists from the same creator, they'll all show up as the name of the creator. ;)

• No channel thumbnails. (This makes the last point a bit harder to work with.)

I threw my initial set of ~160 or so channels at it, first run through took just shy of 10 hours on a home gigabit fibre connection. Subsequent runs to pull in just the new videos in the last 12-24 hours takes, on average, around 10 minutes to explore every channel. I do this three times a day, once in the morning just before I wake up (so my morning newsy type things and stuff are ready when I get up), once just prior to noon, and the last time just before 5pm. I'll warn you: have hard disk space and patience to spare. My current set of casts consumes 1.4 TiB on-disk. (I don't bother downloading episodes to the Podcasts player app any more, I just stream them from my home web server. ;)

[sch1308]

@amcgregor i switched to gpodder and it seems to work 90 percent of the time, it has youtube-dl built in

[amcgregor]

@sch1308 Heh, sorry, I'm trying to reduce a dependence on graphical tools, not add more to the toolbox. Undoubtedly useful for less technical users, though! (Amusingly, I notice the "released" column references the day, not time. ;)

With my sample script, you construct a venv with one line, drop the files into it, modify to list your desired channels, pip install youtube-dl cinje, and run (./pull.sh). Point a web server (I use Nginx; six lines of configuration) at that directory, then set up a cron task to periodically re-run it (crontab -e), and you're done. Never need to touch it again, unless modifying the list of synchronized channels. (Plus the fun side benefit of having very explicit control over the video resolution pulled in. Something you want quality for? Snag the 4K or 1080p. Something you just listen to? Grab just the audio, or limit to 240p, just like Podsync itself, except with explicit control and awareness. What's "high quality"? What's "low quality"? Doing it yourself, you know precisely.)

Edited to add: the scripts have been updated with a substantial amount of documentation by way of comments in the code. Instructions on use, installation instructions, example Nginx configuration to serve the content, etc.

[iHack3x2]

Yeah, itś been a while and my feeds no longer update. I think they were last sync October the 3rd or 4th. I could handle if it was day or two late, but this is a bit ridiculous for me now. I hope things get solved.

Unfortunately, I will probably be dropping my patreon support. If anyone find an alternative solution, let me know. I just want to be able to generate an RSS feed of youtube channels, 300+ would be great and have it sync in a timely manner.

[sch1308]

@iHack3x2 see my comment above

[iHack3x2]

@sch1308 thanks but I tried it, I already have a RSS feed reader, Newsboat and prefer CLI applications But yeah like YouTube RSS feed it only pulls 15 videos, I want to be able to go back and watch older videos without having to go to the website. Newsboat is pretty awesome, I would like to keep using it.

[mxpv]

this needs different implementation, issue to track: #38