riptl / ytpriv

YT metadata exporter

ytpriv 💾

Note: This software is not maintained and will be archived soon. I do not provide support in any way. Learn to code and fork this repository if you need a feature/bugfix.

A fast tool for exporting YouTube data using their undocumented JSON APIs.

No API keys or logins required, and no stability guarantees given.

If you find it useful, please give it a star!

Please only use this tool to the extent permitted by the YouTube ToS.

Installation

From source

Requires a Go 1.14+ toolchain.

Run go install ./cmd/ytpriv to install to $(go env GOPATH)/bin/ytpriv.

Features

ytpriv [command]
  channel     Scrape a channel
  livestream  Scrape a livestream
  playlist    Scrape a playlist
  video       Scrape a video

Channel

ytpriv channel [command]
  overview    Get overview of channel
  videos      Get full list of videos of channel
  videos_page Get videos page of channel

Livestream

ytpriv livestream [command]
  chat        Follow the live chat

Playlist

ytpriv playlist [command]
  videos      Get full list of videos in playlist
  videos_page Get page of videos of playlist

Video

ytpriv video [command]
  comments    Scrape comments of videos
  detail      Get details about a video

Attributions

Developed by @terorie

Using the amazing Go ecosystem including:

• fasthttp and fastjson for fast networking
• testify regression test helpers
• cobra for CLI
• backoff for retries
• zap logging

ytpriv's Issues

Mongo Data store

Use MongoDB as storage for the crawls.

Export livechat of past live streams

While following the live chat of active live streams is supported, this tool cannot export the historical live chat replays of past streams.

It should be straightforward to add.

video comments only save 10 subcomments per comment thread

Hello
I tried ytwrk with ytwrk video comments to save a video's comments to JSON and whilst it did save all threads/comments on the video, each thread only saved 10 subcomments on those comment threads, meaning it only saved the 10 subcomments on each comment thread, not saving the rest.
An example on this comment thread which has 488 subcomments, it would only save the first 10 to the json file, ignoring the rest: https://www.youtube.com/watch?v=e9dZQelULDk&lc=Ugx3Zffzf5edwiRain94AaABAg

How to build

Include instructions on how to build the CLI tool (or a Makefile).

[Scraping] Super Chats

YouTube supports real-time donations on live-streams called Super Chats. Those get a separate chat box, and include a donation currency and amount, and a small text message.

The livestream chat exporter should support dumping superchats.

Community Tab support

YouTube has a Twitter-like community tab for posting images and short text snippets with comment functionality.

Packaging

Publish the package to Snapcraft and maybe Ubuntu PPA.

Shell completions

Implement command shell completions using cobra tooling: https://github.com/spf13/cobra/blob/master/shell_completions.md

Redis job queue

Use a Redis job queue to save known video IDs.
Video IDs that have never been crawled should be immediately added to the job queue.
Video IDs that haven't been crawled for some time should be re-added to the end of the job queue.

Store timestamps as unix time

TODO: Store timestamps as unix time instead of date strings

Continuous Testing

Add GitHub actions, that:

• runs the Go test suite on every commit
• runs the Go test suite weekly, to detect when YouTube changes their APIs, breaking compatibility