silicium14 / peertube_index

A centralized search engine for PeerTube videos

PeerTube Index

PeerTube Index is a centralized search engine for PeerTube videos. It crawls PeerTube instances, index their videos, and discovers new instances. It is hosted at peertube-index.net.

This project is not affiliated to Framasoft, the maintainer of the PeerTube software.

How it works / what it does

• Scans instances if they have not been scanned for more than 24 hours
• Respects robots.txt directives
• Discovers new instances by adding the followed and following instances of a known instance being scanned
• Periodically updates its list of known instances from thefederation.info and instances.joinpeertube.org
• No scanning retry: at the first scanning error (network error, HTTP error, parsing error...) the instance is marked as failed for the next 24h

State of the project

I will stop maintaining PeerTube Index, it will be retired soon.

Framasoft, the maintainer of PeerTube, now has an official solution for global search on the PeerTube federation: Sepia Search at https://search.joinpeertube.org/. People can use it instead of PeerTube Index. It is open source and people can use it to setup their own search engine with their own rules if they want. It does not automatically discovers new instances as PeerTube index does (by fetching following and followed instances).

peertube_index's Issues

Get only local videos

Hi,

peertube index crawls all instances videos, but you are only interested in local videos. Please add filter=local to your request URL (example: https://peertube2.cpy.re/api/v1/videos/?start=0&count=25&sort=-publishedAt&filter=local) to avoid unnecessary load on peertube instances

Search API

Hi, thanks for this useful project!

Please document a search API, so that applications like Thorium can use PeerTube Index for search.

[Feature Request] a NSFW/sensitive check box on the search page

Currently this tool does not return results for content marked sensitive. This is a good precaution, but a user may want sensitive results returned.

I suggest allowing users to check whether they want them returned using a simple check box.

Additionally if issue #2 is implemented it would be a good idea to include this functionality in the API.

Use of Google Fonts expose private data

Currently google fonts are beeing included with this line:

<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Bitter&display=swap">

So each and every call of the webpage is treacked by Google - for an audience which may choose Peertube over Youtube for not beeing tracked.

I kindly ask for removing this anti-feature. The site could work well just with standard fonts, for a pure search the font layout may be mostly irrelevant. Or the fonts could be hosted by oneself, so that not Google is called.

More info about hosting fonts is here:
https://stackoverflow.com/questions/8966740/how-to-host-google-web-fonts-on-my-own-server

Payd content, oder just easy abused?

When I use the search engine with this search string https://peertube-index.net/search?text=deluge I get 8x companys "Deluxe - Deluxe Furnishing Co" then "DÉLUGE" then "Smoke Deluxe Clip "Rockstar"" before I finally get a result where I can find the word i was searching for.

Is this paid content, or just easy abused ?

Missing instances?

It looks like no results from this instance are included, and it's one of the top 20 instances for total videos: https://tube.4aem.com

Rate Limit for Crawler

Hi,

I noticed today that I was getting a lot of requests on peertube.social, for URLs like /api/v1/videos/video-id. At times I was getting around 50 requests per second, and this caused a ton of CPU usage. Now I dont know if this was you, but it definitely looked like a crawl and apparently your site started on the same day, so it seems likely.

The problem is gone for now, probably because the crawler has finished its backlog. But you should definitely add a rate limit to your crawler if you havent already. I suggest something like 1 request per second at most.