wardpearce / syncious

A self-hostable API extension for Invidious, made for custom clients.

Invidious API Extended

A self-hostable API extension for Invidious, made for custom clients.

Features

• Watch progress API
  • Keep your watch progress synced between Invidious clients / sessions.
• Integrates with existing Invidious accounts.
• Doesn't rewrite Invidious auth.
• Doesn't modify any existing Invidious records.
• Auth & response caching.
• API Documentation.

Auth

API-Extended uses Invidious SIDs or tokens to validate requests. These should be passed as a Bearer token. If using Invidious tokens, it must have the GET:feed scope.

Supported clients

• Materialious
  • Modern material design for Invidious.

Deployment

services:
  api_extended:
    image: wardpearce/invidious_api_extended:latest
    restart: unless-stopped
    ports:
      - 3004:80
    environment:
      api_extended_postgre: '{"host": "invidious-db", "port": 5432, "database": "invidious", "user": "kemal", "password": "kemal"}'
      api_extended_allowed_origins: '["https://materialios.localhost"]'
      api_extended_debug: false

      # No trailing backslashes!
      api_extended_invidious_instance: "https://invidious.example.com"
      api_extended_production_instance: "https://extended-api.example.com"

      # You can read more about optimizing Syncious here
      # https://hub.docker.com/r/tiangolo/uvicorn-gunicorn
      WEB_CONCURRENCY: 4

syncious's Issues

Add to documentation how to reduce uvicorn threads and memory load

Old issue text

Cheers @WardPearce got some questions for you about things I noticed when setting syncious up, mainly performance related:

1. Syncious uses a lot of memory, roughly 1.7 GiB directly after it started (comparison: Materialious only uses ~10 MiB). I am slowly getting to a point where the memory on my server is mostly allocated, so adding another 1.7 GiB (probably more) is nothing I can just do and forget about it. Do you have any plans / idea on how to reduce this load in the future or is it "100% normal and expected"?

2. Syncious also always starts with 32 threads (which it probably takes from the total cpu threads of my server). Is there any way to reduce that? I can hard force a cpu limit with docker-compose but that's probably not "best practise" as it still starts with 32 threads.

Final thoughts: Don't get me wrong, but this many threads and this high memory footprint sounds a little bit extreme for a plugin that basically "only" syncs invidious sessions. Please don't take this the wrong way, I love what you're doing for the community.
But I would relly love this to be optimized before I deploy it on my server.

Update / fixed it all myself:
Setting the environment var WEB_CONCURRENCY will reduce the number of threads and drastically reduce the memory load. You should probably add this to your documentation.
Uvicorn docs

Cheers :D

Feature request: Support pgbouncer

Currently, syncious (due to using prepared statements) doesn't support pgbouncer in transaction mode.
My Invidious instance is rather big, so using pgbouncer for all db connections became an absolute must at some point. I would like sycious to use the connection pool of pgbouncer too, instead of using direct connections. I often have daily db connection spikes (during which pgboucner uses an additional reserve pool) and fear that syncious will at some point run into problems when using direct connections.

Request: Please make syncious pgbouncer compatible.

You could add do this by changing the db connection string as follows:
'{"host": "invidious-db", "port": 5432, "database": "invidious", "user": "kemal", "password": "kemal", "pgbouncer": true}'

And then change your code so that prepared statements aren't used when pgbouncer is set to true.

Cheers