Coder Social home page Coder Social logo

tentags.io's Introduction

About

Tentags is a content aggregration site ร  la Reddit/Voat/HN, but works a little differently under the hood:

Tags

Unsurprisingly, tags are at the core of TenTags. Each time you post to the site, you can add up to 10 tags to your post; the tags on a post decide which communities see your post. But since we don't trust you (sorry!), we let the community decide if your tags are legit, and even add their own; Instead of voting on posts users can vote on each tag, allowing the same post to have a different score per filter.

Filters

Filters are 'communities' (subs) of people that want to see the same content based on the tags on each post. They are defined by tags that people want to and don't want to see, e.g. 'sfwpics' could be all posts tagged 'pics' except those tagged 'nsfw'.

Post Once

As the tag scores govern which communities see a post, there's no need to crosspost, you can just add more tags to the post, e.g. you could tag a post 'tech' and 'android' and it would appear in the 'tech' filter, and the 'android' filter, but not the 'apple' filter.

See Once

Fed up of stale content? You can set posts to auto hide after you've voted on them, clicked on them, or even just seen them, so that you always see fresh content, even when moving between devices.

Post Anything

What if you could share a link, AND write something about it? (Crazy right?) We've got you covered. Share links, images, videos, gifs, and albums. Comment on them. Go wild. We'll even let you fix that typo in your title if you're quick.

Subscribe to Anything

No more 'remind me!' comments, simply click to subscribe to individual comment replies, all post replies, or to users to get all updates directly to your inbox.

All the comments in one place

As posts arent linked to filters, you can end up having multiple communities commenting on the same content. For easy browsing, you can filter comments by the community they came from.

No reposts

No one likes reposts, but there's always a first time for everyone. That's why we've added a repost reporting feature. Users can tag a post with its source post, and only people that haven't seen the original will see the new post.

One account, multiple users

Create additional users and easily switch between them without relogging, e.g. for seperate work/home accounts

Passwordless

Fed up of data leaks? We can't do security so we don't try. Passwordless logins make it someone elses problem; we don't even store your email address!

NSFW!!!!

Everyone's workplace is different, and 'NSFW' can range from 'phalic object on screen' to 'keep it legal.' For that reason we have a more fine grained nsfw setting, to avoid any awkward conversations with the boss. While we're at it, there's a seperate NSFL setting, so you won't get surprised by maggots instead of mammaries, or nails instead of nailing.

Installation

Requires docker-compose.

docker-compose -f dc.yml build

sh startdev.sh

Technical

Tentags.io uses Lapis, a web framework built on Openresty, with Redis as a backend database and queue, Backblaze B2 for file storage, and ffmpeg for video processing.

Each post has a set of tags that can be voted up and down by users, and each filter contains a list of post that contain tags required by the filter, and not those unwanted by the filter. In this way every post has its own seperate score per-filter.

Image processing

Links

The image processing service attempts to intelligently get the thumbnail from common websites (Imgur API, gfycat thumbnail URLs, etc), and then falls back to scraping the largest image it can find on the page. The image is then processed to the correct format and size using ImageMagick.

Images

Directly uploaded images are converted to JPG and optimised, then stored as a thumbnail, medium size, and original

  • Gifs are converted to MP4
  • Any other video formats are converted to mp4, scaled and optimized.
  • MP4's more than 15 seconds are converted to a 'preview clip' with 10 1 second segments of the video.
  • MP4's (or their preview) are converted to fallback gifs.

Caching

Tentags.io uses Nginx to cache all logged out requests and images from Backblaze.

Users/Posts/Filters are cached in Openresty shared dict shared memory zones

Valid writes are written directly to the cache and queued to shdict for deferred processing by the background workers, and cache invalidation on other servers.

Other

Tentags.io Uses the Scaling Bloom Filter library by erikdubbelboer to store per-user seen posts, post/comment/tag votes efficiently.

Passwordless logins use a salted hash of the email to identify users without storing their email addresses.

Elasticsearch used for searching URLs, post titles, and post bodies.

tentags.io's People

Contributors

kraftman avatar oudeismetis avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar

Forkers

dochne

tentags.io's Issues

fix the way tags load

currently when loading tags every single tag is loaded, and then scanned through. need to come up with something more efficient that doesnt also fill the db with thousands of tagID: tagName maps.
may be easier to remove tagIDs entirely??

add basic stats

use hyperloglogs to get stats for unique user views per filter per hour day/week/month etc track pageviews, uniques, logged in uniques, subscribers
Track user stats, add count for posts viewed/posts voted/ comments voted etc
hyperloglog sub counts per minute/hour/day

keyboard upvote/downvote

need to be able to use left and right arrows to up/downvote, hide the posts after voting, and load a new post

de-duplicate repeated functions

There are a couple of functions that have been duplicated for testing (e.g. HashIsValid in both api and posts), need to move them out to their own class

per filter user labels

each filter needs a list of labels, and an option for whether they are user selectable or must be given out by mods

check how often updates are performed by workers

may need to move to a queue system, we dont want the workers to be sitting around doing one update every 5 seconds, and we also dont want them repeatedly updating the same posts, so we need to allow some time for repeat updates, and also do as many as need doing

geo redirect

need to have each cdn add its ip to redis, add geoip to redis so we know where they are.
check users ip and redirect them to the nearest edge server.
do we log users out if their session changes?

Basic usability cleanup

check pages redirect correctly after action
Check sidebar displays links in descriptions
check page displays errors when things do
check error messages for each missing field
check error message for sending to a user who has PMs disabled
move username / settings to top right.

add image host

allow users to upload their own images allow users to report images

add scaling system

from bitbucket: track users and allow them to add more tags/have more access once they have used the site more

passwordless login

implement passwordless login

  • generate a unique token + time
  • store hash of token in DB
  • check unique key and time
  • create user session, and store session in the DB
  • allow invalidating sessions
  • invalidate the token when used
  • only allow token for certain time limit

How to invalidate client-side sessions?
On session creation, store:

  • ip
  • device
  • createdAt.
  • a unique sessionID

Store he new session in the db under the user info
Store client side:
session token
store server side:
everything thats currently in the cookie.

session invalidation:
session timeout (days of inactivity)
check IPs match?
add store in redis of terminated sessions, make sure all servers read from it regularly.

set the required/banned thresholds to avoid trolling

need a way to stop people from adding tags deliberately to things in order to get them banned from subs. eg if someone just adds a nsfw tag straight away it will get banned from any nsfw subs, so need to set a threshold when considering tags that a filter doesnt want at a different level as tags that a filter does want

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.