A attempt to rewrite Ptt about pttbbs

[LivingInPortal]

Status: I am still researching the database theory for this.
This work is quite time consuming and I have no time on this since I am a senior high school student and also limited by my finite knowledge. This is high probability to not going to realize this idea by me.

[LivingInPortal]

I finally have my time to start to make some initial progress about the basic structure of server.
I am quite confused about the database.
I found that most of data are stored in a single leveldb database file.(Like commentd, postd)
So apparently, there's only one machine to handle the database works.
Is that right?

[chhsiao1981]

I would like to propose to refactor ptt as horizontally-scalable architecture.

To my knowledge, currently ptt is efficient by heavily utilizing shm and mmap.
Given current tech-advances, the following is what I would like to propose:

1. use redis-cluster (distributed in-mem db) as the replacement of shm.
2. use mongo-cluster (distributed mem / file db) as the replacement of mmap.

We may need to refactor some data-structure as well.
(ex: separating main-content and the comments)

The proposal is welcome to comment in this thread. Once we got consensus about the proposal.
I can start to do the refactor since Dec. this year (2017) in full-time, expected finishing before Jun. 2018.

https://docs.google.com/presentation/d/1If_-ZmcDviIxLWq2USmdTtN8xsWLSlPNpUGMciZ_R-w/edit#slide=id.p

[robertabcd]

I'm experimenting with a FUSE-based solution. Still many things need to be sorted out before it can be tested with current system.

[wens]

You are welcome to try.

[SLMT]

How about Rust ? It it a new language purposed by a programmer of Mozilla. Some people are rewriting Firefox engine using Rust. Here is the link: https://github.com/servo/servo

[LivingInPortal]

Rust is definitely a fairly good idea.
But it is quite difficult to me to understand its philosophy.
And there is an implementation of PTT Webserver in Go language already.
I think it is good to follow the previous practice.

[SLMT]

Ok, I understood.
BTW, it is hard for me to believe that you are just a senior high student. As the way I see, you are more like a experienced programmer 👍

[LivingInPortal]

I know this is quite stupid question.
But I dont understand how Ptt handles such a big flow of requests with one machine....

[wens]

PTT does not use a database. Everything is stored in the file system as individual files.
Metadata is stored in .DIR files, which is just a bunch of structs. .BOARD and .PASSWD
store board and user metadata.

The leveldb databases are later add-ons, and not used directly by the core BBS system.
They may be used by additional services, such as web BBS.

[LivingInPortal]

I see.
I actually want to build the program on the platform called kubernetes.
http://kubernetes.io/
I think this would be a good choice as the platform is mature enough and is deployed in Google production system.
Of course. I don't know if this fit for the existing PTT platform.
I think this is beneficial to PTT as this increases the scalability and redundacy.
But some data structure reformation must be introduced as this is a different system, rather than file-based data structure.
And the cost introduced by the migration might decreases this benefit or even worse.
I am still investigating the possible solution and alternatives.

[wens]

Kubernetes is very low in the application stack. PTT is a monolithic system.
Moving to Kubernetes, or any distributed system for that matter, requires a complete separation
and rewrite of data storage and user connection handling.

[LivingInPortal]

This is exactly what I want to do, though.
I know the data migration is not easy to do, especially for the aging PTT system.
I am still investigating for the solution, and trying to document the work flow of PTT to understand the whole system.
So far this is just a idea, pretty naive.

[tonytonyjan]

@LivingInPortal Love your idea. I wonder if there is any progress so far.

[tonytonyjan]

@LivingInPortal

I know this is quite stupid question.
But I dont understand how Ptt handles such a big flow of requests with one machine....

FYI: http://www.kegel.com/c10k.html

PTT is built on libevent, but you can take some modern solutions into account such as libuv or libev.

[kcwu]

PTT is built on libevent

This is not true.
We did use libevent in some portion, but the major mbbsd is simple fork-based server.

[robhawkins]

Will the plan allow for an API? If we can make PTT's data accessible, then artificial intelligence applications can make use of it for analysis, similar to this or this

[wens]

Access to data is also a policy issue, which is not covered by this technical project here.

[SLMT]

Besides the policy issue, it seems like it is also hard to implement proper API on the current code structure. As my understanding, it may need lots of changes to the structure.

[LivingInPortal]

@chhsiao1981
How's the progress going now?
I was too busy last year since I was dealing with personal issues, and I almost forgot it.
I think there are main two factors that need to be considered before starting refactor: Frontend and Backend.
First, I think we need to reconsider how is our data arranged and how to migrate from the current old files-fashion data store.
Choosing of Database depends on everyone's consensus (and lots of things need our consensus too). But the most important thing to do is how we migrate our data.
I wonder if we can create a FUSE-based virtual file system to fix this...

[chhsiao1981]

hi @LivingInPortal ,

I'm proposing to separate main-content and comments (in #31 )
I'm also trying to integrate some unit-test framework now. (#34 )

The current progress is trying to setup unit-test based on check

https://libcheck.github.io/check/index.html

I think I'll finish the unit-test setup first, and then continue finishing separating main-content / comments.

The storage-mechanism is open for the choices once the separating main-content / comments is done～
(I prefer some database which utilizes the mem-usage)

[robertabcd]

I would like to close this. Please open a separate issue when there is a concrete design. Saying about using language X, database Y, framework Z isn't concrete enough.