[propose] Replace .PASSWDS with DB (mongodb) about pttbbs

[wens]

However, it will be difficult for web-based service because
it's hard to keep track of userec without reading .PASSWD.

This doesn't quite make sense. Regardless of the backend storage, you still need to read out the data,
be it a blob that expands into multiple fields, or a database system that presents the fields to you.

hi @wens ～

Though we still need to read out the data,
but I thought that reading data from horizontal-scalable db with replica through intranet

It will not be horizontally scaled, but likely sitting on the same machine.
We simply do not have machines to do "horizontally scaled systems".
So you are trading single host performance and complexity for unachievable
horizontal scalability. I doubt it would perform better.

are much safer than reading data from 1 local file.

How is that? If you are reading through intranet it will be equally fast.
If the main machine breaks down, well the whole site will be down anyway.
And if the new system keeps on running with state updates, then you end
up with data synchronization issues.

I think "safer" is not the word you are looking for.

(Or now we are using m2ssd and reading data from file is not any more huge issue？～)

AFAIAC you should be more focused on getting the new system up with basic features,
and worry about performance bottlenecks later.

[wens]

However, it will be difficult for web-based service because
it's hard to keep track of userec without reading .PASSWD.

This doesn't quite make sense. Regardless of the backend storage, you still need to read out the data,
be it a blob that expands into multiple fields, or a database system that presents the fields to you.

(.PASSWD is one gigantic file recording all the users.
It's not feasible to know whether data of one user is updated
just by looking at the mtime of the file.)

It would make more sense if you explained "why" you need to know whether individual users were updated.

[chhsiao1981]

hi @wens ～

The proposed web-based service are divided with 2 layers,
including:
backend: the same machine as the original pttbbs server, intending to directly access the data in shared-mem and files.
middleware: a separate machine (hopefully besides the original pttbbs server) dealing with frontend and communicate
with the backend.

middleware is designed to be easily horizontally scalable with easily horizontally scalable db.

For the file-based data in pttbbs, we can have some permanent-db in middleware as the cache to avoid requesting data from backend/pttbbs server too often.
What I think of the cache-validation mechanism is to compare the mtime of the file and the last-update-time in the middleware. => need to know whether individual users were updated.

[chhsiao1981]

However, it will be difficult for web-based service because
it's hard to keep track of userec without reading .PASSWD.

This doesn't quite make sense. Regardless of the backend storage, you still need to read out the data,
be it a blob that expands into multiple fields, or a database system that presents the fields to you.

hi @wens ～

Though we still need to read out the data,
but I thought that reading data from horizontal-scalable db with replica through intranet
are much safer than reading data from 1 local file.
(Or now we are using m2ssd and reading data from file is not any more huge issue？～)

[wens]

Doesn't that mean you end up polling 2M number of users say every minute?
That's not much better than just throwing data away every minute.
Just cache it for 1 minute and re-fetch when it expires.
Or maybe just don't cache it at this time.
Wait until it actually becomes a performance bottleneck.

[chhsiao1981]

However, it will be difficult for web-based service because
it's hard to keep track of userec without reading .PASSWD.

This doesn't quite make sense. Regardless of the backend storage, you still need to read out the data,
be it a blob that expands into multiple fields, or a database system that presents the fields to you.

hi @wens ～
Though we still need to read out the data,
but I thought that reading data from horizontal-scalable db with replica through intranet

It will not be horizontally scaled, but likely sitting on the same machine.
We simply do not have machines to do "horizontally scaled systems".
So you are trading single host performance and complexity for unachievable
horizontal scalability. I doubt it would perform better.

are much safer than reading data from 1 local file.

How is that? If you are reading through intranet it will be equally fast.
If the main machine breaks down, well the whole site will be down anyway.
And if the new system keeps on running with state updates, then you end
up with data synchronization issues.

I think "safer" is not the word you are looking for.

(Or now we are using m2ssd and reading data from file is not any more huge issue？～)

AFAIAC you should be more focused on getting the new system up with basic features,
and worry about performance bottlenecks later.

Sounds good～ I'll close the issue for now～