Comments (14)
Hi,
Found this project randomly. Seems like a cool project to a contribute to. Have researched this exact topic before.
Some links:
https://redis.io/topics/persistence
https://redislabs.com/blog/hood-redis-enterprise-flash-database-architecture
The main approach from my understanding:
- All data is kept in memory.
- Data is stored to disk so that it can be restored in the case of failure.
- The most favorable implementation of the persistence depends heavily on the load.
from lucid.
A common and high performance way to do this is to save the entry when it expires and on a GET you check the disk storage if it isn't in the map and then load it.
from lucid.
Hey guys,
Thanks for your links.
Typically Redis uses two ways to persist the data :
- RDB files (.rdb) for snapshot : It bumps data snapshot in RDB every x minutes. It's purpose is to backup the data and even push them to remote server ;
- AOF (append only file) : for actual persistence in case of system failure / reboot. Since it's only appending to a file, writing in it is fast and made for every new state in memory.
RDB is slow to write but faster to initiate, that is why this is used for snapshot only made every x data write or x minutes.
AOF is fast to write and not prone to corruption because of the "only appending" thing. There are some cases where the append is not fully completed when the disk is full, this issue has been adressed by Redis so we should.
Typically in AOF are written all operations in order to be re-instanciated in case of server failure (operations are executed and data load in memory). The commands look like this https://redis.io/commands/brpoplpush
I suggest a first implementation that builds a AOF system and think about a long term snapshot later on. The real disadvantage of AOF for snapshot is that to be load the server has to execute every operations which can take some times for long AOF.
If you have any thoughts and opinion about that go ahead guys! @CephalonRho @clintnetwork @halvorboe
from lucid.
AOF would probably be the easiest to implement well, but in my opinion the biggest problem is it's unlimited growth. A key being created and deleted very often would mean lots of wasted space. Redis seems to handle this by providing the option to rewrite the file when required, but that can cause sudden hangs and compromises the reliability of the format.
An RDB-like file format doesn't necessarily have to be slow to write, I guess it's just because it's always supposed to be as compact as possible since it's a snapshot and doesn't represent the current state of the database.
A mixed approach is also possible and might even be necessary if we want to have replication sometime in the future since recent transactions have to be synchronized across servers, which would be impossible if a server forgets about it during a restart.
from lucid.
Thanks @CephalonRho
Could you elaborate more on what you envision for "a mixed approch"? Are you talking about a compact as RDB written as AOF? I guess we still will encounter unlimited growth issue, even if it growth lesser than commands as AOF.
Do you guys know why for sure writting is slower with RDB?I think it processes some calculus for building relational schemes and write it, but it's just a guess.
I totally agree for the replication, we have to design something that will be used for it.
from lucid.
Otherwise you just return None.
from lucid.
@CephalonRho, Maybe we can move KvStore
instantiation from the Server
to Lucid
impl, in this way it could be easier to make some operations on the KvStore from subcommands and also facilitate persistence actions, what do you think?
And for the persistence, do you have an idea of how we can achieve this?
from lucid.
Hi,
Two interesting libs for persistence:
structsy
- https://docs.rs/structsy/0.1.0/structsy/persy
- https://docs.rs/persy/0.8.3/persy/
And a cool snippet to make write disk persistence, binary file writing and also compression!
Best regards
CC @CephalonRho
from lucid.
Read up a bit more. Found some more interesting links:
https://redis.io/topics/internals
https://pauladamsmith.com/articles/redis-under-the-hood.html
from lucid.
Hey @halvorboe,
I really enjoy your motivation, for Redis, I already look quickly, I will read deeply tonight.
For contributing no problem, you are welcome!
from lucid.
Hi @halvorboe
@Slals is working on persistence if you want to join us you are welcome!
from lucid.
@CephalonRho could you explain what do you mean by a mixed approach?
We need to provide an easy way of persistence and not many modes, it needs to be simple in my opinion.
from lucid.
By "a mixed approach" I was referring to simply putting an append-only format in front of an RDB-like format. Transactions could be written to both the append-only file and the RDB-like file.
The transactions in the append-only file can be deleted when they're not required anymore (replicated to another instance and written to another file), which wouldn't actually make it append-only anymore but instead allows it to work more like a growable ringbuffer. That should keep it's size small while still ensuring that no data gets lost.
from lucid.
Got you.
The way I understand it does not really differ from Redis approach. They use RDB for snapshot executed at specified intervals. They don't clear AOF though. https://redis.io/topics/persistence
Would we really gain value from mixing both, thus writting transactions on RDB every time the AOF is considered too big, then flush it / or start a new AOF?
I see one good value with this approach is that we stick with one persistence method which is saving transactions, it makes things simpler for end-users.
from lucid.
Related Issues (20)
- Hot configuration with an API endpoint
- [Manifest] About Expiration HOT 4
- [Manifest] About some operations (like LOCK, EXPIRE etc) HOT 1
- [Manifest] File Storage / Compression
- [Roadmap][Alpha] Lucid Alpha Roadmap HOT 3
- [Question] Renaming Lucid?
- Moving configuration file parsing before fern HOT 3
- [Proposition] Dump / Export HOT 2
- [Roadmap][Beta] Lucid Beta Roadmap
- [Manifest] WebUI Requirements
- Prevent Yaml Parsing Crash HOT 3
- RUSTSEC-2020-0016: `net2` crate has been deprecated; use `socket2` instead
- RUSTSEC-2020-0018: crate has been renamed to `block-cipher`
- [docs] DNS NXDomain HOT 1
- I am not sure if this qualifies as a distributed datastore HOT 1
- The docs site seems to be down. HOT 1
- RUSTSEC-2020-0146: arr! macro erases lifetimes
- Failed to compile on Apple chip M1 HOT 2
- unable to run from quickstart docs HOT 1
- Go (golang) client HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lucid.