- 👋 Hi, I’m @zhongl ...
zhongl / ipage Goto Github PK
View Code? Open in Web Editor NEWiPage is a light-weight java key-value store lib for message system.
License: Apache License 2.0
iPage is a light-weight java key-value store lib for message system.
License: Apache License 2.0
If a value updated, the ValueIterator also can get the value version before updating.
So, every iterating value should be check key existed in index and offset is matched.
There are too many slot objects alive in runtime can cause memory problem.
num #instances #bytes class name
----------------------------------------------
1: 5053460 404276800 java.nio.DirectByteBuffer
2: 1906356 167759328 java.util.concurrent.FutureTask$Sync
3: 1906358 61003456 java.util.concurrent.LinkedBlockingQueue$Node
4: 1906356 61003392 com.github.zhongl.benchmarker.Benchmarker$Task
5: 1906356 61003392 java.util.concurrent.Executors$RunnableAdapter
6: 2503680 60088320 com.github.zhongl.index.Slot
7: 1906356 45752544 java.util.concurrent.FutureTask
A flushed key can have multi-version values, and they will remain util key would be removed.
java.lang.IllegalStateException: java.io.IOException: Map failed
at com.github.zhongl.ipage.IndexMerger$InnerPage.<init>(IndexMerger.java:115)
at com.github.zhongl.ipage.IndexMerger.append(IndexMerger.java:90)
at com.github.zhongl.ipage.IndexMerger.mergeRestOf(IndexMerger.java:82)
at com.github.zhongl.ipage.IndexMerger.merge(IndexMerger.java:74)
at com.github.zhongl.ipage.Snapshot.append(Snapshot.java:94)
at com.github.zhongl.ipage.Storage.merge(Storage.java:145)
at com.github.zhongl.ipage.IPage$1$1.call(IPage.java:55)
at com.github.zhongl.ipage.IPage$1$1.call(IPage.java:52)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at com.github.zhongl.ipage.Actor$Core.run(Actor.java:88)
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:761)
at com.google.common.io.Files.map(Files.java:733)
at com.google.common.io.Files.map(Files.java:719)
at com.github.zhongl.ipage.IndexMerger$InnerPage.<init>(IndexMerger.java:113)
... 10 more
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)
... 13 more
Repeat remove some key may cause negative alives.
Group commit是一种保证数据安全的策略. 它是指在两次刷盘的操作之间发起的写操作都会在后一次刷盘操作执行后才被返回. 这样既保证刷盘的性价比, 又保证写数据的安全.
If system crashed when iPage was writing or flushing, it should guarantee the data before the last flush can be recovered.
So, here is the thought:
Precondition
Generally there is a special empty file named ".safe" , which indicate iPage was safe closed.
If there is not ".safe" when iPage reopened, a recovery should be started.Recovery bucket
Bucket read last 8 bytes as CRC of 163 slots, and validate them. If it doesn't match, read key and offset of every occupied slot, then validate data in the chunk by key and offset. Release the slot if data don't exist or match the key .
Recovery chunk
The records of last chunk should be scan one by one, until an record's key can not found in any bucket. Then, this record and rest will be abandon.
为什么key要必须是MD5Key ,为什么不把key换成String 或者Object ,然后内部再转成MD5Key! 难道是效率问题?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.