zhongl / ipage Goto Github PK

View Code? Open in Web Editor NEW

42.0 5.0 12.0 936 KB

iPage is a light-weight java key-value store lib for message system.

License: Apache License 2.0

Java 100.00%

ipage's Introduction

What is it?

iPage is a java key-value store library, which is designed for message system.

Features

Very fast write by sequence appending
Random reading by index
[A]synchronize invocation
Flush By write count or time elpase
Group commit
Date recovery after crash
Garbage data collecting (or Defragment)
Degrade after OOM
Monitor and control by JMX

Please see issues.

Benchmark

| Case            |   Times   | Total Eplase | Average Elapse | TPS         | Concurrency |
|-----------------|-----------|--------------|----------------|-------------|-------------|
| Random get      | 1,000,000 | 6.009 s      | 6,008   ns     | 166444.74   | 8           |
| Async add       | 1,000,000 | 24.90 s      | 24,904  ns     | 40154.19    | 16          |
| Async remove    | 1,000,000 | 32.84 s      | 32,842  ns     | 30448.82    | 16          |
| Sync add        | 16,384    | 5.142 s      | 313,857 ns     | 3186.31     | 32          |

Read and write data is 1KB

Enviroment

Linux 2.6.18-164.el5
OpenJDK 64-Bit Server VM (build 1.6.0-b09, mixed mode)
Intel(R) Xeon(R) CPU E5620 @ 2.40GHz 8 core
Memory 24GB
SCSI 1TB

Getting started

Preconditions

Make sure your enviroment has:

Git
Maven

Run test

> git clone [email protected]:zhongl/iPage.git
> cd iPage
> mvn clean test

Run benchmark

> mvn clean test -Dtest=IPageBenchmark -DargLine="-Xmx512m -Xms512m"

Usage

Please see:

IPageTest

Design Doc

ipage's People

Contributors

Stargazers

Watchers

Forkers

nathanchen jonzhang708 vintagewang chenld universsky ihoney 19990909 xnkl640 puzzydog antholoj brsteg12 viv-merv

ipage's Issues

Avoid concurrent modification problem of value iterator

Reduce line file creating nums

JVM Crash by iterating values

Add use guide

Improve get performance

Data recovery

If system crashed when iPage was writing or flushing, it should guarantee the data before the last flush can be recovered.

So, here is the thought:

Precondition

Generally there is a special empty file named ".safe" , which indicate iPage was safe closed.
If there is not ".safe" when iPage reopened, a recovery should be started.

Recovery bucket

Bucket read last 8 bytes as CRC of 163 slots, and validate them. If it doesn't match, read key and offset of every occupied slot, then validate data in the chunk by key and offset. Release the slot if data don't exist or match the key .

Recovery chunk

The records of last chunk should be scan one by one, until an record's key can not found in any bucket. Then, this record and rest will be abandon.

UnderflowException if autoGC is true

Flushing would slow down

Map failed

java.lang.IllegalStateException: java.io.IOException: Map failed
    at com.github.zhongl.ipage.IndexMerger$InnerPage.<init>(IndexMerger.java:115)
    at com.github.zhongl.ipage.IndexMerger.append(IndexMerger.java:90)
    at com.github.zhongl.ipage.IndexMerger.mergeRestOf(IndexMerger.java:82)
    at com.github.zhongl.ipage.IndexMerger.merge(IndexMerger.java:74)
    at com.github.zhongl.ipage.Snapshot.append(Snapshot.java:94)
    at com.github.zhongl.ipage.Storage.merge(Storage.java:145)
    at com.github.zhongl.ipage.IPage$1$1.call(IPage.java:55)
    at com.github.zhongl.ipage.IPage$1$1.call(IPage.java:52)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at com.github.zhongl.ipage.Actor$Core.run(Actor.java:88)
Caused by: java.io.IOException: Map failed
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:761)
    at com.google.common.io.Files.map(Files.java:733)
    at com.google.common.io.Files.map(Files.java:719)
    at com.github.zhongl.ipage.IndexMerger$InnerPage.<init>(IndexMerger.java:113)
    ... 10 more
Caused by: java.lang.OutOfMemoryError: Map failed
    at sun.nio.ch.FileChannelImpl.map0(Native Method)
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)
    ... 13 more

ConcurrentModification during iterating if new merging completed

Keep last flush offset for IPage recovery

Maven compile failure

Should not use md5 of record as it Md5Key

Improve indices merging by Grouping

Auto garbage collection

Lower defrag frequencies

ReadOnlyBufferException after restart

Too many slot objects alive

There are too many slot objects alive in runtime can cause memory problem.

 num     #instances         #bytes  class name
----------------------------------------------
   1:       5053460      404276800  java.nio.DirectByteBuffer
   2:       1906356      167759328  java.util.concurrent.FutureTask$Sync
   3:       1906358       61003456  java.util.concurrent.LinkedBlockingQueue$Node
   4:       1906356       61003392  com.github.zhongl.benchmarker.Benchmarker$Task
   5:       1906356       61003392  java.util.concurrent.Executors$RunnableAdapter
   6:       2503680       60088320  com.github.zhongl.index.Slot
   7:       1906356       45752544  java.util.concurrent.FutureTask