blockstore
provides block allocated, single file, key value storage for JavaScript and is written in JavaScript.
Its API is multi-named so that it is also similar to memcached
and Redis
, i.e. set
is the same as setItem
.
It can be used as a server side API compatible replacement for localStorage
or LevelUP
.
Less than 275 lines of code with zero dependencies, 10K uncompressed, 5K compressed, and 2K gzipped.
Reads, writes, and deletes are all asynchronous.
The standard API is similar to localStorage
. The localStorage
API can also be used; however, currently no events are emmited.
Tested at up to 10,000,000 records of 1024 bytes (i.e. 1K) and keys of up to 16 characters on an i5 8GB Win 10 machine with a non-SSD hard drive.
Performance is impacted at large multiples of records, e.g. as record counts goes up by millions. (A testament more to the Chrome v8 engine hash lookup than anything else). Key size has an impact up to a length of 6. Small keys are up to twice as fast as the performance numbers below for 8 character keys. Other variances are probably due to garbage collection.
With caching:
Test Size: 1,000,000 Key Size: 8
Write Records Sec: 17,513
Cached Read Records Sec: 665,336
Uncached Read Records Sec: 31,460
Test Size: 2,000,000 Key Size: 8
Write Records Sec: 17,078
Cached Read Records Sec: 581,395
Uncached Read Records Sec: 31,068
Test Size: 5,000,000 Key Size: 8
Write Records Sec: 18,701
Cached Read Records Sec: 495,835
Uncached Read Records Sec: 28,731
Without caching:
Test Size: 10,000,000 Key Size: 8
Write Records Sec: 19,500
Uncached Read Records Sec: 23,624
npm install blockstore
The code is only appropriate for server use and is not transpiled. It makes extensive use of async/await and has been tested on NodeJS v7.8.0.
A thin wrapper JSONBlockstore
that automatically handles serializing and restoring non-circular JSON is also available.
const BlockStore = require("blockstore"),
bstore = new BlockStore("./test/data",true,"utf8"); // note, the directory must already exist
await bstore.setItem("akey","test string"); // bstore will automatically open on the first attempt to read or write
const data = await bstore.get("akey");
The core API is documented below. Currently you must review the code or unit tests for further functionality:
new BlockStore(path,options:{clear:false,encoding:"utf8",cache:true,compress:{keys:true,data:false}})
If clear=true
the existing data will be deleted. If compress
is missing or compress.keys
is true, keys will be compressed at start-up. If both compress.keys
and compress.data
are true then keys and data will be compressed at start-up. It is not possible to compress data
without also compressing keys.
const cnt = await <instance>.count()
Returns the number of keys in the store.
await <instance>.delete(key)
Deletes the data associated with the key
. Returns true
is keys existed and false
if it did not.
const k = await <instance>.key(number)
Returns the key at index number
.
await <instance>.flush(id)
Flushes the cache for the id
. If no id
is provided, the entire cache is flushed. If caching is turned on, returns the hit count for id
or the average hit count, otherwise undefined
.
Buffer buff = await <instance>.get(key)
Returns a Buffer
with the contents stored at the key
.
await <instance>.set(key,bufferOrString)
Adds the bufferOrString
to storage with the key
.
await <instance>.compress()
Reclaims disk storage by eliminating blank space from deleted or updated records. Returns an object with before and after sizes.
For compatibility with localStorage
the property length
and the methods getItem
, removeItem
, and setItem
are also supported.
BlockStore holds all keys in memory and any caching wrapper would also need to hold keys in memory. Basic cache management requires only 10 lines of code, so it was added to BlockStore v0.0.9 to help reduce ultimate code size and also memory load due to the duplication of keys in RAM in a high read environment. Currently, BlockStore does not automatically flush cache entries in low memory situations.
Caching is on by default. It can be turned off by setting <instance>.cache = false
. To free memory, flush the cache, <instance>.flush()
, immediately before or after turning caching
off.
A BlockStore consists of two files:
store.json
which stores the actual data. If encoded asutf8
, this can actually be viewed in a text editor.blocks.json
which stores the keys and offsets and sizes of data blocks instore.json
as well as offsets and sizes of keys in itself.
The contents of blocks.json
is kept in memory at all times, although it is written incrementally as updated.
We are seeking collaborators to enhance this module. A recovery program is needed to "intelligently" analyze and restore files in case of corruption.
We also have some thoughts about how multi-record transactions might be added.
And, of course, perhaps someone can make this smaller and faster.
v0.1.7 2017-08-08 open
was setting files size to large if datafile already existed. Fixed.
v0.1.6 2017-08-08 delete
was writing at incorrect offset. Broke in v0.1.4, now fixed.
v0.1.5 2017-07-30 open
made more robust, it now returns a Promise that may be shared among multiple open requests.
v0.1.4 2017-07-29 Entire API now asynchronus to simplify usage and expose an interface compatible with distributed sharding/clustering.
v0.1.3 2017-07-28 Simplified architecture and API to increase speed and decrease both library and runtime memory usage by removing free space tracking and encoding arguments to functions other than constructor. Store keys will autocompress on start or command. Blockstore
constructor now takes a location and an options object for arguments. Storage path is automatically
created if it does not exist. This is also in preparation for sharding/clustering mode operation.
v0.1.2 2017-07-23 Documentation updates.
v0.1.1 2017-07-23 Possible breaking change! Converted clear
to async, although underlying implementaion is still synchronous. Makes blockstore
more compatible/replaceable wih other storage.
v0.0.10 2017-07-19 Fixed issue related to not adding cache property to blocks when first loaded.
v0.0.9 2017-07-19 Added caching. After considerable thought we decided to add just a few lines of code to support caching in order to save the duplication of in memory record keys with a separate cache.
v0.0.8 2017-07-03 Codacy driven style improvements. Fixed a typo related to file encoding flag. Any users employing other than default utf8
encoding should upgrade.
v0.0.7 2017-07-03 Added unit tests. delete
now returns true
if the key existed and false
if it did not. compress
now returns before and after file sizes in an object.
v0.0.5 2017-07-01 Added length
for localStorage
v0.0.4 2017-07-01 Made API optionally compatible with localStorage
v0.0.3 2017-06-30 Minor documentation update.
v0.0.2 2017-06-30 Added load tests and made the specification of encoding more regular. Documented.
v0.0.1 2017-06-29 Generalized from JSONBlockStore v0.0.1 which was extracted from ReasonDB v3.2