shmul / lightningmdb

Lightningdbm is a thin wrapper around OpenLDAP Lightning Memory-Mapped Database (LMDB)

General

Lightningmdb is a thin wrapper around OpenLDAP Lightning Memory-Mapped Database (LMDB) .

Installation

Prerequisites

• Lua 5.1.x or greater. It should be compatible with Luajit though it wasn't thoroughly tested.
• OpenLDAP Lightning Memory-Mapped Database (LMDB). A clone of just the LMDB code is also available on Github

Building

LMDB

• Get LMDB by git clone https://github.com/LMDB/lmdb
• Building LMDB is a simple

cd lmdb/libraries/liblmdb/
make
make install

Lightningmdb

Luarocks

luarocks can be used to build and install the files from source.

Manually

• edit the Lightningmdb Makefile and set the Lua and Lightningmdb paths.
• run make to generate the library.

Docker

A set of docker files are provided also, primarily for building the library against multiple Lua versions. Run ./docker/build_container.sh lua5.1 to test that the library successfully builds with Lua 5.1 (versions 5.2 and 5.3 are supported as well).

Usage

Every attempt was made to honor the original naming convention. The documentation is therefore scarce and the database's documentation should be used.

5 lua objects are wrapping the access to the DB. Their mappings to the LMDB functions/constants is provided below.

The (Lua) tests files provide usage reference. Some of them are direct translation of LMDB's test files.

lightningmdb

All the LMDB enums and defines are available through this table as well as the following functions:

• version - lmdb_version
• strerror - mdb_strerror
• env_create - mdb_env_create

env

• open - mdb_env_open
• copy - mmddbb__env_copy
• stat - mdb_env_stat
• info - mdb_env_info
• sync - mdb_env_sync
• close - mdb_env_close
• set_flags - mdb_env_set_flags
• get_flags - mdb_env_get_flags
• get_path - mdb_env_get_path
• set_mapsize - mdb_env_set_mapsize
• set_maxreaders - mdb_env_set_maxreaders
• get_maxreaders - mdb_env_get_maxreaders
• set_maxdbs - mdb_env_set_maxdbs
• txn_begin - mdb_env_txn_begin
• dbi_close - mdb_env_dbi_close

txn

• commit - mdb_txn_commit
• abort - mdb_txn_abort
• reset - mdb_txn_reset
• renew - mdb_txn_renew
• dbi_open - mdb_txn_dbi_open
• stat - mdb_txn_stat
• dbi_drop - mdb_txn_dbi_drop
• get - mdb_txn_get
• put - mdb_txn_put
• del - mdb_txn_del
• cmp - mdb_txn_cmp
• dcmp - mdb_txn_dcmp
• cursor_open - mdb_txn_cursor_open
• cursor_renew - mdb_txn_cursor_renew

cursor

• close - mdb_cursor_close
• txn - mdb_cursor_txn
• dbi - mdb_cursor_dbi
• get - mdb_cursor_get
• get_key - mdb_cursor_get but the data is not returned (this isn't a part of the original API).
• put - mdb_cursor_put
• del - mdb_cursor_del
• count - mdb_cursor_count

lpack

As a utility, LHF's lpack is included in the library for Lua versions lower than 5.3.

Usage

The following is copied verbatim from lpack's original documentation.

The library adds two functions to the string library: pack and unpack.

pack is called as follows: pack(F,x1,x2,...), where F is a string describing how the values x1, x2, ... are to be interpreted and formatted. Each letter in the format string F consumes one of the given values. Only values of type number or string are accepted. pack returns a (binary) string containing the values packed as described in F. The letter codes understood by pack are listed in lpack.c (they are inspired by Perl's codes but are not the same). Numbers following letter codes in F indicate repetitions.

unpack is called as follows: unpack(s,F,[init]), where s is a (binary) string containing data packed as if by pack, F is a format string describing what is to be read from s, and the optional init marks where in s to begin reading the values. unpack returns one value per letter in F until F or s is exhausted (the letters codes are the same as for pack, except that numbers following `A' are interpreted as the number of characters to read into the string, not as repetitions).

The first value returned by unpack is the next unread position in s, which can be used as the init position in a subsequent call to unpack. This allows you to unpack values in a loop or in several steps. If the position returned by unpack is beyond the end of s, then s has been exhausted; any calls to unpack starting beyond the end of s will always return nil values.

License

• Lightningmdb is distributed under the MIT license.
• OpenLDAP Lightning Memory-Mapped Database (LMDB) is distributed under the OpenLDAP Public license.
• lpack's code was placed under the public domain by its author.

MIT License

Copyright (c) 2012,2015 Trusteer Ltd.

Author: Shmulik Regev

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, sub}}`

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

The OpenLDAP Public License

Version 2.8, 17 August 2003

Redistribution and use of this software and associated documentation ("Software"), with or without modification, are permitted provided that the following conditions are met:

1. Redistributions in source form must retain copyright statements and notices,

2. Redistributions in binary form must reproduce applicable copyright statements and notices, this list of conditions, and the following disclaimer in the documentation and/or other materials provided with the distribution, and

3. Redistributions must contain a verbatim copy of this document.

The OpenLDAP Foundation may revise this license from time to time. Each revision is distinguished by a version number. You may use this Software under terms of this license revision or under the terms of any subsequent revision of the license.

THIS SOFTWARE IS PROVIDED BY THE OPENLDAP FOUNDATION AND ITS CONTRIBUTORS ``AS IS'' AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OPENLDAP FOUNDATION, ITS CONTRIBUTORS, OR THE AUTHOR(S) OR OWNER(S) OF THE SOFTWARE BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

The names of the authors and copyright holders must not be used in advertising or otherwise to promote the sale, use or other dealing in this Software without specific, written prior permission. Title to copyright in this Software shall at all times remain with copyright holders.

OpenLDAP is a registered trademark of the OpenLDAP Foundation.

Copyright 1999-2003 The OpenLDAP Foundation, Redwood City, California, USA. All Rights Reserved. Permission to copy and distribute verbatim copies of this document is granted.

lightningmdb's Issues

Some bugs.

1;
env:get_path() return is nil
2;
env:set_mapsize(102400)
env:open(db_path, lmdb.MDB_FIXEDMAP + lmdb.MDB_NOTLS, 420)
If grow again.program crash
Maybe we need mdb_set_relctx function?

Segfault on 0.9.17.3-1

Hello and thanks for making this project available! I am using this library quite heavily in DIGITS.
We are installing the library through this command:

luarocks install lightningmdb LMDB_INCDIR=/usr/include LMDB_LIBDIR=/usr/lib/x86_64-linux-gnu

Yesterday our automatic scripts picked up a regression (see).

On my local setup I used to have version 0.9.17-1 and this was working fine. If I run the luarocks install lightningmdb command again then Luarocks fetches version 0.9.17.3-1 which causes a systematic segfault.

If I download the top of the master branch on Github then version 0.9.17.4-1 which seems to be working well.

Question: is there any known regression in 0.9.17.3-1? Can the Luarocks registry be updated to install 0.9.17.4-1 by default?

Thanks!

LuaRocks

Hello, this module is not available in LuaRocks. Could you write a rockspec for it and push it to MoonRocks? Do you need help with this?

Support for custom comparator function

As far as I can tell, there is not support for setting a custom key comparator via the function call

int mdb_set_compare ( MDB_txn * txn, MDB_dbi dbi, MDB_cmp_func * cmp )

Is this correct ? I understand that callback functions suffer from big performance problems, but not sure if that's the reason for not supporting it.

Any hint how could I use a custom comparator ?

Thanks,
Luis

Possible regression in 0.9.18.2-1

Hello, I am now seeing this error when loading lightningmdb:

/home/travis/torch/install/lib/lua/5.1/lightningmdb.so: undefined symbol: mdb_txn_id

See NVIDIA/DIGITS#908 (comment)

Is this related to version 0.9.18.2-1? Note that version 0.9.18.1-1 seems to be working fine. I am using Ubuntu and liblmdb-dev version 0.9.10-1.

mdb_test.lua:41: bad argument #2 to 'open'

When I try to run test file:
/usr/bin/lua: mdb_test.lua:41: bad argument #2 to 'open' (number expected, got nil)
stack traceback:
[C]: in function 'open'
mdb_test.lua:41: in function 'mtest'
mdb_test.lua:237: in main chunk
[C]: in ?

/usr/bin/lua test.lua
Lightning MDB version: MDB 0.9.11: (January 15, 2014)
Lightning error: Successful return: 0
-- globals --
MDB_KEYEXIST -30799
MDB_VERSION_MISMATCH -30794
MDB_PREV 12
MDB_RESERVE 65536
MDB_NEXT 8
MDB_PREV_NODUP 14
MDB_NOSUBDIR 16384
MDB_NEXT_MULTIPLE 10
MDB_TXN_FULL -30788
env_create function: 0xb7777940
MDB_SET_RANGE 17
MDB_READERS_FULL -30790
MDB_SET_KEY 16
version function: 0xb77779f0
MDB_DUPSORT 4
MDB_LAST_DUP 7
MDB_PAGE_NOTFOUND -30797
MDB_PREV_DUP 13
MDB_MAP_RESIZED -30785
strerror function: 0xb7777670
MDB_NEXT_NODUP 11
MDB_NODUPDATA 32
MDB_APPENDDUP 262144
MDB_NOSYNC 65536
MDB_INTEGERKEY 8
MDB_NOMETASYNC 262144
MDB_REVERSEDUP 64
MDB_SET 15
MDB_NOOVERWRITE 16
MDB_FIRST 0
MDB_TLS_FULL -30789
MDB_LAST 6
MDB_GET_MULTIPLE 5
MDB_GET_CURRENT 4
MDB_SUCCESS 0
MDB_WRITEMAP 524288
MDB_MULTIPLE 524288
MDB_NOTFOUND -30798
MDB_MAP_FULL -30792
MDB_RDONLY 131072
MDB_FIRST_DUP 1
MDB_REVERSEKEY 2
MDB_CURRENT 64
MDB_MAPASYNC 1048576
MDB_NEXT_DUP 9
MDB_GET_BOTH_RANGE 3
MDB_GET_BOTH 2
MDB_CURSOR_FULL -30787
MDB_FIXEDMAP 1
MDB_INCOMPATIBLE -30784
MDB_PANIC -30795
MDB_INTEGERDUP 32
MDB_CREATE 262144
MDB_DUPFIXED 16
MDB_DBS_FULL -30791
MDB_CORRUPTED -30796
MDB_PAGE_FULL -30786
MDB_APPEND 131072
MDB_INVALID -30793
userdata: 0x9623524
userdata: 0x9623524
fixedmap 1
read only 131072
-- stats --
ms_leaf_pages 0
ms_overflow_pages 0
ms_entries 0
ms_branch_pages 0
ms_psize 4096
ms_depth 0
-- info --
me_numreaders 0
me_last_txnid 0
me_maxreaders 126
ms_last_pgno 1
me_mapsize 1048576
get_path ./temp/foo
txn userdata: 0x9623704
txn userdata: 0x9623744
-- txn stat --
ms_leaf_pages 0
ms_overflow_pages 0
ms_entries 0
ms_branch_pages 0
ms_psize 4096
ms_depth 0
--- grow_db ---
userdata: 0x9623804
userdata: 0x9623804
making more room at 505
userdata: 0x962b424
userdata: 0x962b424

lua 5.2, MDB 0.9.11, Debian sid i686,

Stack manipulation issues

You have several invalid operations from a stack point of view in this library. You can see it if you use it with a Lua built with -DLUA_USE_APICHECK.

For instance your *_register functions look like this:

void env_register(lua_State* L) {
  luaL_newmetatable(L,ENV);           -- +1
  lua_set_funcs(L,ENV,env_methods);   -- 0
  lua_settable(L,-1);                 -- -2

  luaL_getmetatable(L,ENV);           -- +1
  lua_setfield(L,-1,"__index");       -- -1
}

I have added the stack changes as comments on the side. As you can see it is wrong at lua_settable except if you have a string at the top of the stack before calling the function, which is not the case.

I suppose what you want is something like this (I don't really understand what you were trying to achieve with lua_settable):

void env_register(lua_State* L) {
  luaL_newmetatable(L,ENV); 
  lua_set_funcs(L,ENV,env_methods);

  luaL_getmetatable(L,ENV);
  lua_setfield(L,-1,"__index");
  lua_pop(L, 1);
}

On 5.1 I think you also have an issue with the call to lua_setglobal in lua_set_funcs.

Use compat-5.3 instead of lpack

string.pack and string.unpack in 5.3 are slightly different to lpack.
For full compatability, consider using compat-5.3 instead.
https://github.com/keplerproject/lua-compat-5.3