drycpp / lmdbxx Goto Github PK
View Code? Open in Web Editor NEWC++11 wrapper for the LMDB embedded B+ tree database library.
Home Page: http://lmdbxx.sourceforge.net
License: The Unlicense
C++11 wrapper for the LMDB embedded B+ tree database library.
Home Page: http://lmdbxx.sourceforge.net
License: The Unlicense
in lmdb::dbi_put there is no way to put an element like :
int data[10]
lmdb::val lmdbData((void*)data, sizeof(int)*10 );
...
dbi.put(wtxn, "data", lmdbData))
Not really sure if this is an issue as I don't think there is any fix for this given the available lmdb library, but I think it is worth noting on github wiki.
As per the lmdb documents for commiting a tx "Earlier documentation incorrectly said all cursors would be freed. Only write-transactions free cursors.".
So when a wite tx is committed AND a cursor object live, the underlying MDB_cursor* is freed during the commit, then freed when the cursor object is destroyed resulting in memory violation. As such the ordering of committing and destroying (or closing) below will crash.
wtxn.commit(); cursor.close();
I do not think there is a way for a cursor to know if the tx has been committed, so there seems to be no trivial fix if one at all.
An easy solution from a users perspective is just to make sure you actively close all cursors of a write tx before before committing. Its pretty simple but it took me a while to figure out where the memory violation was coming from. Perhaps this could be added somewhere to your wiki to warn users of this.
cursor.close(); wtxn.commit();
Error is "lmdb++.h:97:12: error: thread-local storage is not supported for the current target
static thread_local char buffer[1024];"
Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin15.2.0
Thread model: posix
Right now, catching an lmdb::error
exception doesn't give much information to aid in troubleshooting incorrect uses of LMDB or runtime errors. For example, mdb_stat: Invalid argument
is particularly uninformative and unhelpful. Error messages in exceptions should ideally contain some information on what the parameters passed to the underlying LMDB API function actually were.
Weird output from cursor get: I would get both key and value inside the key variable, then the value inside the value variable :
#include <cstdio>
#include <cstdlib>
#include <lmdb++.h>
using namespace lmdb;
int getsize(const lmdb::env& e){
auto t = lmdb::txn::begin(e.handle(), nullptr, MDB_RDONLY);
auto d = lmdb::dbi::open(t, nullptr);
int r=d.size(t);
t.abort();
return r;
}
int main() {
auto env = lmdb::env::create();
env.set_mapsize(1UL * 1024UL * 1024UL * 1024UL); /* 1 GiB */
env.open("./example.mdb", 0, 0664);
{
auto wtxn = lmdb::txn::begin(env);
auto dbi = lmdb::dbi::open(wtxn, nullptr);
char a[6] = "hello";
dbi.put(wtxn, "email", "hello");
dbi.put(wtxn, "key", "value");
dbi.put(wtxn, "user", "johndoe");
wtxn.commit();
}
{
auto rtxn = lmdb::txn::begin(env);
auto dbi = lmdb::dbi::open(rtxn, nullptr);
auto cursor = lmdb::cursor::open(rtxn, dbi);
lmdb::val k, v;
while(cursor.get(k, v, MDB_NEXT)){
printf("We got '%s'\nValue '%s'\n", k.data(), v.data());
}
}
{
std::printf("size is %d\n", getsize(env));
}
return EXIT_SUCCESS;
}
Expected Output:
We got 'email'
Value 'hello'
We got 'key'
Value 'value'
We got 'user'
Value 'johndoe'
size is 3
Output :
We got 'emailhello'
Value 'hello'
We got 'keyvalue'
Value 'value'
We got 'userjohndoe'
Value 'johndoe'
size is 3
Hi, I am trying yo use lmdb++ to work with LMDB. I have it working via the example you provided. I have a problem when trying to use cursor_put with MDB_MULTIPLE. I cant seem to get it to work right. Here is the code:
`std::vector rawData(1000000, std::rand());
MDB_val value1, value2, key;
value1.mv_size = sizeof(int);
value1.mv_data = const_cast<void*> (static_cast<const void*> (&rawData[0]));;
value2.mv_size = 1000000;
MDB_val mData[2] = { value1, value2 };
int i = 0;
auto env = lmdb::env::create();
env.set_mapsize(1UL * 1024UL * 1024UL * 1024UL); /* 1 GiB */
env.open(path.c_str(), 0, 0664);
auto wtxn = lmdb::txn::begin(env);
auto dbi = lmdb::dbi::open(wtxn, nullptr, MDB_DUPSORT | MDB_DUPFIXED | MDB_INTEGERDUP | MDB_CREATE);
auto cursor = lmdb::cursor::open(wtxn, dbi);
key.mv_size = sizeof(i);
key.mv_data = &i;
cursor_put(cursor, &key, mData, MDB_MULTIPLE);`
I think I am probably misusing the lib or some part of it. So any advice would be good.
@core-process noticed and fixed this issue in our C++17 fork of lmdbxx:
If an exception was throw by txn.commit() (ie MDB_MAP_FULL), and this transaction was later aborted (because it went out of scope while unwinding the stack), then a double-free would occur.
You can use the following test to observe this (address sanitizer should be enabled, as it is by default in our Makefile):
https://github.com/hoytech/lmdbxx/blob/5223582ebf92a9b14608ce6768535aef0c65910f/check.cc#L329
I have already tried to use lmdb++ but I failed.
What I want to do is simple, it is like the example.cc but I want to get an specific value of a key of the data base instead of go over all the values that have been saved on it with cursors.
For example:
int main() {
auto env = lmdb::env::create();
env.open("./example.mdb", 0, 0664);
auto wtxn = lmdb::txn::begin(env);
auto dbi = lmdb::dbi::open(wtxn, nullptr);
dbi.put(wtxn, "username", "jhacker");
dbi.put(wtxn, "email", "[email protected]");
dbi.put(wtxn, "fullname", "J. Random Hacker");
wtxn.commit();
auto rtxn = lmdb::txn::begin(env);
std::string value;
dbi.get(rtxn, "username", value);
std::printf("value: '%s'\n",value.c_str());
rtxn.abort();
return EXIT_SUCCESS;
}
But it returns me:
Segmentation fault (core dumped)
How can I do this?
First of all I want to say thanks, lmdbxx has been very useful to me!
For anyone using C++17 in their projects, std::string_view
is a very nice complement to LMDB so I integrated it into lmdbxx. Since Arto hasn't checked in for a while, I just want to let anyone browsing here that I will maintain my fork by answering questions and incorporating bugfixes and so on.
Here's the link to my fork: https://github.com/hoytech/lmdbxx
Hi,
When I try to save orgin image data using lmdb++, I find that I cannot save images containing piexls with gray-level zero. As I have to transform the image data into a string first, the string usually 'breaks' whenever there exits zero pixels. How can I solve this problem ?
Thanks very much!
I tried to simply delete an entry, and can't really know if I'm wrong or if this is a bug, the only documentation I could find was the api reference, if anyone has better material I'd be glad to take it.
#include <cstdio>
#include <cstdlib>
#include <lmdb++.h>
using namespace lmdb;
int main() {
auto env = lmdb::env::create();
env.set_mapsize(1UL * 1024UL * 1024UL);
env.open("./example.mdb", 0, 0664);
auto wtxn = lmdb::txn::begin(env);
auto dbi = lmdb::dbi::open(wtxn, nullptr);
dbi.put(wtxn, "key_entry", "value_entry");
wtxn.commit();
dbi.del(wtxn, "key_entry");
return EXIT_SUCCESS;
}
This yields the following error :
terminate called after throwing an instance of 'lmdb::runtime_error'
what(): mdb_del: Invalid argument
In the example code provided in your Readme.MD, the second transaction uses the "dbi" instance that was opened using the first transaction:
auto dbi = lmdb::dbi::open(wtxn, nullptr);
...
auto cursor = lmdb::cursor::open(rtxn, dbi);
Is this recommended? Or should one open a new database for every transaction?
Why is it that when I do a put with a key that already exists, it doesn't update the value for that key? Do I have to delete the old key/value before adding the new key/value?
I'd like to use lmdb++ but as it stands I cannot because of it's dependency on pthreads... The real problem is that on Windows it looks like the current code requires pthread_t from mingw and I am using Visual Studio on Windows. Since it requires a modern implementation of C++ anyway, might it be possible to use only standard features?
I tried to compile your example.cc using the latest lmdb++.h
in /usr/include
like so:
g++ example.cc
But I get the followin error:
/tmp/ccUhugbK.o: In function `lmdb::env_create(MDB_env**)':
example.cc:(.text+0x14): undefined reference to `mdb_env_create'
/tmp/ccUhugbK.o: In function `lmdb::env_open(MDB_env*, char const*, unsigned int, unsigned int)':
example.cc:(.text+0x5b): undefined reference to `mdb_env_open'
/tmp/ccUhugbK.o: In function `lmdb::env_close(MDB_env*)':
example.cc:(.text+0x8e): undefined reference to `mdb_env_close'
/tmp/ccUhugbK.o: In function `lmdb::env_set_flags(MDB_env*, unsigned int, bool)':
example.cc:(.text+0xc8): undefined reference to `mdb_env_set_flags'
/tmp/ccUhugbK.o: In function `lmdb::env_set_mapsize(MDB_env*, unsigned long)':
example.cc:(.text+0x106): undefined reference to `mdb_env_set_mapsize'
/tmp/ccUhugbK.o: In function `lmdb::txn_begin(MDB_env*, MDB_txn*, unsigned int, MDB_txn**)':
example.cc:(.text+0x14f): undefined reference to `mdb_txn_begin'
/tmp/ccUhugbK.o: In function `lmdb::txn_commit(MDB_txn*)':
example.cc:(.text+0x182): undefined reference to `mdb_txn_commit'
/tmp/ccUhugbK.o: In function `lmdb::txn_abort(MDB_txn*)':
example.cc:(.text+0x1b5): undefined reference to `mdb_txn_abort'
/tmp/ccUhugbK.o: In function `lmdb::dbi_open(MDB_txn*, char const*, unsigned int, unsigned int*)':
example.cc:(.text+0x1e6): undefined reference to `mdb_dbi_open'
/tmp/ccUhugbK.o: In function `lmdb::dbi_put(MDB_txn*, unsigned int, MDB_val const*, MDB_val*, unsigned int)':
example.cc:(.text+0x239): undefined reference to `mdb_put'
/tmp/ccUhugbK.o: In function `lmdb::cursor_open(MDB_txn*, unsigned int, MDB_cursor**)':
example.cc:(.text+0x28b): undefined reference to `mdb_cursor_open'
/tmp/ccUhugbK.o: In function `lmdb::cursor_close(MDB_cursor*)':
example.cc:(.text+0x2be): undefined reference to `mdb_cursor_close'
/tmp/ccUhugbK.o: In function `lmdb::cursor_get(MDB_cursor*, MDB_val*, MDB_val*, MDB_cursor_op)':
example.cc:(.text+0x2ef): undefined reference to `mdb_cursor_get'
/tmp/ccUhugbK.o: In function `lmdb::error::what() const':
example.cc:(.text._ZNK4lmdb5error4whatEv[_ZNK4lmdb5error4whatEv]+0x1c): undefined reference to `mdb_strerror'
collect2: error: ld returned 1 exit status
Looking inside lmdb++.h
it seems that mdb_env_create
hasn't been defined anywhere, but it IS referenced.
PS: Thanks for this great library, I hope to make use of it.
Would it be possible to tag a new release?
Thanks!
In my case, I'm storing key/value pairs as int/string.
The problem arises when I wan't to check if given pair exists in database. Currently I'm only able to check if given key exists via lmdb::cursor::find. Is it somehow possible to do this for key/value pair?
Describe the bug
This is one single header wrapped in an entire package. Maybe it should just be merged into lmdb? They accept pull requests.
It makes sense that the env
and txn
classes are moveable but not copyable, since they represent resources that are closed on destruction. But dbi
isn't like that โ the destructor is a no-op. So I don't see a reason for not having a copy constructor and copying assignment.
Having these would make it cleaner to pass dbi
instances as parameters, store them in member variables, etc.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.