symisc / vedis Goto Github PK
View Code? Open in Web Editor NEWAn Embedded Implementation of Redis
Home Page: http://vedis.symisc.net
License: Other
An Embedded Implementation of Redis
Home Page: http://vedis.symisc.net
License: Other
Hi, I like this project, but I need command LRANGE
, I didn't find it..
The vedis.c file in github is incomplete (200kb vs 700kb on the website) and didn't compile for me.
Also, would you be interested [to implement / if I implemented] a dbsize command which returns the number of bytes used by the journal (in memory)? I know there's an alternative through xAlloc / xFree in the allocators but it's harder to segment. This would help a lot to in cleaning up and avoiding out of memory errors.
Thank you for this great library
I discovered an issue in the vedis kv store due to the handling of defragmenting slave pages. The defragment function only uses the cell array on the current page, rather than the master page. The fix is to replace the final line in this:
static int lhPageDefragment(lhpage *pPage) { lhash_kv_engine *pEngine = pPage->pHash; unsigned char *zTmp,*zPtr,*zEnd,*zPayload; lhcell *pCell; /* Get a temporary page from the pager. This opertaion never fail */ zTmp = pEngine->pIo->xTmpPage(pEngine->pIo->pHandle); /* Move the target cells to the begining */ pCell = pPage->pMaster->pList; /* replaced with pPage->pMaster */
I have attached an example which will demonstrate corruption of the kv store when performing the sequence of operations in the test file.
Broken-vedis.zip
Hi.
I try use vedis but got some wrongs :(
>>>LPUSH name a1 a2 a3
[3]
>>>LINDEX name 0
[a1]
>>>LPOP name
[a1]
>>>LINDEX name 0
[nil] //Here should be "a2"
>>>LINDEX name 0
[nil] //Here should be "a2"
>>>LINDEX name 0
[nil] //Here should be "a2"
>>>LLEN name
[2]
the command "LINDEX" must be wrong after the “LPOP” .
Hello, friends
Please tell me, maybe I’m blind, but I don’t see a method for getting all the keys. Is there any analogue of the keys command in redis?
vedis contains all C source code in one file. it is not easy to read and understand.
May I have the vedis separate files ? I need it to study how it work
thank you
Sometimes vedis fails to store key. vedis_kv_store
returns VEDIS_OK
, but the successive vedis_kv_fetch
returns VEDIS_NOTFOUND
for the same key.
It doesn't occur for in-memory store. Also I can't reproduce it without vedis_kv_delete
calls.
Here is the test case. Usually it fails within 1 minute with "Unable to fetch: -6".
#include "vedis.h"
#include <stdlib.h>
#include <time.h>
#include <iostream>
// generate random bytes of variable (important!) length
char *randomBytes(int* len)
{
*len = rand() % 4 + 1;
char *res = new char[*len];
for (int i = 0; i < *len; i++)
{
res[i] = rand() % 256;
}
return res;
}
int main()
{
vedis_lib_init();
vedis *store;
int res = vedis_open(&store, "vedis.db");
if (res != VEDIS_OK)
{
std::cout << "Error opening vedis:" << res << std::endl;
return 0;
}
srand(time(NULL));
char buffer[255];
vedis_int64 inout;
int block = 0;
int count = 0;
while(1)
{
int keyLen;
int valLen;
char *key = randomBytes(&keyLen);
char *val = randomBytes(&valLen);
// insert random key
res = vedis_kv_store(store, key, keyLen, val, valLen);
if (res != VEDIS_OK)
{
std::cout << "Unable to store: " << res << std::endl;
return 0;
}
// verify it was stored correctly
inout = 255;
res = vedis_kv_fetch(store, key, keyLen, &buffer[0], &inout);
if (res != VEDIS_OK)
{
std::cout << "Unable to fetch: " << res << std::endl;
return 0;
}
// generate new random key and delete if it exists
delete []key;
key = randomBytes(&keyLen);
inout = 255;
res = vedis_kv_fetch(store, key, keyLen, &buffer[0], &inout);
if (res == VEDIS_OK)
{
res = vedis_kv_delete(store, key, keyLen);
if (res != VEDIS_OK)
{
std::cout << "Unable to delete: " << res << std::endl;
return 0;
}
}
delete []key;
delete []val;
count++;
if (count > 10000)
{
block++;
count = 0;
std::cout << block << " 10K iterations" << std::endl;
}
}
return 0;
}
I am trying to include vedis in an esp32 project using esspesif's esp-idf and it throws several errors and won't compile.
Is this even possible.
I have defined
#define OS_OTHER
../main/vedis.c: In function 'vedisTokenizeInput':
../main/vedis.c:3076:73: error: parameter 'pUserData' set but not used [-Werror=unused-but-set-parameter]
static sxi32 vedisTokenizeInput(SyStream *pStream,SyToken *pToken,void *pUserData,void *pCtxData)
~~~~~~^~~~~~~~~
../main/vedis.c:3076:89: error: parameter 'pCtxData' set but not used [-Werror=unused-but-set-parameter]
static sxi32 vedisTokenizeInput(SyStream *pStream,SyToken *pToken,void *pUserData,void *pCtxData)
~~~~~~^~~~~~~~
../main/vedis.c: In function 'MemHashInit':
../main/vedis.c:10197:55: error: parameter 'iPageSize' set but not used [-Werror=unused-but-set-parameter]
static int MemHashInit(vedis_kv_engine *pKvEngine,int iPageSize)
~~~~^~~~~~~~~
then lots of
'if' clause does not guard...
parameter 'pLex' set but not used
variable 'zExtra' set but not used
which shouldn't be a problem but also
but then
../main/vedis.c:1848:33: warning: 'vedisExportBuiltinVfs' used but never defined
VEDIS_PRIVATE const vedis_vfs * vedisExportBuiltinVfs(void);
Hi. Resieving this traceback:
----------------------------------
2022-12-21 12:16:48.042288
Traceback (most recent call last):
File "/home/vedischeck/dbworker.py", line 30, in write_user_attribute
db[f'{user_id}{attribute}'] = data
File "vedis.pyx", line 383, in vedis.Vedis.__setitem__
File "vedis.pyx", line 311, in vedis.Vedis.store
File "vedis.pyx", line 425, in vedis.Vedis.check_call
OSError: b'IO error while opening journal file: /home/vedischeck/main.vdb_vedis_journal\n'
----------------------------------
2022-12-21 12:16:48.061854
Traceback (most recent call last):
File "/home/vedischeck/dbworker.py", line 30, in write_user_attribute
db[f'{user_id}{attribute}'] = data
File "vedis.pyx", line 383, in vedis.Vedis.__setitem__
File "vedis.pyx", line 311, in vedis.Vedis.store
File "vedis.pyx", line 425, in vedis.Vedis.check_call
OSError: b'IO error while opening the target database file: /home/vedischeck/main.vdb\n'
(venv) root@vm3616877:/home/vedischeck#
Using Vedis as simple one-file memory-cache in some python projects.
At first i tried to .queue my actions, also tried asyncio Semaphore, to prevent concurrent access. But then i noticed, that trouble ouccurs only on VPS, on my home laptop everything works perfect, and concurrent acces is pergectly proceeded by Vedis itself....
And trouble occurs exatly on 1010 th iteration all the time...
1009
async_write - True
1010
async_read - None
1011
async_write - False
1012
async_read - 965939
1013
async_write - False
1014
async_read - False
My testing code is here:
https://github.com/yeralexey/Vedischeck
Thanks for any help.
Hi,
What format is the database file when vedis generates it on disk?
which software should I use to open it properly?
From: coleifer/vedis-python#22
An error has been made in the SyOSUtilRandomSeed, when running for a long time, leads to a leak of file descriptors, since the /dev/urandom descriptor is not closed anywhere
The patch:
diff --git a/src/vedis.c b/src/vedis.c
index 844cfbe..c6336b0 100644
--- a/src/vedis.c
+++ b/src/vedis.c
@@ -12924,6 +12924,7 @@ static sxi32 SyOSUtilRandomSeed(void *pBuf, sxu32 nLen, void *pUnused)
fd = open("/dev/urandom", O_RDONLY);
if (fd >= 0 ){
if( read(fd, zBuf, nLen) > 0 ){
+ close(fd);
return SXRET_OK;
}
/* FALL THRU */
Hi guys,
I wonder does vedis support cluster deployment?
In telecommunication, we seek for low latency and high architectural flexibility. I want my service to be stateless, so I choose redis at first but figure out that the several hundred microseconds networking latency between redis client and server is unsatisfactory. Then I turn to this embedded database.
So if I'm to build a cluster, do I have to share memory/files and care about details like data consistency between nodes? Is there some official guide on vedis cluster?
Thanks!
Hi,
Thanks for your great package.
I've try to run "vedis_shell" on my OPENWRT (MIPS/Big Endian) Arduino Yun but found some strange behaviours.
Steps as following:
Binaries are built with uClibc library.
Have you come across this problem?
Thanks
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.