symisc / vedis Goto Github PK

I am trying to include vedis in an esp32 project using esspesif's esp-idf and it throws several errors and won't compile.
Is this even possible.
I have defined
#define OS_OTHER
../main/vedis.c: In function 'vedisTokenizeInput':
../main/vedis.c:3076:73: error: parameter 'pUserData' set but not used [-Werror=unused-but-set-parameter]
static sxi32 vedisTokenizeInput(SyStream *pStream,SyToken *pToken,void *pUserData,void *pCtxData)
~~~~~~^~~~~~~~~
../main/vedis.c:3076:89: error: parameter 'pCtxData' set but not used [-Werror=unused-but-set-parameter]
static sxi32 vedisTokenizeInput(SyStream *pStream,SyToken *pToken,void *pUserData,void *pCtxData)
~~~~~~^~~~~~~~
../main/vedis.c: In function 'MemHashInit':
../main/vedis.c:10197:55: error: parameter 'iPageSize' set but not used [-Werror=unused-but-set-parameter]
static int MemHashInit(vedis_kv_engine *pKvEngine,int iPageSize)
~~~~^~~~~~~~~
then lots of
'if' clause does not guard...
parameter 'pLex' set but not used
variable 'zExtra' set but not used
which shouldn't be a problem but also

but then

../main/vedis.c:1848:33: warning: 'vedisExportBuiltinVfs' used but never defined
VEDIS_PRIVATE const vedis_vfs * vedisExportBuiltinVfs(void);

Hi.
I try use vedis but got some wrongs :(

>>>LPUSH name a1 a2 a3
[3]
>>>LINDEX name 0
[a1]
>>>LPOP name
[a1]
>>>LINDEX name 0
[nil]    //Here should be "a2"
>>>LINDEX name 0
[nil]    //Here should be "a2"
>>>LINDEX name 0
[nil]    //Here should be "a2"
>>>LLEN name
[2]

the command "LINDEX" must be wrong after the “LPOP” .

Hi,

Thanks for your great package.
I've try to run "vedis_shell" on my OPENWRT (MIPS/Big Endian) Arduino Yun but found some strange behaviours.

Steps as following:

1. vedis_shell /dev/shm/db
2. HSET hello world 1
3. HSET hello today 2
4. exit
5. vedis_shell /dev/shm/db
6. HGET hello world -> "null"

1. vedis_shell /dev/shm/db2
2. LPUSH hello world
3. LPUSH hello today
4. exit
5. vedis_shell /dev/shm/db2
6. LINDEX hello 0 -> "null"

Binaries are built with uClibc library.
Have you come across this problem？

Thanks

vedis contains all C source code in one file. it is not easy to read and understand.
May I have the vedis separate files ? I need it to study how it work
thank you

From: coleifer/vedis-python#22

An error has been made in the SyOSUtilRandomSeed, when running for a long time, leads to a leak of file descriptors, since the /dev/urandom descriptor is not closed anywhere

The patch:

diff --git a/src/vedis.c b/src/vedis.c
index 844cfbe..c6336b0 100644
--- a/src/vedis.c
+++ b/src/vedis.c
@@ -12924,6 +12924,7 @@ static sxi32 SyOSUtilRandomSeed(void *pBuf, sxu32 nLen, void *pUnused)
        fd = open("/dev/urandom", O_RDONLY);
        if (fd >= 0 ){
                if( read(fd, zBuf, nLen) > 0 ){
+                 close(fd);
                        return SXRET_OK;
                }
                /* FALL THRU */

Hello, friends

Please tell me, maybe I’m blind, but I don’t see a method for getting all the keys. Is there any analogue of the keys command in redis?

Sometimes vedis fails to store key. vedis_kv_store returns VEDIS_OK, but the successive vedis_kv_fetch returns VEDIS_NOTFOUND for the same key.

It doesn't occur for in-memory store. Also I can't reproduce it without vedis_kv_delete calls.

Here is the test case. Usually it fails within 1 minute with "Unable to fetch: -6".

#include "vedis.h"
#include <stdlib.h>
#include <time.h>
#include <iostream>

// generate random bytes of variable (important!) length
char *randomBytes(int* len)
{
    *len = rand() % 4 + 1;
    char *res = new char[*len];

    for (int i = 0; i < *len; i++)
    {
        res[i] = rand() % 256;
    }

    return res;
}

int main()
{
    vedis_lib_init();
    vedis *store;
    int res = vedis_open(&store, "vedis.db");
    if (res != VEDIS_OK)
    {
        std::cout << "Error opening vedis:" << res << std::endl;
        return 0;
    }

    srand(time(NULL));

    char buffer[255];
    vedis_int64 inout;

    int block = 0;
    int count = 0;

    while(1)
    {
        int keyLen;
        int valLen;
        char *key = randomBytes(&keyLen);
        char *val = randomBytes(&valLen);

        // insert random key
        res = vedis_kv_store(store, key, keyLen, val, valLen);
        if (res != VEDIS_OK)
        {
            std::cout << "Unable to store: " << res << std::endl;
            return 0;
        }

        // verify it was stored correctly
        inout = 255;
        res = vedis_kv_fetch(store, key, keyLen, &buffer[0], &inout);
        if (res != VEDIS_OK)
        {
            std::cout << "Unable to fetch: " << res << std::endl;
            return 0;
        }

        // generate new random key and delete if it exists
        delete []key;
        key = randomBytes(&keyLen);

        inout = 255;
        res = vedis_kv_fetch(store, key, keyLen, &buffer[0], &inout);
        if (res == VEDIS_OK)
        {
            res = vedis_kv_delete(store, key, keyLen);
            if (res != VEDIS_OK)
            {
                std::cout << "Unable to delete: " << res << std::endl;
                return 0;
            }
        }

        delete []key;
        delete []val;

        count++;
        if (count > 10000)
        {
            block++;
            count = 0;
            std::cout << block << " 10K iterations" << std::endl;
        }
    }
    return 0;
}

Hi guys,
I wonder does vedis support cluster deployment?
In telecommunication, we seek for low latency and high architectural flexibility. I want my service to be stateless, so I choose redis at first but figure out that the several hundred microseconds networking latency between redis client and server is unsatisfactory. Then I turn to this embedded database.
So if I'm to build a cluster, do I have to share memory/files and care about details like data consistency between nodes? Is there some official guide on vedis cluster?
Thanks!

Hi,
What format is the database file when vedis generates it on disk?
which software should I use to open it properly?

The vedis.c file in github is incomplete (200kb vs 700kb on the website) and didn't compile for me.

Also, would you be interested [to implement / if I implemented] a dbsize command which returns the number of bytes used by the journal (in memory)? I know there's an alternative through xAlloc / xFree in the allocators but it's harder to segment. This would help a lot to in cleaning up and avoiding out of memory errors.

Thank you for this great library

I discovered an issue in the vedis kv store due to the handling of defragmenting slave pages. The defragment function only uses the cell array on the current page, rather than the master page. The fix is to replace the final line in this:

static int lhPageDefragment(lhpage *pPage) { lhash_kv_engine *pEngine = pPage->pHash; unsigned char *zTmp,*zPtr,*zEnd,*zPayload; lhcell *pCell; /* Get a temporary page from the pager. This opertaion never fail */ zTmp = pEngine->pIo->xTmpPage(pEngine->pIo->pHandle); /* Move the target cells to the begining */ pCell = pPage->pMaster->pList; /* replaced with pPage->pMaster */

I have attached an example which will demonstrate corruption of the kv store when performing the sequence of operations in the test file.
Broken-vedis.zip

Hi, I like this project, but I need command LRANGE, I didn't find it..

Hi. Resieving this traceback:

----------------------------------
2022-12-21 12:16:48.042288
Traceback (most recent call last):
  File "/home/vedischeck/dbworker.py", line 30, in write_user_attribute
    db[f'{user_id}{attribute}'] = data
  File "vedis.pyx", line 383, in vedis.Vedis.__setitem__
  File "vedis.pyx", line 311, in vedis.Vedis.store
  File "vedis.pyx", line 425, in vedis.Vedis.check_call
OSError: b'IO error while opening journal file: /home/vedischeck/main.vdb_vedis_journal\n'

----------------------------------
2022-12-21 12:16:48.061854
Traceback (most recent call last):
  File "/home/vedischeck/dbworker.py", line 30, in write_user_attribute
    db[f'{user_id}{attribute}'] = data
  File "vedis.pyx", line 383, in vedis.Vedis.__setitem__
  File "vedis.pyx", line 311, in vedis.Vedis.store
  File "vedis.pyx", line 425, in vedis.Vedis.check_call
OSError: b'IO error while opening the target database file: /home/vedischeck/main.vdb\n'
(venv) root@vm3616877:/home/vedischeck#

Using Vedis as simple one-file memory-cache in some python projects.
At first i tried to .queue my actions, also tried asyncio Semaphore, to prevent concurrent access. But then i noticed, that trouble ouccurs only on VPS, on my home laptop everything works perfect, and concurrent acces is pergectly proceeded by Vedis itself....

And trouble occurs exatly on 1010 th iteration all the time...

1009
async_write - True
1010
async_read - None
1011
async_write - False
1012
async_read - 965939
1013
async_write - False
1014
async_read - False

My testing code is here:

https://github.com/yeralexey/Vedischeck

Thanks for any help.