Coder Social home page Coder Social logo

akashbabu / redis-json Goto Github PK

View Code? Open in Web Editor NEW
66.0 7.0 17.0 885 KB

A wrapper library to store JSON Objects in redis-hashsets and retrieve it back as JSON objects

License: MIT License

JavaScript 2.77% TypeScript 97.23%
node nodejs redis-key json-objects redis-json jsoncache

redis-json's Introduction

redis-json npm version Build Status Coverage Status Maintainability

Nodejs library to store/retrieve JSON Objects in RedisDB without loosing type information, i.e. WYSIWYG (What You Store Is What You Get)

Description

Every time set is called JSON object is flattened(embeded objects are converted to path keys) and then stored in Redis(just like a normal hashset), on get the hashset is unflattened and converted back to the original JSON object(with the same types as was in the original object).

What's new in v6.0.0?

  • In response to issue: #24, we now replace the array in the cache when array is found in set object.

If you are on V5 then please check this Migration Guide to V6

API

Please visit this page for detailed API documentation.

Usage

Simple

import Redis from 'ioredis';
import JSONCache from 'redis-json';

const redis = new Redis() as any;

const user = {
  name: 'redis-json',
  age: 25,
  address: {
    doorNo: '12B',
    locality: 'pentagon',
    pincode: 123456
  },
  cars: ['BMW 520i', 'Audo A8']
}

const jsonCache = new JSONCache<typeof user>(redis, {prefix: 'cache:'});


await jsonCache.set('123', user)

await jsonCache.get('123')
// output
// {
//   name: 'redis-json',
//   age: 25,
//   address: {
//     doorNo: '12B',
//     locality: 'pentagon',
//     pincode: 123456
//   },
//   cars: ['BMW 520i', 'Audo A8']
// }

await jsonCache.set('123', {gender: 'male'})
await jsonCache.get('123')
// output
// {
//   name: 'redis-json',
//   age: 25,
//   address: {
//     doorNo: '12B',
//     locality: 'pentagon',
//     pincode: 123456
//   },
//   cars: ['BMW 520i', 'Audo A8']
//   gender: 'male'
// }

await jsonCache.get('123', 'name', 'age');
// output
// {
//   name: 'redis-json',
//   age: 25,
// }

await jsonCache.get('123', 'name', 'address.doorNo');
// {
//   name: 'redis-json',
//   address: {
//     doorNo: '12B'
//   }
// }

await jsonCache.clearAll();

await jsonCache.get('123');
// undefined


await jsonCache.incr('123', {age: 1}) // increments age by 1

With custom stringifier and parser:

const jsonCache = new JSONCache(redis, {
  stringifier: {
    Date: (val: Date) => val.toISOString()
  },
  parser: {
    Date: (str: string) => new Date(str)
  }
})

const date = new Date()
await jsonCache.set('test', {
  date: date
})

// Redis hashset
> hgetall jc:test /// data
1) "date"
2) "2020-05-17T14:41:45.861Z"
> hgetall jc:test_t /// type info
1) "date"
2) "Date"


const result = await jsonCache.get('test')
result.date == date /// true

With transactions:

const transaction = redisClient.multi();

transaction
  .set('name', 'foo')
  .set('bar', 'baz')

await jsonCache.set('test', {name: 'testing'}, {transaction})
await jsonCache.del('test1', {transaction})
await jsonCache.rewrite('test2', {name: 'testing', age: 25}, {transaction})

transaction
  .exec(function(err, replies) => {
    /// your logic here after
  })

Please note that only set(), rewrite(), del() & incr() supports transaction, where as get() & clearAll() do NOT support transaction because we process those results before returning it to the calling function. Moreover there is no real usecase in supporting transaction in get() & clearAll() methods!

Changelogs

Please refer to this page

Coverage Report

npm run coverage

Contributions

This is open-source, which makes it obvious for any PRs, but I would request you to add necessary test-cases for the same.

Pre-requisites:

Run your redis-server and then point the same client to the same. An easier way to start redis-server, provided you've already installed docker (else visit this page) is by running this command:

docker run --rm -it --name redis -p 6379:6379 redis

We follow TDD approach, so write your test cases first and then run the same paralelly during development by running the following command:

npm run test:dev

LICENCE

MIT License

redis-json's People

Contributors

akashbabu avatar akashbabuna avatar akashpe avatar luxcium avatar omgimalexis avatar subasshrestha avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

redis-json's Issues

Upgrade redis to 4+

Unable to use if running on Redis 4+ as HMSET is removed in version 4.
Any plans for upgrading?

Support for numerical increments

Hi there,

This is initially a question. Is there any way to increment an integer without fetching the current value and saving the incremented value?

Based on my limited understanding of REDIS, isn't this supported natively by redis with INCR and HINCRBY?

jsonCache.set with "expiry"

Let a condition where we want to store the user's session, and the session is valid for only 1 day.
For now, there is no way to expire it from Redis.

Multi / Exec capability

Hi: first of all, I think this is a great library!!!

The scenario that I'm facing is that right after storing an object into Redis, I need to add the key into a set as well, and both commands should happen within the same transaction (Multi / Exec). I manually execute the MULTI command from ioredis before calling the redis-json commands, and I'd like to have the option for redis-json to not execute the EXEC command internally, so I can combine more commands into a single transaction.

Thanks,

Franco

Availability in Java?

Is this library is available in JAVA? so that both NODE and JAVA developers can use the same library.

Cannot set property 'state' of null

Hi, I'm getting an error thrown in your library.

I was hoping you could add a check for this with a more descriptive error being thrown?

TypeError: Cannot set property 'state' of null
  File "/usr/local/bin/mothership/node_modules/redis-json/lib/jsonCache.js", line 329, col 26, in Flattener.scaffoldStructure
    tree[k1] = val;
  File "/usr/local/bin/mothership/node_modules/redis-json/lib/jsonCache.js", line 278, col 18, in null.<anonymous>
    this.scaffoldStructure(result, splittedKeys, val);
  ?, in Array.some
  File "/usr/local/bin/mothership/node_modules/redis-json/lib/jsonCache.js", line 267, col 35, in Flattener.unflatten
    Object.entries(typedData).some(([key, val]) => {
  File "/usr/local/bin/mothership/node_modules/redis-json/lib/jsonCache.js", line 479, col 35, in JSONCache.<anonymous>
    return this.flattener.unflatten(result);
  ?, in Generator.next
  File "/usr/local/bin/mothership/node_modules/redis-json/lib/jsonCache.js", line 23, col 24, in fulfilled
    step(generator.next(value));
  ?, in runMicrotasks
  File "internal/process/task_queues.js", line 97, col 5, in processTicksAndRejections

Empty array is stored as empty string

a = { category: '', title: '', description: [], authToken: '' }
jsonCache.set(userId, userQuestion, options)

On redis CLI

HGETALL "Key:XXXXXXXXX"
1) "category"
2) "1"
3) "title"
4) ""
5) "description"
6) ""
7) "authToken"
8) ""

How can I make sure, description is stored as an empty array?

Duplicate value

Hi team

My bug is really strange :)

I use Redis-json to store large dataset (api cache) for my ecommerce middle office
I call api wich return array of data, and i store it on redis after conversion in object
with this function

obj = obj.reduce((r, o) => ((r[o.id] = o), r), []); obj = Object.assign({}, obj); await jsonCache.set(keyCache, obj);

I use a conversion before insertion in order to edit specific data with there ID
I this case I use :

var object = {}; object[key] = obj; await jsonCache.set(keyCache, object)

In my test, with few data it's work !
but, ... with production (8MO json)
there is a duplicate when I retrieve all data !
But if I use

var text = await jsonCache.get(keyCache, key);;

edited value is well in place

4 hours of research... need your help !

Unable to delete by key

Hi everyone,

I have a problem with my project, I can't delete the saved value by key.

redis.del (key)

// or
jsonCache.del (key)

Please help me

Type of get with no field arguments

Hi there, great library btw. Just a quick question about the get type signature. Should it have at least two definitions where, if no field arguments are provided, then the return type is no longer Partial<T> but just T?

interface IJSONCache<T> {
  // ...
  get(key: string): Promise<T | undefined>;
  get(key: string, ...fields: string[]): Promise<Partial<T> | undefined>;
  // ...
}

redis cluster support

This doesn't seem to support redis clusters. Would you be able to add support for clusters?

Have a problem when try to import JSONCache using Typescript

package.json

"typescript": "^4.6.3"
"ioredis": "^5.0.4"
"redis-json": "^6.0.3"

Code Section:

const redis = new Redis(Configuration.redisConnStr)

const jsonCache = new JSONCache(redis)

Error:

const jsonCache = new redis_json_1.default(redis);
                  ^

TypeError: redis_json_1.default is not a constructor

TypeError: redis_json_1.default is not a constructor

Hi, recently I installed redis-json package but when I try to run my code that compiled with typescript it throws following error:

const redisJson = new redis_json_1.default(redis);
                  ^

TypeError: redis_json_1.default is not a constructor

Here's my code, all I done is just installing redis and redis-json:

import {createClient} from 'redis';
import JSONCache from 'redis-json';

const redis = createClient();
const redisJson = new JSONCache(redis);

also this is my tsconfig.json as well:

{
  "compilerOptions": {
      "module": "commonjs",
      "resolveJsonModule": true,
      "target": "es6",
      "outDir": "dist"
  },
  "include": [
      "./src/**/*"
  ],
  "exclude": [
      "node_modules"
  ]
}

I try to enable esModuleInterop but other packages get error.
I'm not sure is this an issue with redis-json package or typescript, but I hope I can get some help here. thank you.
also my Node.js version is 14.17.3. and I'm using Typescript version 4.3.5

DeprecationWarning: Calling promisify on a function that returns a Promise is likely a mistake.

Hi,

We are using ioredis 5.0.4 which already returns Promise for all the operations.. So if we use redis-json to get/set JSON from Redis, it prints the following message to the console:

(node:13167) [DEP0174] DeprecationWarning: Calling promisify on a function that returns a Promise is likely a mistake.

I believe that's because you've wrapped the functions with promisify.

hmset: promisify(redisClient.hmset).bind(redisClient),
hmget: promisify(redisClient.hmget).bind(redisClient),
hgetall: promisify(redisClient.hgetall).bind(redisClient),
expire: promisify(redisClient.expire).bind(redisClient),
del: promisify(redisClient.del).bind(redisClient),
scan: promisify(redisClient.scan).bind(redisClient),
hincrbyfloat: promisify(redisClient.hincrbyfloat).bind(redisClient),

Any chance you can remove that and upgrade to ioredis v5? Or patch it so that you don't call promisify if ioredis ^5.0 ? Thanks!

Data is stored from client and viewable from redis-cli but returns undefined in console.log

I've copied the sample code with a few adjustments, and am able to see the data from SET getting into redis from redis-cli, however I cannot get the data back into node to print to console.

const redis = require('redis');
const JSONCache = require('redis-json');
const redis_port = process.env.REDIS_PORT || 6379

const client = redis.createClient({host: "127.0.0.1", port: redis_port})
const jsonCache = new JSONCache(client, {prefix: 'cache:'});

const user = {
    name: "kenta",
    age: 22,
    address: {
        street: "2840 Bay Colony Lane",
        city: "Lexington",
        state: "KY"
    },
    cars: ['300zx', 'WD21']
}

async function setCache(cache, object) {
    await cache.set('0', object)
    let response = await cache.get('0')
    console.log(response)
}

setCache(jsonCache, user)

However, when I run this the response is always undefined

Yet in redis-cli:

127.0.0.1:6379> hgetall cache:0
 1) "name"
 2) "kenta"
 3) "age"
 4) "22"
 5) "address.street"
 6) "ABCD 1234"
 7) "address.city"
 8) "NYC"
 9) "address.state"
10) "NY"
11) "cars.0"
12) "300zx"
13) "cars.1"
14) "WD21"
15) "__jc_root__"
16) "0"

Empty array is returned as string '[ ]'

So sometimes when I store an empty array it's being returned as string '[ ]'. It doesnt happen very often but it occures.

simple case

await RedisJson.set('123', [])
const x = await RedisJson.get('123')

then sometimes x equals '[ ]'

version
"redis-json": "^6.0.1",

Possible to cache.set an array of JSON objects and get back an array?

As is, calling cache.set on an array of JSON objects like this:

[
  {
    id: 1,
    name: "John"
    age: 22
  },
  {
    id: 2,
    name: "James"
    age: 24
  }
]

Works, however when you cache.get it returns like this:

{
  '0': {
    id: '1',
    name: 'John',
    age: '22'
  },
  '1': {
    id: '2',
    name: 'James'
    age: '24'
  }
}

As is, I can revert this returned Object of objects by running something like this:

let result= await cache.get(key)
let array = []
for(let i = 0; i < Object.keys(reslt).length; i++){
    array.push(result[i])
}

I suppose it's not an issue but perhaps something that could be improved upon?

Thanks

Is it possible to update nested object?

Hi,

Please I would like to know if its possible to dynamically update a nested object?

===e.g===
const user = {
name: 'redis-json',
age: 25,
address: {},
friends: []
}

await jsonCache.set('123', user);

const user = await jsonCache.get('123');
// output
// {
// name: 'redis-json',
// age: 25,
// address: {},
// friends: []
// }

user.friends.push({name: 'tom'}, {name: 'jerry});
await jsonCache.set('123', ????); ===> what do I pass as the second argument? (user object of the updated property?

Also, can I dynamically add objects to the address property?
===e.g===
user.address[new-key] = new-value;
await jsonCache.set('123', ????); ===> what do I pass as the second argument? (user object of the updated property?

Question: Expire

Dumb question but what is the expire value based on ? minutes ?

While updating an existing key, the TTL value is lost

When using redis-json (v^3.2.0), we have observed few keys get a new entry in redis with appended _t without any TTL.
From my understanding it occurs when we are updating an existing key, I believe its by design.

But the problem is it doesn't copies the TTL value of the the original key, and the original key's TTL is also lost.

How to prevent this from happening ?

incorrect handling of removed array indexes and object properties

I've found a strage behavior of the .set() method working with arrays.

The strage and not documented behavior is that if an array becomes shorter than the last stored in redis, the last few keys are note deleted.

Test case:

const client = this.redisService.getClient();
const usersMap = new JSONCache<number[]>(client);
const arr1 = [1,2,3];
usersMap.set('test', arr1);

let arr2 = usersMap.get('test');

arr2  = arr2 .filter(el=> {
    return el % 2 == 1;
});

// arr2 = [ 1, 3 ]

usersMap.set('test', arr2);

const arr3 = usersMap.get('test');

// arr3 = [ 1, 3, 3 ]

This 'bug' happens because this library uses hmset redis command, that does only upsert but does not remove missing keys.

This library offerse the possibility to retrieve partial objects, so is not possibile to remove missing object props / array keys on set.

I think that a big disclaimer on this behavior should be added to the docs.

Would you accept a pull request that adds some new methods to allow a user to 'set and replace with delete missing props' objects in this cache in the stored objects are very little?

Thanks
Francesco

Option to set no prefix

Hi, I would like to have no prefix to be added to my keys. Giving no prefix option or giving an empty string in prefix results in a prefix of "jc:". Can you give an option to support no prefix?

This:
const jsonCache = new JSONCache(redis);
or
const jsonCache = new JSONCache(redis, { prefix: "" });
adds a prefix of "jc"

ttl

function to get ttl

_t type cache automatically

Hi,
Can you please help to remove the key which automatically gets created as _t", as it's taking unnecessary memory?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.