Implementing Redis datastore about hydrus

[Mec-iS]

Please make a comprehensive list of the tools you are thinking about using.

As far as I understood you want to use Redis 4.0 and add redisgraph.io module. There are many other ways of storing/querying a graph in Redis; for the sake of your application, it would be nice to cover the different possibilities to have a clearer picture. You can start with a naive implementation (like a prototype) about using simple tuples or Redis built-in data structures, and then raise the bar to understand how redisgraph internals work. You can create your own repository for researching purposes, running queries and tests on a local db instance. For this sake (in later times) it would be nice also to have some automation in creating a VM (Docker or Vagrant) and provisioning scripts to install the needed Redis version.
As much you can write down about these subjects, more possibilities your application is going to have.

Thanks for dedicating time to this task.

[Mec-iS]

@vddesai1998 it is not decided yet, depends from proposals and how the implementation goes on. We may have different layers implemented with Postgres and Redis.

@sandeepsajan0 you have already much material. Focus on writing a good proposal.

[Mec-iS]

there is no connection between Github issues and GSOC applications. Please read https://github.com/HTTP-APIs/hydrus/wiki/Starting-Material

[vddesai1871]

Are we going to use Redis as a 1st class database?

[sandeepsajan0]

++UPDATE++
Hey @Mec-iS, I have used Redis built-in data structure with redis-cli and also by creating a repo(a prototype) in which data is stored, which is getting from the web(simple html page) in Redis.
I have used string, sets, sorted sets, lists, and hashes(usually uses as the tuple). These all have its own requirements and properties and I conclude, hashes are the best option for representation of the object. And now, I am trying to implement graph(redis-graph) as data representation but I got an error at the time saving the graph and I also trying to fix it. I will update it soon.
+++
Provide suggestions to proceed ahead.

[sandeepsajan0]

hey @Mec-iS as I read about Redis in a blog that

These are the two primary reasons Redis sucks as a primary store:
-You have to be able to fit all your data in memory, and
-If your server fails between disk syncs you lose anything that was sitting in memory.

So, we should use Redis with existing database or we can use Redis as a primary database?

[vddesai1871]

@sandeepsajan0 I already have asked Lorenzo about it and he said we should include it in our proposal and then we'll decide

[Mec-iS]

@sandeepsajan0 Redis is completely safe at the same level of a SQL database if we take some strong design decisions since the beginning. Everything you heard can be avoided by setting Redis properly and provide replication and write-to-disk policies. Redis is designed to support natively these features. Every DBMS is a single point of failure if it is run in a single instance on a single machine. With a master-slave configuration Redis provides the same level of resilience of any other datastore/database.

As specified, there isn't anything set yet. We may decide to use Redis only as a cache in the beginning.

[EDIT] some concepts here

[Mec-iS]

This task has been moved to HTTP-APIs/hydra-python-agent#8