anywhichway / reasondb

A 100% JavaScript object database: SQL like syntax, full-text search, auto object sync, swapable persistence engines, asynchronous cursors, 30+ built-in plus in-line fat arrow predicates, predicate extensibility,joins, nested matching.

reasondb

A multi model 100% JavaScript database supporting:

1. key/values, graphs, documents

2. Industry standard Storage API (except ReasonDB is asynchronous), or a graph API similar to GunDB, or SQL like syntax.

3. Can be used with almost any key/value backing store.

4. joins

5. standard array like functions for iterating over results, e.g. map, forEach, etc.

6. inline functions or over 30 pre-defined predicates, e.g. $eq, $gt, $isIPAddress

7. automatic data expiration based on dates or durations

8. full text indexing

Full documentation is available at https://anywhichway.github.io/reasondb

Updates (reverse chronological order)

2019-06-10 v1.0.7b Switched to SSPL license.

2019-01-22 v1.0.6b Improved full-text tokenizer. Fixed issue where graph was not flushing to storage for putItem.

2019-01-21 v1.0.5b Enhanced JOQULAR to support $return.

2019-01-19 v1.0.4b Corrected years in this update list. Enhanced JOQULAR to support projections and validation.

2019-01-13 v1.0.3b Basic replication now working.

2019-01-12 v1.0.2b Documentation enhancements. Added update to SQL like commands and statistical methods to cursors.

2019-01-03 v1.0.1b Merged AnyWhichWay into ReasonDB. NOTE: There are substantial API and functonality changes. Databases and applications developed with v0.. are not compatible.

2017-05-23 v0.3.2 Improved fastForEach

2017-02-12 v0.3.1 Added full-text indexing and search. See .fullTextKeys and $search in documentation. Refactored 20 un-necessary nested Promises. Added fastForEach. Added minified version of src/index.js for Chrome and node v7.x users.

2017-02-10 v0.3.0 Code base made more modular with respect to server side drivers. Drivers must now be loaded separately. See documentation above. Added a deferKeys option to classes that prevents a full index being created on a property but still allows the property to be queried using JOQULAR. Addressed a scoping issue with JSONBlockStore that prevented it from restoring classes properly in some situations.

2017-02-10 v0.2.10 Fixed Issue 19.

2017-01-21 v0.2.9 Added limit(count) and page(offset) to select added .page(page,size) to Cursor instances which returns reduced size cursor.

2017-01-17 v0.2.8 Fixed Issue 15, changes to intersection in v2.6 had been copied from a non-strict codebase and broke during babelify with no errors during compile.

2017-01-11 v0.2.7 Fixed Issue 13 where updates were being saved to the Object index when a constructor could not be found for the classes of objects being updated.

2016-12-23 v0.2.6 Added more performant intersection. Added issues folder under test for managing resolution to issue reports.

2016-12-01 v0.2.5 Added function queries. where clause can now be a function that returns an array of rows of objects and ignores the normal look-up process, i.e. array of arrays. ReasonDB continues to handle projections and statistical sampling or row count limits.

2016-11-29 v0.2.4 Added skipKeys as a class configuration option to prevent indexing of specified properties.

2016-11-27 v0.2.3 Added saveIndexAsync:true as a database startup option. Saves indexes only during idle time, tripling or quadrupling insert speed for locally hosted databases.

2016-11-25 v0.2.2 Introduced the use of const producing substantial performance improvements. Tested against local copy of Redis.

2016-11-24 v0.2.1 Updated examples to use /lib/uuid.js since the update to v3.0.0 of uuid made uuid not directly browser loadable. Documentation updates.

2016-11-23 v0.1.9 Documentation updates, code quality improvements.

2016-11-23 v0.1.8 Documentation updates, code quality improvements updated uuid package to v3.0.0.

2016-11-20 v0.1.7 Documentation updates.

2016-11-20 v0.1.6 Added JSONBlockStore.

2016-11-15 v0.1.5 Added performance tests in examples/load directory.

2016-11-13 v0.1.4 Further optimizations to ensure action sequencing is correct when using a remote datastore. This fixed issues with Redis. Simplified coding to add new persistence stores.

2016-11-02 v0.1.3 Optimizations to help ensure all the actions required to support one change to a data element are complete prior to initiating another on the same element. This involved replacing Promise calls is functions with a passed reference to the resolver for a top level Promise. Added support for multiple arguments for insert, delete. Added LevelUpStore. Identified an fixed a couple of edge case Promises that contained this references. Corrected a join issue that resulted in right sides that were unrestricted for Redis and Memcache.

2016-10-31 v0.1.2 1.1 was pushed with incorrect test case config.

2016-10-31 v0.1.1 Added support for IronCache, Redis, and Memcached. Improved documentation.

2016-10-30 v0.1.0 Added first, random, and sample to select. Made cursor calls to forEach, every, some, get asynchronous. See documentation for rationale. Deprecated shared indexes, they did not scale well under volume and made working with localStorage somewhat obscure. This resulted in dropping the as clause for insert.

2016-10-28 v0.0.6 Added Update statement. Enhanced database to take a start-up flag that makes activating objects for automatic database and index update optional. Repaired 'delete' which broke when cursor.count was changed to a function. Added documentation. Published to npm.

2016-10-27 v0.0.5 Added documentation. Repaired 'when' which broke when cursor.count was changed to a function. Published to npm.

2016-10-26 v0.0.4 Added documentation. Changed count on Cursor instances to a function and added maxCount as a data member. Not published to npm.

2016-10-25 v0.0.3 First npm publication.

Prior to being re-named, ReasonDB existed as the first auto-synchronizing in-memory JavaScript object database JOQULAR, originally published in April of 2015.

License

This software is provided as-is under the MIT license.

reasondb's Issues

Schema Based Validations

Schema based validations are being considered. This feature would support the optional use of schema to support object validation, default values, and index specification.

This topic is open for questions, comments, and design thoughts. Give it a thumbs-up if it is important to you.

Large data collections cause garbage collection errors

Using: JSONBlockStore
Number of records: 20,000
Node: 6.9 and 7.5 (--harmony-async-await and using /src/index.js)

When inserting large recordsets into a single collection I get the following error. I have tried inserting in batch, stopping and starting the app between batches, but I never get above 20k.

  544228 ms: Mark-sweep 1878.0 (2037.1) -> 1877.6 (2037.1) MB, 3625.3 / 0.0 ms [allocation failure] [GC in old space requested].
  547861 ms: Mark-sweep 1877.6 (2037.1) -> 1877.5 (2037.1) MB, 3632.8 / 0.0 ms [allocation failure] [GC in old space requested].
  551578 ms: Mark-sweep 1877.5 (2037.1) -> 1881.4 (2021.1) MB, 3716.4 / 0.0 ms [last resort gc].
  555225 ms: Mark-sweep 1881.4 (2021.1) -> 1885.2 (2021.1) MB, 3646.2 / 0.0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x27fbaddcfb51 <JS Object>
    1: PromiseEnqueue(aka PromiseEnqueue) [native promise.js:~98] [pc=0x2c5278ddd396] (this=0x27fbadd04381 <undefined>,y=0xd6b8f552bd1 <JS Array[18]>,C=0x3f7ffe835d41 <JS Array[2]>,x=1)
    2: then [native promise.js:~215] [pc=0x2c5278d07605] (this=0x18998b0e4c41 <a Promise with map 0x30c4b83231b1>,z=0x3f7ffe835ab1 <JS Function s (SharedFunctionInfo 0x1b026b524119)>,A=0x3f7ffe835af9 <JS Functio...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x10cf7dc [node]
 3: v8::Utils::ReportApiFailure(char const*, char const*) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::CopyFixedArrayAndGrow(v8::internal::Handle<v8::internal::FixedArray>, int, v8::internal::PretenureFlag) [node]
 6: v8::internal::Isolate::EnqueueMicrotask(v8::internal::Handle<v8::internal::Object>) [node]
 7: v8::internal::Runtime_EnqueueMicrotask(int, v8::internal::Object**, v8::internal::Isolate*) [node]
 8: 0x2c52754092a7
Aborted (core dumped)

Manual referencing not resolving

I created a reference within a document body, something like:

Collection: Parent

{
  "name": "Parent Document",
  "friend": {
    "@key": "RelatedDocument@b0f16ff1-15aa-4187-b501-418678463fe8"
  }
}

The intention is to relate one document to another. It works if I reference an Object but not a custom type. Error is thrown here with an error similar to

"err":{"message":"RelatedDocument is not defined"

There is a RelatedDocument db, and it's populated with a document with that @key

Is there a better way to insert a related document too?

Querying by $eq or $neq no longer working

In version 2.5, the following query worked:

{$p: {name: {$neq: null}}}

I believe changes made in 2.6 (not released to npm) broken this functionality

ACID and Transactional Improvements

General improvements to transactional capability and the ACID nature of ReasonDB are being considered.

This topic is open for questions, comments, and design thoughts. Give it a thumbs-up if it is important to you.

Statistical Inference

Statistical inference capabilities are being considered. This a feature set that would include the abilities to:

1. Infer missing values based on other values using Bayesian stattistics or information entropy and an ID3 like algorithm.
2. Retrieve the frequency of an object with given values existing in the database (which may or may not represent its probability of occurrence beyond the bounds of the database).

This topic is open for questions, comments, and design thoughts. Give it a thumbs-up if it is important to you.

Firebase adapter?

Is there a Firebase adapter?
This seems misleading: https://github.com/anywhichway/reasondb/blob/master/package.json#L22

Handle empty arrays

If an object is inserted and the value of a key is [] the insert never completes. No errors are thrown either.

Example:

{
  name: "John",
  friends: []
}

Updates aren't retained after app restart

When using JSONBlockStore on the server, with clear=false, activate=true

Steps to reproduce:

• Launch app
• Insert document
• Update document
• Observe that document is updated (1)
• Relaunch app
• Get document (looks fine)
• Update document
• Observe that document is updated (2)
• Relaunch app
• Get document
• Observe that document has reverted to the last version before the first App relaunch

Multiple instances communicating with one central server?

Is this possible?

Ability to paginate and limit

Currently there doesn't seem to be a good way to paginate or limit the results. An ideal solution would be something like:

.select()
  .from({$p: Person})
  .where({$p: {name: "Joe"}})
  .limit(5)  // Result limit
  .offset(1) // Page

More Examples

Use this thread to tell us what you would like in terms of examples. Give a thumbs-up to example requests posted by others if you also desire the example.

Client to Server Synchronization

The addition of browser/server synchronization is under consideration. The feature would support automatic updating of objects on a server if updated by a client after retrieval from a server. The client would typically be a browser. The anticipated implementation would use sockets although a URL based http mechanism would be feasible.

This topic is open for questions, comments, and design thoughts. Give it a thumbs-up if it is important to you.

Passive Objects

Adding support for passive object inserts is being considered, i.e. inserts of objects that do not automatically cause a database update if they are changed after insert.

This topic is open for discussion and design input. Give it a thumbs-up if it is important to you.

Full Text Indexing

The addition of full text indexing and search is being considered.

This topic is open for questions, comments, and design thoughts. Give it a thumbs-up if it is important to you.

Not a lot of updates recently...

Is active development on this project still happening?

Is ReasonDB dead?

@anywhichway What is the status of this project?

Selective field results

To retrieve only the fields you need, it could be useful to extend the .select() method to limit the returned fields. Something like:

.select({'name': 1, 'age': 1}).from({$p: Person})

also accepting arrays:

.select(['name', 'age']).from({$p: Person})

No way to search unordered array properties

First, I like what this project has to offer, I hope you will continue working on it :)

I was playing around, creating my own store adapter for the Database that I want to use, and that seems to be good to go. But, I wanted to test some of the capabilities and ran into some issues with querying arrays. Basically, there is no real way that I could find to query objects based on the contents of an array property. See example below..

Example:

class Person {
  constructor(name, friends) {
    this.name = name;
    this.friends = friends;
  }
}

const me = new Person('Matt', ['Jay', 'Logan', 'Sky']);

db.insert(me).exec().then(() => {
  db.select()
    .from({ $p: Person })
    // This doesn't work at all:
    .where({ $p: { friends: { $contains: 'Logan' } } })
    // This works, but I have to know the index :(
    .where({ $p: { friends: { 1: 'Logan' } } })
    // This doesn't work bc the `value` is not the actual array, it is a reference to the array..
    .where({ $p: { friends: { $: (value) => value.includes('Logan') } } })
    .exec()
    .then((cursor) => {
      cursor.forEach((row) => { console.log("Person", JSON.stringify(row[0])); });
    })
})

If you have a solution, that would be great, but otherwise I wanted to bring it to your attention, that this seems to be a real limitation.

Enhanced Joins

Enhancing joins to support more than two classes is being considered.

This topic is open for questions, comments, and design thoughts. Give it a thumbs-up if it is important to you.

Subscribable Queries

The addition of subscribable queries is being considered. This feature would push updates to clients if the result sets for a query made by the client against a server were to change as a result of other activity on the server.

This topic is open for questions, comments, and design thoughts. Give it a thumbs-up if it is important to you.