meteor-community-packages / ground-minimax

Minimax is a thin layer for Meteor providing EJSON.minify and EJSON.maxify

ground:minimax

This package adds object compression and decompression of objects.

This is a small package that adds:

• MiniMax.minify Compress object to array structure
• MiniMax.maxify Decompress to object
• MiniMax.stringify ~ Like a compressed version of EJSON.stringify
• MiniMax.parse ~ Like the decomressing version of EJSON.parse

Usage

In short: Schema and schema less documents are minified to an array format:

1. Array of keywords
2. Array of data schemas
3. Array of data

  // The default got false, true, null and undefined in dictionary
  var normalMiniMax = new MiniMax();

  // Add additional words to the dictionary, eg. if used on a database,
  // This will help MiniMax compress the data even more.
  // The order and combination is vital for when uncompressing the data
  var myMiniDB = new MiniMax({
    dictionary: ['createdAt', 'createdBy', 'UpdatedAt', 'UpdatedBy'],
    // progressive: false // Default true, puts values in dictionary
    // - if false only keys are added
  });

  // By default an instance of MiniMax is available using the default
  // dictionary
  MiniMax.minify(obj, [skipFunctions=false])
  MiniMax.maxify(obj)
  MiniMax.stringify(obj)
  MiniMax.parse(string)

How does it work? (example)

Our data object:

var data = {
  "5qSjMxCjkNF2SFBy6": {
  _id: "5qSjMxCjkNF2SFBy6",
  foo: "test"
},
  "rbieX9SbdGgfSWCd7": {
  _id: "rbieX9SbdGgfSWCd7",
  foo: "test",
  bar: "okay"
}
};

In EJSON.stringify = 136 chars

{"5qSjMxCjkNF2SFBy6":{"_id":"5qSjMxCjkNF2SFBy6","foo":"test"},
"rbieX9SbdGgfSWCd7":{"_id":"rbieX9SbdGgfSWCd7","foo":"test",
"bar":"okay"}} 

In EJSON.minify = 117 chars saved 14% space

[["5qSjMxCjkNF2SFBy6","_id","foo","rbieX9SbdGgfSWCd7","bar"], // Keywords
[0,[-1,2,4],[0,3]], // Schema
[2,[1,0,"test"],[1,3,"test","okay"]]] // Data

The data array it self in this example is only about 36 chars ~ 27% of the original EJSON.stringify - if both server and client have keywords and the schema only the data would have to be sent.

[2,[1,0,"test"],[1,3,"test","okay"]] 

  Keywords:
  [["5qSjMxCjkNF2SFBy6","_id","foo","rbieX9SbdGgfSWCd7","bar"],
  Schemas:
  [0,[-1,2,4],[0,3]],
  data:
  [2,[1,0,"test"],[1,3,"test","okay"]]] 

The keyword array contains key names

  [1,0,"test"]
  The "1" referes to the object schema [-1,2,4] - if pointing to
  schema 0 then the data is an array.
  {0, "test"}
  In the schema the "-1", the minus says that the value is a
  keyword reference and the 1 points to the keyword "_id"
  In the schema the "2" points to keyword "foo" and use the value
  in the data.
  The "4" in the schema isnt used - this is an extension of the
  schema to match the "bar" in the other object.
  [-1,2,4] -> [_id, foo, bar_]
  [-1,+2,+4] -> [0 = "5qSjMxCjkNF2SFBy6", "test"]
  -> {_id: "5qSjMxCjkNF2SFBy6", foo: "test"_}

Future

• Faster code
• Better compression

ground-minimax's Issues

Create test for functions in objects

We have to add tests for objects like:

var foo = function() {};
{
  foo: foo
}

// Do minify and the maxify and check if foo is foo...

Maxify does not recreate _id properly when it is ObjectID

I have a collection where the _ids are Mongo ObectID's and kept getting Error: Meteor does not currently support objects other than ObjectID as ids.

A colleague of mine and I located the error in minimax's method of maxifying _ids; they are recreated as regular Objects with _strattribute, which minimongo does not accept.

Our solution was adding a check to see if the document being maxified had an _id._str attribute and in those cases recreate the _id as Meteor.Collection.ObectID(_id._str)

I'll send a pull request shortly with our fix, but I thought I'd create an issue first

Tamper - Related project

Have you seen this project from NYTimes?

http://nytimes.github.io/tamper/

It says to achieve impressive compression when you pass the structure of data and the dictionaries for fields with difined range of values, very similar to this project.