qminer / qminer

Analytic platform for real-time large-scale streams containing structured and unstructured data.

QMiner

QMiner is an analytics platform for large-scale real-time streams containing structured and unstructured data. It is designed for scaling to millions of data points on high-end commodity hardware, providing efficient storage, retrieval and analytics mechanisms with real-time response.

Project homepage

Examples

• Twitter sentiment extraction
• Recommendation system

Prerequisites

• node.js v14.x, v12.x, v10.x and npm 5.3 or higher To test that your node.js version is correct, run node --version and npm --version. Not compatible with nodejs v0.10 or older.

Windows

• Visual C++ Redistributable Packages for Visual Studio 2015 download vcredist_x64.exe for node.js x64 or vcredist_x86.exe for node.js x86.

Install

To Install the qminer package run:

npm install qminer

Test. To test if the package was successfully installed run:

node -e "require('qminer'); console.log('OK')"

Compile from Source

If you wish to compile the package from source, please address the instructions.

Documentation

The package has a full documentation available online:

• JavaScript API
  • qm module
  • la module
  • analytics module
• C++ Documentation

Acknowledgments

QMiner is developed by Department of Artificial Intelligence at Jozef Stefan Institute, Quintelligence, Qlector and other contributors.

The authors would like to acknowledge funding from the European Union Seventh Framework Programme, under Grant Agreements 288342 (XLike), 611346 (XLime), 611875 (Symphony), 317534 (Sophocles), 318452 (Mobis), 600074 (NRG4Cast), 619437 (Sunseed), 632840 (FI-Impact) and 612329 (ProaSense).

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 636160-2 (Optimum).

qminer's Issues

Feature Space: filter function

Execution stopped: C++ exception thrown, if length of given vector is less than the index start of the given feature extractor.

var ftr = new qm.FeatureSpace(base, [
                           { type: "numeric", source: "FtrSpaceTest", field: "Value" },
                           { type: "categorical", source: "FtrSpaceTest", field: "Category", values: ["a", "b", "c"] },
                           { type: "categorical", source: "FtrSpaceTest", field: "Category", hashDimension: 2 },
                           { type: "multinomial", source: "FtrSpaceTest", field: "Categories", values: ["a", "b", "c", "q", "w", "e"] },
                           { type: "multinomial", source: "FtrSpaceTest", field: "Categories", hashDimension: 4 }
            ]);
var in_vec = new qm.la.Vector([1, 0, 0, 1]);
var out_vec = ftr.filter(in_vec, 3, false);

Store.js Exception

Execution stopped: (_Vals==-1)||(_Vals >= 0) in file ds.h, line 968.

Unit test in file store.js, that causes the exception:

it('should throw an exception if sample parameter is negative', function () {
     table.addMovie(table.movie);
     table.addMovie(table.movie2);

     assert.throws(function () {
          table.base.store("People").sample(-10);
     })
})

Feature Space: ftrVec and ftrSpVec functions

Throws a C++ exception, if the given record is of a different store of if record index is out of range.

base.createStore({
     "name": "Mobile",
     "fields": [
      { "name": "TeaMobile", "type": "float" },
      { "name": "AvtoMobile", "type": "string" },
      { "name": "PerpetumMobile", "type": "string_v" },
      ],
      "joins": [],
      "keys": []
      });
base.store('Mobile').add({ TeaMobile: 10, AvtoMobile: "car", PerpetumMobile: ["more", "cars"] });
var ftr = new qm.FeatureSpace(base, { type: "numeric", source: "FtrSpaceTest", field: "Value" });

ftr.ftrSpVec(base.store('Mobile')[0]); // throws exception

ftr.ftrVec(base.store('Mobile')[2]); // throws exception

Add option to value key index which lets us ignore character cases

e.g. EriK -> erik

get total memory usage for all stores and index

It would be good to have a method that would return the memory usage for individual stores and index. This should include the size of all hash tables, vectors and other data structures in a store. There is currently an implementation of a similar method but it only returns the size of the data in cache.

separate unit tests for LIBSVM parameter

Stream aggregate functions: onAdd

Unexpectively exits node when the onAdd function is not defined in the constructor.

// exits node
var aggr = new qm.StreamAggr(base, new function () {
                    var length = 0;
                    this.name = 'nameLength';
                });

two methods should be optimized using TChA (ChangeStrAll missing there)

TStr TXmlLx::GetUsAsciiStrFromXmlStr(const TStr& XmlStr){
TStr UsAsciiStr=XmlStr;
UsAsciiStr = UsAsciiStr.ChangeStrAll("è", "c");
UsAsciiStr = UsAsciiStr.ChangeStrAll("È", "C");
UsAsciiStr = UsAsciiStr.ChangeStrAll("�", "s");
UsAsciiStr = UsAsciiStr.ChangeStrAll("�", "S");
UsAsciiStr = UsAsciiStr.ChangeStrAll("�", "z");
UsAsciiStr = UsAsciiStr.ChangeStrAll("�", "Z");
TChA UsAsciiChA=TXmlLx::GetPlainStrFromXmlStr(UsAsciiStr);
for (int ChN=0; ChN<UsAsciiChA.Len(); ChN++){
char Ch=UsAsciiChA[ChN];
if ((Ch<' ')||('~'<Ch)){UsAsciiChA.PutCh(ChN, 'x');}
}
return UsAsciiChA;
}

TStr TXmlLx::GetChRefFromYuEntRef(const TStr& YuEntRefStr){
TStr ChRefStr=YuEntRefStr;
ChRefStr = ChRefStr.ChangeStrAll("&ch;", "è");
ChRefStr = ChRefStr.ChangeStrAll("&Ch;", "È");
ChRefStr = ChRefStr.ChangeStrAll("&sh;", "�");
ChRefStr = ChRefStr.ChangeStrAll("&Sh;", "�");
ChRefStr = ChRefStr.ChangeStrAll("&zh;", "�");
ChRefStr = ChRefStr.ChangeStrAll("&Zh;", "�");
ChRefStr = ChRefStr.ChangeStrAll("&cs", "c");
ChRefStr = ChRefStr.ChangeStrAll("&Cs;", "C");
ChRefStr = ChRefStr.ChangeStrAll("&dz;", "dz");
ChRefStr = ChRefStr.ChangeStrAll("&Dz;", "Dz");
return ChRefStr;
}

Add shebang line to test/nodejs/test.sh

Add shebang #!/bin/bash (changed for portability reasons) #!/usr/bin/env bash to test.sh. (It crashes when executed with sh.)

Check casting double to uint or uint64 (needed by ARM systems)

Using casts uint64(-5.0) results in undefined behaviour, one was fixed in json.h, check if there are other cases.

Rename THistogramDiff to TVecDiff

Since the stream aggregate THistogramDiff is more generic then just taking a difference of two histograms

Stream aggregate functions: getTimestampAt

Throws a C++ exception for an empty timestamp vector:

var aggr = {
      name: 'TimeSeriesWindowAggr',
      type: 'timeSeriesWinBuf',
      store: 'Function',
      timestamp: 'Time',
      value: 'Value',
       winsize: 2000
};
var sa = store.addStreamAggr(aggr);
assert.throws(function () {
       var date = sa.getTimestampAt(0); // throws exception
});

SVDFactorization doesn't work for matrices with at least one dimension 1

Method

void TNumericalStuff::SVDFactorization(const TFltVV& A, TFltVV& U, TFltV& Sing, TFltVV& VT)

breaks if A is of dimension 1xn or nx1. The problem is that UpDiag is empty and we are trying to access it's first element

weighted sampling

Two problems:

1. The function below (qminer_core.cpp) doesn't use weights when sampling.
2. SortedP doesn't sample (not what you would expect) - it only truncates.

///////////////////////////////
// QMiner-ResultSet
void TRecSet::GetSampleRecIdV(const int& SampleSize, 
        const bool& SortedP, TUInt64IntKdV& SampleRecIdFqV) const {

    if (SampleSize == -1) {
        SampleRecIdFqV = RecIdFqV;
    } else if (SortedP) { 
        const int SampleRecs = TInt::GetMn(SampleSize, GetRecs());
        SampleRecIdFqV.Gen(SampleRecs, 0);
        for (int RecN = 0; RecN < SampleRecs; RecN++) {
            SampleRecIdFqV.Add(RecIdFqV[RecN]);
        }
    } else {
        for (int RecN = 0; RecN < GetRecs(); RecN++) {
            SampleRecIdFqV.Add(RecIdFqV[RecN]); 
        }
        if (SampleSize < GetRecs()) { 
            TRnd Rnd(1); SampleRecIdFqV.Shuffle(Rnd); 
            SampleRecIdFqV.Trunc(SampleSize); 
        }
    }
}

Update documentation and build instructions

We now all recent major versions of node.

Feature Space: updateRecord function

C++ assertion failed in node_object_wrap.h: Expression: handle->InternalFieldCount() > 0

var ftr = new qm.FeatureSpace(base, { type: "numeric", source: "FtrSpaceTest", normalize: true, field: "Value" });

ftr.updateRecord({ Value: 1.1, Category: "a", Categories: ["a", "q"], Date: "2014-10-10T00:11:22", Text: "Something something." }); // throws assertion failed

base.close()

var s = base.store('Store1');
base.close(); // all objects that have a reference to base are invalid now
s.length; // crashes whole program

Solution:

A C++ Semaphore object that holds the state of TBase: either valid or invalid. Add asserts for valid state in all methods that depend on TBase. Each JS wrapped object will hold a smart pointer to the Semaphore. The close() method on TNodeJsBase will tell the Semaphore that the state is now invalid and will clean up TBase.

MDS

add an example, documentation, unit test and control over all internal params when building

Feature Space: constructor

Crashes Node, if any or all of the parameters are not given:

var ftr = new qm.FeatureSpace(base, { type: "numeric", source: "FtrSpaceTest", field: "Value" }); // works ok
var ftr = new qm.FeatureSpace(base); // crashes Node
var ftr = new qm.FeatureSpace({ type: "numeric", source: "FtrSpaceTest", field: "Value" }); // crashes Node
var ftr = new qm.FeatureSpace(); // crashes Node

linear search

base.search should work even if no keys are specified, by doing a linear search.

Feature Space: filter function

Doesn't work properly. The output sparse vector has the wrong values.

var ftr = new qm.FeatureSpace(base, [
               { type: "numeric", source: "FtrSpaceTest", field: "Value" },
               { type: "categorical", source: "FtrSpaceTest", field: "Category", values: ["a", "b", "c"] },
               { type: "categorical", source: "FtrSpaceTest", field: "Category", hashDimension: 2 },
               { type: "multinomial", source: "FtrSpaceTest", field: "Categories", values: ["a", "b", "c", "q", "w", "e"] },
               { type: "multinomial", source: "FtrSpaceTest", field: "Categories", hashDimension: 4 }
            ]);

var in_vec = ftr.ftrVec(Store[0]);
var out_vec = ftr.filter(in_vec, 4); // returns empty vector []

module loading + prototypes

Example: A sparse matrix created from analytics will not get the additional javascript implementations from scripts/la.js.
A soulution would be to load the prototype at the start of the program and link it to the template in the init function, so that all vectors would behave the same

Feature Space: invFtrVec function

Throws exception if parameter is a vector.

var ftr = new qm.FeatureSpace(base, { type: "numeric", source: "FtrSpaceTest", field: "Value" });
var ftrvec = ftr.ftrVec(Store[0]);
var vec = ftr.invFtrVec(ftrvec); // throws exception

Extend store.push to work with Object (currently supports JSON only)

// currently only JSON objects can be added
store.push(record); // record is of type qm.Record

support datetime objects in records - not pure JSON objects. Currently we need to transform date to ISO string.
store.push({date : new Date()})

SchemaTimeWindowDefinition update documentation

https://rawgit.com/qminer/qminer/master/nodedoc/module-qm.html#~SchemaTimeWindowDefinition

TODO:

• update documentation and example
• include garbageCollect() in example

api changes and patches. Anything missing?

EMPTY STRING, CLR
TStr::GetNullStr() removed -> use: TStr()
str.Clr() now private -> use: str = TStr();

IMMUTABLE STRING:
void TStr::toTrunc() removed. Use: str = str.getTrunc();
void TStr::toLc() removed. Use: str = str.getLc();
void TStr::toUc() removed. Use: str = str.getUc();
void TStr::changeStr(..,..) -> TStr TStr::changeStr(). Use: str = str.changeStr(...,...)
void TStr::changeStrAll(..,..) -> TStr TStr::changeStrAll(). Use: str = str.changeStrAll(...,...)
void TStr::changeCh(..,..) -> TStr TStr::changeCh(). Use: str = str.changeCh(...,...)
void TStr::changeChAll(..,..) -> TStr TStr::changeChAll(). Use: str = str.changeChAll(...,...)
bool TStr::DelStr(...) -> TStr TStr::DelStr(...). Since we don't return a boolean anymore (indicating that a deletion happened), compare the new length to the old length

Broken example: svc

SVM example, qminer/exmaples/svc, seems to need attention as it passes too many arguments to newFeatureExtractor.

More restrictive travis checks: no warnings, no tabs!

-release + debug tests
-code without warnings
-code without tabs

FS, stemmer: "none" Node.js crash

If the feature space is defined as

var ftr = new qm.FeatureSpace(base, {
      type: "text", source: "FtrSpaceTest", normalize: false, field: "Text",
      tokenizer: { type: "simple", stopwords: "none", stemmer: "none" }
      });

node crashes. In the documentation it says that "none" it's an option for stemmer...

record set: getMat function

Throws a C++ exception, if the parameter is a non-existing field.

Store: {
        "name": "Basketball",
        "fields": [
            { "name": "Player", "type": "string", "primary": true },
            { "name": "Score", "type": "float_v" }
        ]
    }
var mat = recordSet.getMat("Game");  // throws the exception

TRec::DoSingleJoin should have native implementation

Right now it just calls DoJoin and gets first element from Record Set.

String split methods (TStr branch)

TStr branch
The following methods should not break API (if possible)

/// Split on the index, return Pair of Left/Right strings, omits the target index
TStrPr SplitOnChN(const int& ChN) const;
/// Split on first occurrence of SplitCh, return Pair of Left/Right strings, omits the target character
/// if the character is not found the whole string is returned as the left side
TStrPr SplitOnCh(const char& SplitCh) const;
/// Split on last occurrence of SplitCh, return Pair of Left/Right strings
/// if the character is not found the whole string is returned as the right side
TStrPr SplitOnLastCh(const char& SplitCh) const;
/// Split on all occurrences of SplitCh, write to StrV, optionally don't create empy strings (default true)
void SplitOnAllCh(const char& SplitCh, TStrV& StrV, const bool& SkipEmpty=true) const;
// Split on all occurrences of any char in SplitChStr, optionally don't create empy strings (default true)
void SplitOnAllAnyCh(const TStr& SplitChStr, TStrV& StrV, const bool& SkipEmpty=true) const;
// Split on the occurrences of any string in StrV
void SplitOnWs(TStrV& StrV) const;
// Split on the occurrences of any non alphanumeric character
void SplitOnNonAlNum(TStrV& StrV) const;
// Split on all the occurrences of SplitStr
void SplitOnStr(const TStr& SplitStr, TStrV& StrV) const;

Improve error handling

In C++ methods that execute javascript callbacks we need to improve error handling. Instead of rethrowing exceptions thrown from javascript code, we need to convert them to glib/qminer exceptions and throw them.

This was fixed for one method in 3c02d50 and it has to be fixed in all places where ReThrow was used.

BAD way:

    v8::Local<v8::Value> ArgV[Argc] = { JsRec1, JsRec2 };   
    v8::Local<v8::Value> ReturnVal = Callbck->Call(GlobalContext, Argc, ArgV);
    if (TryCatch.HasCaught()) {
        TryCatch.ReThrow();
        return false;
    }

GOOD way:

    v8::Local<v8::Value> ArgV[Argc] = { JsRec1, JsRec2 };
    v8::Local<v8::Value> ReturnVal = Callbck->Call(GlobalContext, Argc, ArgV);
    if (TryCatch.HasCaught()) {
        v8::String::Utf8Value Msg(TryCatch.Message()->Get());
        throw TQm::TQmExcept::New("Javascript exception from callback triggered in TJsRecPairFilter::operator() :" + TStr(*Msg));
    }

field join types that use other data types

Currently field joins use 8 bytes for storing record id and 4 bytes to store Fq. It would be helpful if this would be customizable.
For storing record it it would be nice if storing using 4 bytes would be supported.
For storing record fq it would be good to support 0 byte, 1 byte and 4 bytes (as currently). In case of 0 bytes, the Fq would simply return 1 if join would exist and 0 otherwise.

Store.push(rec): Doesn't throw exception if rec format is not consistent with store schema.

In some cases (see example bellow), exception is not thrown when trying to push record with inconsistent schema to the store. Error is logged, but exception is not thrown. When this happens, .push(rec) method returns id 4294967295.

Example:

var qm = require('qminer');

var base = new qm.Base({
    mode: 'createClean',
    dbPath: './db'  
});

base.createStore([{
        "name": "store",
        "fields": [
            { "name": "DateTime", "type": "datetime", "primary": true },
            { "name": "Measure", "type": "float", "null": true },
        ]}
]);

// record examples
var recs = [
    {"DateTime": "2015-03-10T00:00:00", "Measure": 5}, // Record ok. Rec successfully stored.
    {"DateTime": "2015-03-10 00:01:00", "Measure": 5}, // Wrong DateTime format (missing T). PROBLEM: Does not throw exception! Retruns ID: 4294967295!
    {"DateTime": "2015-03-10 00:02:00", "Measure": 5}, // Wrong DateTime format (missing T). PROBLEM: Does not throw exception! Retruns ID: 4294967295!
    {"DateTime": "2015-03-10T00:03:00", "Measure": 5}, // Record ok. Rec sucessfully stored.
    {"DateTime": "2015-03-10T00:04:00", "Measure": "5"}, // Wrong type Measure (should be numeric, but is string). Correctly throws exception.
    {"DateTime": "2015-03-10T00:05:00", "Measure": 5}, // Record ok: Rec sucessfully stored.
    {"DateTime": "2015-03-10T00:05:00", "Measure": "5"}, // Wrong type Measure, and includes DateTime which allready exists. PROBLEM: Does not throw exception! Retruns ID: 4294967295!
    {"DateTime": "2015-03-10T00:05:00", "Measure": "5"}, // Same rec as before. This time exception is thrown correctly.
]

recs.forEach(function (rec) {
    try {
        var id = base.store("store").push(rec);
        console.log("\x1b[32mSuccessfully stored rec " + JSON.stringify(rec) + " with ID: " + id + '\x1b[0m\n')
    } catch (err) {
        console.log("\033[31mUps! " + err + '\x1b[0m\n')
    }
})

Output:

Successfully stored rec {"DateTime":"2015-03-10T00:00:00","Measure":5} with ID: 0

[TStoreImpl::AddRec] Error parsing out reference to existing record:
The parameter is incorrect.
 (c:\users\bkazic\documents\develop\test\bugtest2\node_modules\qminer\src\glib\base\os.cpp line 198:
 SystemTimeToFileTime(&SysTm, &FileTm))
Successfully stored rec {"DateTime":"2015-03-10 00:01:00","Measure":5} with ID: 4294967295

[TStoreImpl::AddRec] Error parsing out reference to existing record:
The parameter is incorrect.
 (c:\users\bkazic\documents\develop\test\bugtest2\node_modules\qminer\src\glib\base\os.cpp line 198:
 SystemTimeToFileTime(&SysTm, &FileTm))
Successfully stored rec {"DateTime":"2015-03-10 00:02:00","Measure":5} with ID: 4294967295

Successfully stored rec {"DateTime":"2015-03-10T00:03:00","Measure":5} with ID: 1

Ups! TypeError: [addon] Exception: [except] Provided JSon data field Measure is not numeric.

Successfully stored rec {"DateTime":"2015-03-10T00:05:00","Measure":5} with ID: 2

[TStoreImpl::AddRec] Error parsing out reference to existing record:
Provided JSon data field Measure is not numeric.
Successfully stored rec {"DateTime":"2015-03-10T00:05:00","Measure":"5"} with ID: 4294967295

Ups! TypeError: [addon] Exception: [except] Provided JSon data field Measure is not numeric.

Invalid Date in Merger aggregator

If the datetime is greater than 1970-01-01T00:00:00.0, it works perfectly. If less, the date is invalid.

var qm = require('qminer');
var assert = require('../../src/nodejs/scripts/assert.js');

var base = new qm.Base({
    mode: 'createClean',
    schema: [{
        name: 'Cars',
        fields: [
            { name: 'NumberOfCars', type: 'float' },
            { name: 'Time', type: 'datetime' }
        ]
    },
    {
        name: 'Temperature',
        fields: [
            { name: 'Celcius', type: 'float' },
            { name: 'Time', type: 'datetime' }
        ]
    },
    {
        name: 'Merged',
        fields: [
            { name: 'NumberOfCars', type: 'float' },
            { name: 'Celcius', type: 'float' },
            { name: 'Time', type: 'datetime' }
        ]
    }]
});

var store = base.store('Merged');

var aggr = {
    name: 'MergerAggr',
    type: 'stmerger',
    outStore: 'Merged',
    createStore: false,
    timestamp: 'Time',
    fields: [
        { source: 'Cars', inField: 'NumberOfCars', outField: 'NumberOfCars', interpolation: 'linear', timestamp: 'Time' },
        { source: 'Temperature', inField: 'Celcius', outField: 'Celcius', interpolation: 'linear', timestamp: 'Time' }
    ]
};
var merger = new qm.StreamAggr(base, aggr);
base.store("Cars").push({ NumberOfCars: 5, Time: '1601-01-01T00:00:00.000' });
base.store("Temperature").push({ Celcius: 28.3, Time: '1601-01-01T00:00:01.000' });
base.store("Cars").push({ NumberOfCars: 15, Time: '1601-01-01T00:00:02.000' });
base.store("Temperature").push({ Celcius: 30.3, Time: '1601-01-01T00:00:03.000' });

//console.log(store[0].toJSON());

assert.equal(store.length, 2);
assert.equal(store[0].NumberOfCars, 10);
assert.equal(store[0].Celcius, 28.3);
assert.equal(store[0].Time.getTime(), 1000); // store[0].Time gives Invalid Date

assert.equal(store[1].NumberOfCars, 15);
assert.equal(store[1].Celcius, 29.3);
assert.equal(store[1].Time, 2000); // store[1].Time gives Invalid Date

Alerts on Continuous queries

Thank you for sharing this good work.

Trying to verify if this can be used for alert when something interesting happens in a stream scenario.

i.e., can some one "listen" to a stream for "data changes"?

For example, if last 3 values are greater than the mean of the of last 8 data points then raise an alert (emit event).

Or, instead of emitting an event, it could create a new data point in a child stream, so that who ever is "listening" on that child stream can see that data point and do whatever is necessary.

Base stream:  v1   v2   v3   v4   v5   v6   v7   v8 ....
child stream:    .     .     .      .      1     .      .   .....

I am looking for creating "user defined functions" as "alert triggering" mechanism on streams, so that user can define his own "monitoring" javascript function to know about interesting changes on his data.

Can this package achieve such things? Creating child streams and listening on child streams for data changes etc..?

Any pointers or example would be of great help.

propagate LIBSVM's error messages

Stream aggregate functions: timeseries win buffer

An unexpected node quit if some keys are missing in the defenition of the stream aggregate.

// missing timestamp and value key
var aggr = {
                name: 'TimeSeriesWindowAggr',
                type: 'timeSeriesWinBuf',
                store: 'Function',
                winsize: 2000
            };

// missing store key
var aggr = {
                name: 'TimeSeriesWindowAggr',
                type: 'timeSeriesWinBuf',
                timestamp: 'Time',
                value: 'Value',
                winsize: 2000
            };

store: cell function

Throws a C++ exception when the fieldName does not exist in the store.

// "Date" is not a field in the store "People"
var date = table.base.store("People").cell(0, "Date"); 

Load UnicodeDef.bin on require('qminer')

At the moment it loads only when creating or opening qminer.Base.

weighed records

Searching by using a key of type 'text' should return a term-frequency weighted record set, but the result is not weighted.

var qm = require('qminer');
// Create a store People which stores only names of persons.
var base = new qm.Base({
    mode: 'createClean',
    schema: [
        { name: 'People', 
         fields: [{ name: 'name', type: 'string', primary: true }],
         keys: [{ field: 'name', type: 'value'}, { field: 'name', name: 'nameText', type: 'text'}]
        }
    ]
});

base.store('People').add({name : 'John Smith'});
base.store('People').add({name : 'Mary Smith Smith'});
base.search({$from : 'People', nameText: 'Smith'}); // returns the record set containing both 'John Smith' and 'Mary Smith', the search is based on the key of type 'text'  

Publish binaries for all suported nodejs versions

ARMv7 needs a script that publishes binaries for node 0.12/4/5.

Memory alignment when building for ARM systems

Check for getters and setters that use pointer casts as they may not be portable on some devices. ARMv7 for example doesn't support unaligned access to doubles. See f682240 for an example of a fix if you encounter a bus error.

http://cpansearch.perl.org/src/DENIK/Lingua-Identify-CLD2-0.03/src/cld2/internal/port.h

Rename of "docjs" directory to "docjs.todo" broke homepage example links

The links in the examples section of the QMiner homepage have been broken since the "docjs" directory was renamed "docjs.todo" in commit af8a084.

`recordSet.getVector`, `store.getVector`, stream aggregates should return signed Unix timestamps in milliseconds

Record set: Sort function

Exit's Node, when the callback function doesn't have a body OR no parameter si given:

recordSet.sort(function (rec, rec2) { }); // recordset.js line 311
recordSet.sort(); // recordset.js line 316

additional char field type

It would be nice to be able to have an unsigned char as a possible field type. Useful for storing data such as flags, etc.

prevent LIBSVM's internal functions from writing to stdout

record set: getVec function

Throws a C++ exception, if the parameter is a non-existing field.

Store: {
        "name": "People",
        "fields": [
            { "name": "Name", "type": "string", "primary": true },
            { "name": "Gender", "type": "string", "shortstring": true }
        ]}
var arr = recordSet.getVec("DateOfBirth"); // throws the C++ exception