How normalize discrete data input about brain.js

[robertleeplummerjr]

Forgot to mention: I'm working to try and make it so you could use numbers, or really whatever type of data on either side of the input or output.

[robertleeplummerjr]

What will your outputs be? Do you have any training data?
In this case, I'd probably lean much closer to a recurrent neural net, where you have exact lookups. This is still being implemented (ie, the shortened means of training a recurrent neural net with inputs and outputs, the actual network is implemented and mostly solid with current tests), and things like hiddenSizes and learning rate may need to be experimented with. Use https://github.com/harthur-org/brain.js/tree/rnn-train-pattern branch and try something like:

import LSTM from '../../src/recurrent/lstm';
import fs from 'fs';
//var json = JSON.parse(fs.readFileSync('transaction-training-data.json').toString()); //<- uncomment to read for further training
let net = new LSTM();
let transactionTypes = {
  credit: 'one',
  debit: 'two',
  personalCard: 'three',
  other: 'four'
};
var trainingData = [{
  input: transactionTypes.credit,
  output: 'credit'
}, {
  input: transactionTypes.debit,
  output: 'debit'
}, {
  input: transactionTypes.personalCard,
  output: 'personal card'
}, {
  input: transactionTypes.other,
  output: 'other'
}];
net.train(trainingData, { iterations: 200, log: true });
fs.writeFileSync('transaction-training-data.json', JSON.stringify(net.toJSON()));
console.log(net.run(transactionTypes.credit)); //-> 'credit'
console.log(net.run(transactionTypes.debit)); //-> 'debit'
console.log(net.run(transactionTypes.personalCard)); //-> 'personal card' 
console.log(net.run(transactionTypes.other)); //-> 'other'

Just ran that locally, and just added a fix that should give you a basis for trying it out.

Note: If you get any exceptions, I'll be glad to help resolve. This is experimental, but I've not seen anything like it anywhere, and thought what better time to implement. Also, this seems to train crazy fast!

[Dok11]

What will your outputs be? Do you have any training data?

Yes, I will create repository with my data and some test nn for use them
Training data very more, 2 GB json data 😃

[Dok11]

@robertleeplummerjr I created index.js with my NN, and folder «data» with some training data.
Of course this is not working - one interation and 0.0036 error.

Look here https://github.com/Dok11/brain.js__test/tree/iteration-1

[IonicaBizau]

@Dok11 I guess you'll have to fix the npm dependency to contain the branch information:

  ...
  "brain.js": "git+https://github.com/harthur-org/brain.js.git#rnn-train-pattern",
  ...

[IonicaBizau]

@Dok11 Sorry, even with that it's still giving one iteration.

[IonicaBizau]

@Dok11 Seems it stops after one iteration because the error is small enough.

[Dok11]

@IonicaBizau and so low error because data not normalized to range 0..1
@robertleeplummerjr writed about lstm nn for same case, but I don't full understand him :)

look at example of my input data:

a: 1 // this and another idx from dictionary [1,2,3,4,5,6,7..]
c: 1
d: 0.252 // time in year
f: 1
l: 5000000 // price
p: 7.14
pkg: 1
t: 1

[robertleeplummerjr]

Here: https://github.com/Dok11/brain.js__test/blob/iteration-1/index.js#L4 use:
var net = new brain.recurrent.LSTM();

[robertleeplummerjr]

As for the training data, you need to use an array of objects that have an input and output key like this:

var trainingData = [{
  input: 'value',
  output: 'other value'
}];

Optionally you could use arrays:

var trainingData = [{
  input: [1,2,3,4,5],
  output: [5,4,3,2,1]
}];

Still working out how to include other data.

[Dok11]

@robertleeplummerjr , I updated package.json to #rnn-train-pattern and get:

$ node index.js
...\brain.js__test\index.js:4
var net         = new brain.recurrent.LSTM();
                  ^

TypeError: brain.recurrent.LSTM is not a constructor
    at Object.<anonymous> (...\brain.js__test\index.js:4:12)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)
    at run (bootstrap_node.js:394:7)
    at startup (bootstrap_node.js:149:9)
    at bootstrap_node.js:509:3

[robertleeplummerjr]

try: new brain.recurrent.LSTM.default()
This is fixed in master, I'll merge in soon.

[Dok11]

try: new brain.recurrent.LSTM.default()

New err:

$ node index.js
...\brain.js__test\node_modules\brain.js\dist\recurrent\rnn.js:472
      throw new Error('not yet implemented');
      ^

Error: not yet implemented
    at LSTM.formatData (...\brain.js__test\node_modules\brain.js\dist\recurrent\rnn.js:472:13)
    at LSTM.train (...\brain.js__test\node_modules\brain.js\dist\recurrent\rnn.js:401:19)
    at Object.<anonymous> (...\brain.js__test\index.js:14:23)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)
    at run (bootstrap_node.js:394:7)

[robertleeplummerjr]

Is recurrent set? If not, you may be on an older (npm released) version, and not the above mentioned branch.

[Dok11]

Is recurrent set?

Maybe... I wrote:

var net		= new brain.recurrent.LSTM.default();
console.log(net);

and in ouput object has key «clipval» from https://github.com/harthur-org/brain.js/blob/d7508dd706d82a1d864cb74ae52fa4df6070a486/src/recurrent/rnn.js#L238

[robertleeplummerjr]

do this before the above statement: console.log(brain.recurrent)

[robertleeplummerjr]

What does it output?

[Dok11]

Really, I don't understand what this mean :)

var fs		= require('fs');
var walk	= require('fs-walk');
var brain	= require('brain.js');
console.log(brain);
var net		= new brain.recurrent.LSTM.default();

$ node index.js
{ crossValidate:
   { testPartition: [Function: testPartition],
     shuffleArray: [Function: shuffleArray],
     default: [Function: crossValidate] },
  likely: { default: [Function: likely] },
  lookup: { default: [Function: lookup] },
  NeuralNetwork: { default: { [Function: NeuralNetwork] trainDefaults: [Object] } },
  TrainStream: { default: [Function: TrainStream] },
  recurrent:
   { RNN: { default: [Object] },
     LSTM: { default: [Function: LSTM] },
     GRU: { default: [Function: GRU] } },
  utilities:
   { max: { default: [Function: max] },
     mse: { default: [Function: mse] },
     ones: { default: [Function: ones] },
     random:
      { randomF: [Function: randomF],
        randomI: [Function: randomI],
        randomN: [Function: randomN] },
     randomWeight: { default: [Function: randomWeight] },
     randos: { default: [Function: randos] },
     range: { default: [Function: range] },
     toArray: { default: [Function: toArray] },
     Vocab: { default: [Function: Vocab] },
     zeros: { default: [Function: zeros] } } }
...\brain.js__test\node_modules\brain.js\dist\recurrent\rnn.js:472
      throw new Error('not yet implemented');
      ^

Error: not yet implemented

[robertleeplummerjr]

I just updated this branch, pull, and try this again:
var net = new brain.recurrent.LSTM();

[Dok11]

Successfully updated from «npm update» and...

$ node index.js
...\brain.js__test\node_modules\brain.js\dist\recurrent\rnn.js:472
      throw new Error('not yet implemented');
      ^

Error: not yet implemented
    at LSTM.formatData (...\brain.js__test\node_modules\brain.js\di
st\recurrent\rnn.js:472:13)
    at LSTM.train (...\brain.js__test\node_modules\brain.js\dist\re
current\rnn.js:401:19)
    at Object.<anonymous> (...\brain.js__test\index.js:15:23)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)
    at run (bootstrap_node.js:394:7)

[robertleeplummerjr]

I believe you are on master, you need rnn-train-pattern
run this: rm -rf node_modules/brain.js && npm i git+https://github.com/harthur-org/brain.js.git#rnn-train-pattern

[robertleeplummerjr]

In #48 I added 7bb8431#diff-b29f454b6a67cbd17b8df5bfd95cac7bR136

[robertleeplummerjr]

Another way that just occurred to me that you could do is using the standard net, and do:

var net = new brain.NeuralNetwork();
net.train([
  {input: { credit: 1, debit: 1, 'personal card': 1, other: 1 }, output: { qualified: 1 }},
  {input: { credit: 0, debit: 1, 'personal card': 0, other: 1 }, output: { qualified: 0 }},
  {input: { credit: 1, debit: 0, 'personal card': 1, other: 0 }, output: { qualified: .6 }}
]);

var output = net.run({ credit: 1, debit: 0, 'personal card': 1, other: 0 }); //-> output greater than .5

Working here: https://jsfiddle.net/robertleeplummerjr/fnmx86kf/

[Dok11]

I believe you are on master, you need rnn-train-pattern
run this: rm -rf node_modules/brain.js && npm i git+https://github.com/harthur-org/brain.js.git#rnn-train-pattern

I did this and added this link in pakage.json, as you can look here:
https://github.com/Dok11/brain.js__test/blob/iteration-1/package.json

And update my branch here:
https://github.com/Dok11/brain.js__test/tree/iteration-1

When i ran "node index.js" console was stopped and nothing did.
You can try it too.

[Dok11]

Another way that just occurred to me that you could do is using the standard net, and do:

Ye, but as I wrote in topicstart this way force me make many many input neurons.
This not impossible, but is really best way for decision?

[robertleeplummerjr]

Yea, that is a good point. I still am leaning towards the recurrent neural network for this solution. I'll see if I can get your code working later today and provide a pr.

[robertleeplummerjr]

Also, thank you for your time to help us help you.

[Dok11]

Why I can't division input value to [0.1, 0.2, 0.3 etc]?

[robertleeplummerjr]

In its current implementation a recurrent neural network is more suited towards specific inputs and outputs.

(used from @karpathy's fantastic post on neural networks: http://karpathy.github.io/2015/05/21/rnn-effectiveness/)

[robertleeplummerjr]

That being said, you could (I'm naively hypothesizing at this point) convert them to strings, and the network can iterate over them.

[robertleeplummerjr]

Further clarification: the network isn't really aware (at least initially) that 0.000001 is less, more, or equal than/to 0.01, to it, it is just text, but with training it will likely begin to understand that... But I've not tried it.

[Dok11]

Good think, what i did: I dividied index key from dictionary to 1000.
This method allowed the fit input data in range 0..1
It seems work, I will check

[robertleeplummerjr]

Did this end up being solved?

[mubaidr]

If you still need help, feel free to re-open this issue.