jeffheaton / encog-java-core Goto Github PK

The method Propagation.iteration(final int) in file java/org/encog/neural/networks/training/propagation/Propagation.java
does not iterate 'count' times as expected, the code indeed shows that no loop is implemented in there, and it does just one single iteration.

This is with commit 986d075.

Kernel method for SVM cannot be chosen in Workbench. In the parameters dialog, only C and Gamma parameters can be set - both for "Basic SVM Training" and "Search SVM Training". According to http://www.heatonresearch.com/wiki/Support_Vector_Machine, there should be 5 kernels available: rbf, linear, poly, precomputed, sigmoid (rbf is used automatically).

Have you thought about supporting PMML? This XML standard allows you to exchange predictive models with other applications and it is quite well supported now. That would allow for instance a developer to build a neural network with Encog and deploy it into a more complex worflow in Knime or Weka for instance. Or vice-versa.

http://en.wikipedia.org/wiki/Predictive_Model_Markup_Language

http://www.dmg.org/v4-0-1/NeuralNetwork.html

Add the Encog Realtime Protocol. This needs further definition.

Please add "Select all" button (or checkbox) to the Scatter Plot dialog that would toggle all checkboxes. It is annoying to do it manually one by one every time.

When building Workbench by maven, a warning appears:

Some problems were encountered while building the effective model for org.encog:encog-workbench:jar:3.1.0-SNAPSHOT
'build.plugins.plugin.version' for org.apache.maven.plugins:maven-jar-plugin is missing. @ line 106, column 15

It is highly recommended to fix these problems because they threaten the stability of your build.
For this reason, future Maven versions might no longer support building such malformed projects.

This can be fixed by adding version tag to pom.xml - line 109:

<version>2.3.2</version>

• Once .egb file is open in Workbench, it cannot be renamed via popup menu / "Properties" (nothing happens).
• This "Properties" option should also be called "Rename" as it only shows a file rename dialog.

From a forum post.

Basically add a means of processing a temporal 2d array.

First of all, I'd like to thank you for your great effort on Encog - this is the greatest tool!!!

I'm a fresh on Encog and NN, currently I'm using Encog for my project - a lotto prediction project, I need to handle 2d array to predict a 1d array, like:
Input:{{1,2,3,4,5,6},{22,234,45,56,56,23},{1,23,34,2,6,67}}
Ideal: {2,34,56,78,34,12}

I'm using time-series, but I got the problem - after I invoke analyze(double[][]), I can't find corresponding process(double[][]) method.

For class org.encog.ml.data.MLData.TemporalWindowArray of Encog 3.0, we got analyze(double[]) and process(double[]) , but only analyze(double[][]) without process(double[][]) method.

I would be very appreciated if you could help me out of this.

One more question is, what are your recommended NNs for lotto prediction. Thank you very much.

After generating .ega file using Analyst Wizard in Workbench, .ega file should be open in a new tab. It is a logical action (though a double-click does the job) and I think it would be handy (not only) for newcomers.

For more info see:

http://www.heatonresearch.com/comment/reply/2471#comment-form

A bug report -- this is strange in that I cannot be the first trying to create a PNN for classification on Windows VISTA...

Steps to produce the error:

1. File->New file
2. Choose Machine Learnign Method, type some name, click OK
3. In next dialog, choose PNN/GRNN, click OK
4. In the next dialog, put/choose something, anything, click OK, and you'll get the error

BTW, I tried to download the workbench source but the latest I can find is 2.5.3 in Google Code. Tried git, but the tarball has a few dozen java files in the root dir. Is there a better way to get the workbench source code?

Thanks in advance!

Encog Version: 3.0.1
Encog Workbench Version: 3.0.1
Java Version: 1.6.0_30
Java Vendor: Sun Microsystems Inc.
OS Name: Windows Vista
OS Arch: x86
OS Version: 6.0

Core Count: 4

Exception: java.lang.NullPointerException
org.encog.neural.pnn.PersistBasicPNN.save(PersistBasicPNN.java:233)
org.encog.persist.EncogDirectoryPersistence.saveObject(EncogDirectoryPersistence.java:184)
org.encog.persist.EncogDirectoryPersistence.saveObject(EncogDirectoryPersistence.java:148)
org.encog.workbench.EncogWorkBench.save(EncogWorkBench.java:340)
org.encog.workbench.process.CreateNeuralNetwork.process(CreateNeuralNetwork.java:123)
org.encog.workbench.process.CreateNewFile.performCreateFile(CreateNewFile.java:63)
org.encog.workbench.frames.document.EncogMenus.actionPerformed(EncogMenus.java:216)
org.encog.workbench.frames.document.EncogDocumentFrame.actionPerformed(EncogDocumentFrame.java:112)
javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
javax.swing.DefaultButtonModel.setPressed(Unknown Source)
javax.swing.AbstractButton.doClick(Unknown Source)
javax.swing.plaf.basic.BasicMenuItemUI.doClick(Unknown Source)
javax.swing.plaf.basic.BasicMenuItemUI$Handler.mouseReleased(Unknown Source)
java.awt.Component.processMouseEvent(Unknown Source)
javax.swing.JComponent.processMouseEvent(Unknown Source)
java.awt.Component.processEvent(Unknown Source)
java.awt.Container.processEvent(Unknown Source)
java.awt.Component.dispatchEventImpl(Unknown Source)
java.awt.Container.dispatchEventImpl(Unknown Source)
java.awt.Component.dispatchEvent(Unknown Source)
java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
java.awt.Container.dispatchEventImpl(Unknown Source)
java.awt.Window.dispatchEventImpl(Unknown Source)
java.awt.Component.dispatchEvent(Unknown Source)
java.awt.EventQueue.dispatchEventImpl(Unknown Source)
java.awt.EventQueue.access$000(Unknown Source)
java.awt.EventQueue$1.run(Unknown Source)
java.awt.EventQueue$1.run(Unknown Source)
java.security.AccessController.doPrivileged(Native Method)
java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown Source)
java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown Source)
java.awt.EventQueue$2.run(Unknown Source)
java.awt.EventQueue$2.run(Unknown Source)
java.security.AccessController.doPrivileged(Native Method)
java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown Source)
java.awt.EventQueue.dispatchEvent(Unknown Source)
java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
java.awt.EventDispatchThread.pumpEvents(Unknown Source)
java.awt.EventDispatchThread.pumpEvents(Unknown Source)
java.awt.EventDispatchThread.run(Unknown Source)

Hi there,

Was just poking through the code and noticed that the performance of the MatrixMath.dotProduct method isn't as good as it could be for two reasons: it creates copies of arrays unnecessarily using the Matrix.toPackedArray method, and the Matrix.toPackedArray method actually returns arrays of type Double rather than double.

On my machine (Intel i7, Ubuntu 11.04, Sun Java 1.6) the following alternate method runs between 10 and 14 times faster (the double/Double and array copying contribute approximately equally to the performance difference):

    public static double dotProduct2(final Matrix a, final Matrix b) {
        if (!a.isVector() || !b.isVector()) {
            throw new MatrixError("To take the dot product, both matrices must be vectors.");
        }

        final double[][] aArray = a.getData();
        final double[][] bArray = b.getData();

        final int aLength = aArray.length == 1 ? aArray[0].length : aArray.length;
        final int bLength = bArray.length == 1 ? bArray[0].length : bArray.length;

        if (aLength != bLength) {
            throw new MatrixError("To take the dot product, both matrices must be of the same length.");
        }

        double result = 0;
        if (aArray.length == 1 && bArray.length == 1) {
            for (int i = 0; i < aLength; i++) {
                result += aArray[0][i] * bArray[0][i];
            }
        }
        else if (aArray.length == 1 && bArray[0].length == 1) {
            for (int i = 0; i < aLength; i++) {
                result += aArray[0][i] * bArray[i][0];
            }
        }
        else if (aArray[0].length == 1 && bArray.length == 1) {
            for (int i = 0; i < aLength; i++) {
                result += aArray[i][0] * bArray[0][i];
            }
        }
        else if (aArray[0].length == 1 && bArray[0].length == 1) {
            for (int i = 0; i < aLength; i++) {
                result += aArray[i][0] * bArray[i][0];
            }
        }

        return result;
    }

TrainFlatNetworkResilient.getError() returns the error measured before the last call to TrainFlatNetworkResilient.train().

It would be nice to document that behaviour in the function as it can be surprising.

You can observe the issue by replacing the end of XORFlat.ava with this block. You'll see that the mse is different from the one displayed with getError after the the last iteration.

    double[] output = new double[1];
    // test the neural network
    System.out.println("Neural Network Results:");
    double mse = 0;
    for(MLDataPair pair: trainingSet ) {
        double[] input = pair.getInput().getData();
        network.compute(input, output);
        System.out.println(input[0] + "," + input[1] + ":" + output[0]);
        mse += (output[0] - pair.getIdeal().getData()[0]) * (output[0] - pair.getIdeal().getData()[0]); 
    }
    System.out.println("MSE: "+(mse / trainingSet.getRecordCount()));
    train.iteration();
    System.out.println("Error: "+train.getError());

Since your stopping condition in XORFlat.java is based on getError() it actually runs one unnecessary iteration. So this type of construct is a bit misleading because in some cases the MSE can increase from one epoch to the next and the actual MSE of the network when you leave the loop might not meet the stopping condition. However in general I don't think it will make a huge difference though.

Already completed.

This issue is already completed.

The random weights initialization doesn't work properly in version 3.0.1.

Follow this steeps to reproduce the problem:

• Create a feedforward neural network.
• Randomize the network using random [-1,1] values.
• If you visualize the weight histogram you can observe that the weights follows a random distribution.
• Train the network with some patterns.
• Stop the training and close the training window.
• Randomize again the network using random [-1,1] values.
• If you visualize the weight histogram you can observe that the weights DOESN'T follow a random distribution.

1. Train NN in Workbench 3.0.1 (because of issue #41) and exit Workbench.
2. Open the project in Workbench 3.1 (git snapshot), doubleclick .ega file, select "task-evaluate", hit "Execute" and then "Start". The following exception will be thrown:

Exception: java.lang.NullPointerException org.encog.app.analyst.script.normalize.AnalystField.encodeEquilateral(AnalystField.java:367) org.encog.app.analyst.script.normalize.AnalystField.encode(AnalystField.java:333) org.encog.app.analyst.script.normalize.AnalystField.encode(AnalystField.java:355) org.encog.app.analyst.csv.normalize.AnalystNormalizeCSV.extractFields(AnalystNormalizeCSV.java:102) org.encog.app.analyst.csv.AnalystEvaluateCSV.process(AnalystEvaluateCSV.java:188) org.encog.app.analyst.commands.CmdEvaluate.executeCommand(CmdEvaluate.java:106) org.encog.app.analyst.EncogAnalyst.executeTask(EncogAnalyst.java:487) org.encog.app.analyst.EncogAnalyst.executeTask(EncogAnalyst.java:514) org.encog.workbench.tabs.analyst.AnalystProgressTab.run(AnalystProgressTab.java:334) java.lang.Thread.run(Unknown Source)

Here's my data (and project) to reproduce the bug: http://dione.zcu.cz/~toman40/encog/encog_eval_bug.zip

There is no information about progress after the training is started, i.e. after this message appears in the log pane

Thu Feb 09 20:30:58 CET 2012 [DEBUG][Thread-5]: Beginning training

only iteration count is changing. I propose to add something like:

EncogLogging.log(EncogLogging.LEVEL_DEBUG, "Training with parameters C = " + c + ", gamma = " + gamma);

to line 255 in SVMTrain.java,

EncogLogging.log(EncogLogging.LEVEL_DEBUG, "Training error from iteration: e = " + e);

to line 278 in SVMSearchTrain.java and

EncogLogging.log(EncogLogging.LEVEL_DEBUG, "Training error due to libsvm bug -> e = 100.0");

to line 281 in SVMSearchTrain.java.

Hello,
please consider implementing regularization, as it is essential to deal with the overfitting problem.

I recommend watching 12 min. video "Regularization and Bias/Variance" of lesson X. ADVICE FOR APPLYING MACHINE LEARNING at https://class.coursera.org/ml/lecture/preview (Stanford ML course).

It would also be useful to enhance Encog Analyst - it could split data into 3 sets (training, cross validation, testing) and try to find the optimal regularization parameter automatically.

You could implement Flexibale Neural Trees, it appears to show good results on Forecasting.
Have you ever dealed with it?

Search for:

Flexible neural trees ensemble for stock index modeling. Chen Yuehui, Yang Bo, Ajith Abraham

on google you'll have the article.

From:

http://www.heatonresearch.com/node/2459

I'm currently using an SVM with 686 inputs and one output. The issue is that the results of getError() are erratic, and seem far from correct. This can be seen easily using the below example.

Current Error: 0.21111051567344982
(results of EncogUtility.evaluate() on the testing set)
, Actual=0.3972, Ideal=0.0000
, Actual=-0.1095, Ideal=0.2500
, Actual=0.4550, Ideal=0.5000
, Actual=0.0829, Ideal=0.7500
, Actual=0.3323, Ideal=1.0000
, Actual=0.1957, Ideal=0.0000
, Actual=0.3133, Ideal=0.2500
, Actual=-0.0573, Ideal=0.5000
, Actual=0.1611, Ideal=0.7500
, Actual=0.3852, Ideal=1.0000
, Actual=0.2117, Ideal=0.0000
, Actual=0.3612, Ideal=0.2500
, Actual=0.3014, Ideal=0.5000
, Actual=0.2159, Ideal=0.7500
, Actual=0.2739, Ideal=1.0000

Most of these values are way off correct. It implies the SVM has at least some level of training - when in fact the true margin of error is something like 0.48 - so no real training at all.

I've seen other cases of incorrect error calculation (such as a margin of error of 0.25 whatever values of gamma and C are used) - but I suspect they'd all be stemming from the same bug.

If necessary, I can post source code?

EG files store the weight array slightly differently for large format networks, so that the weights are not on a single ginormous line, that can't be read into memory. Training is failing because these networks are not being either loaded or saved correctly and the end result is an array of zeros for most of the weight matrix. Such a neural network is not trainable.

--- From original report--
It seems that error reporting is broken in the latest Workbench (built from git sources) - at least for RProp and SVMSearch - "Current Error" just hangs after couple iterations (and the chart is also freezed, if displayed). Interestingly, it works with QProp, for example. (I have no problems with Workbench 3.0.1, using the same data.)

Sometimes project files (left pane) in Workbench get out of sync. Currently, "Refresh" command in the popup menu can be used to synchronize. However, there is a better way to ensure proper synchronization, which allows replacing manual refresh completely - by means of WatchService (see http://docs.oracle.com/javase/tutorial/essential/io/notification.html).

Here's the code the Elliott Activation function in case someone is interested. It is not as popular as tanh and sigmoid but I've seen it used in a few papers.

The implementation is based on this report:

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.46.7204&rep=rep1&type=pdf

Since I discovered that something like 70% of the training time is spent in Math.tanh or Math.exp I was looking for a cheap alternative. The main advantage of this activation function is that it is very fast to compute. It is bounded between -1 and 1 like tanh but will reach those values more slowly so it might be more suitable for classification tasks.

I've had very mixed and results with this implementation so far. Used with Rprop on a xor problem it seems to perform quite badly in terms of number of iterations and getting stuck in local minima or not being able to go below high MSE values. It is quite unexpected so I'm wondering if maybe there's a mistake somewhere with the derivative.

On the other hand I've also observed excellent results with evolutionary algorithms like GA (and my version of PSO) with often very fast convergence compared to tanh and sigmoid. That's why I put this code here in case it might be useful to someone else.

/*
 */
package org.encog.engine.network.activation;

/**
 * Computationally efficient alternative to ActivationTANH.
 * Its output is in the range [-1, 1], and it is derivable.
 * 
 * It will approach the -1 and 1 more slowly than Tanh so it 
 * might be more suitable to classification tasks than predictions tasks.
 * 
 * Elliott, D.L. "A better activation function for artificial neural networks", 1993
 * http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.46.7204&rep=rep1&type=pdf
 */
public class ActivationElliott implements ActivationFunction {

    /**
     * Serial id for this class.
     */
    private static final long serialVersionUID = 1234L;

    /**
     * The parameters.
     */
    private final double[] params;

    /**
     * Construct a basic HTAN activation function, with a slope of 1.
     */
    public ActivationElliott() {
        this.params = new double[0];
    }

    /**
     * {@inheritDoc}
     */
    @Override
    public final void activationFunction(final double[] x, final int start,
            final int size) {
        for (int i = start; i < start + size; i++) {
            x[i] = 1.0 / (1.0 + (Math.abs(x[i])));
        }
    }

    /**
     * @return The object cloned;
     */
    @Override
    public final ActivationFunction clone() {
        return new ActivationElliott();
    }

    /**
     * {@inheritDoc}
     */
    @Override
    public final double derivativeFunction(final double b, final double a) {
        return (1.0 - a) * (1.0 - a);
    }

    /**
     * {@inheritDoc}
     */
    @Override
    public final String[] getParamNames() {
        final String[] result = {};
        return result;
    }

    /**
     * {@inheritDoc}
     */
    @Override
    public final double[] getParams() {
        return this.params;
    }

    /**
     * @return Return true, Elliott activation has a derivative.
     */
    @Override
    public final boolean hasDerivative() {
        return true;
    }

    /**
     * {@inheritDoc}
     */
    @Override
    public final void setParam(final int index, final double value) {
        this.params[index] = value;
    }

}

Hello,
running the latest Encog Workbench (built from git sources) on the data below, I get the following exception:

We are very sorry but an unexpected error has occured.
Would you consider sending this information to us?
No personal information will be transmitted, just what you see below.

This information is very useful to us to make Encog a better program.

Encog Version: 3.1.0
Encog Workbench Version: 3.1
Java Version: 1.6.0_30
Java Vendor: Sun Microsystems Inc.
OS Name: Windows Vista
OS Arch: x86
OS Version: 6.0
Core Count: 2
ISO3 Country: USA
Display Country: United States
Radix: .

Grouping: ,

Exception: java.lang.ArrayIndexOutOfBoundsException: -1
org.encog.mathutil.libsvm.Solver_NU.select_working_set(svm.java:1069)
org.encog.mathutil.libsvm.Solver.Solve(svm.java:540)
org.encog.mathutil.libsvm.Solver_NU.Solve(svm.java:962)
org.encog.mathutil.libsvm.svm.solve_nu_svc(svm.java:1437)
org.encog.mathutil.libsvm.svm.svm_train_one(svm.java:1567)
org.encog.mathutil.libsvm.svm.svm_train(svm.java:2097)
org.encog.ml.svm.training.SVMTrain.iteration(SVMTrain.java:235)
org.encog.ml.svm.training.SVMSearchTrain.iteration(SVMSearchTrain.java:271)
org.encog.app.analyst.commands.CmdTrain.performTraining(CmdTrain.java:219)
org.encog.app.analyst.commands.CmdTrain.executeCommand(CmdTrain.java:121)
org.encog.app.analyst.EncogAnalyst.executeTask(EncogAnalyst.java:487)
org.encog.app.analyst.EncogAnalyst.executeTask(EncogAnalyst.java:514)
org.encog.workbench.tabs.analyst.AnalystProgressTab.run(AnalystProgressTab.java:335)
java.lang.Thread.run(Unknown Source)

This problem was also reported here:

http://www.heatonresearch.com/node/2368 (partly fixed?)
http://www.heatonresearch.com/node/2398 (no replies)

data.csv:

i1,i2,i3,i4,i5,i6,i7,i8,i9,y
0.0003,1.666666667,1,0.000124023,-0.000225,-0.000704,-0.001492,-0.001547,2.07E-05,1
0.0003,1,1,0.000348262,0.000155,-0.000344,-0.001144,-0.0012415,5.80E-05,1
0.0006,2.25,10,0.000234719,-0.000195,-0.003012,-0.003864,-0.004737,3.91E-05,0
0.0004,2,3,5.80E-05,-2.00E-05,-0.002582,-0.003959,-0.0048975,9.66E-06,1
0.0004,1.5,1,0.00039255,0.00042,-0.002062,-0.003513,-0.004471,6.54E-05,0
0.0002,1.5,0.5,0.000156911,0.00041,-0.0018,-0.003533,-0.0045565,2.62E-05,0
0.0003,1.666666667,1,6.20E-05,-0.000135,-0.000922,-0.0034,-0.0045225,1.03E-05,0
0.0003,1,1,0.000449639,0.00033,-0.000272,-0.002828,-0.00396,7.49E-05,0
0.0006,1.2,2,0.000376355,-0.00017,-0.001406,-0.004004,-0.005835,6.27E-05,0
-1.00E-04,4,0.166666667,0.000236876,-0.00013,-0.001478,-0.004007,-0.005891,3.95E-05,0
0.0008,1.714285714,1.5,-0.000290168,-0.00089,-0.003022,-0.005279,-0.0079045,-4.84E-05,1
0.0005,1.2,0.5,0.000288856,-0.000135,-0.002226,-0.004466,-0.0071965,4.81E-05,0
0.0002,2.666666667,0.666666667,0.000246055,0.000455,0.00085,-0.00101,-0.0044385,4.10E-05,0
0.0012,1.153846154,0.25,0.000585271,0.00062,0.00124,-0.000548,-0.004005,9.75E-05,0
-1.00E-04,2.5,6,-0.000184848,-0.000125,0.00069,-0.000868,-0.0042985,-3.08E-05,1
0.0011,1,1,0.000784416,0.0009,0.001764,0.000259,-0.0031595,0.000130736,1
0.0012,1.25,0.25,0.000791247,0.00112,0.002248,0.00105,-0.0022235,0.000131875,1
0.0008,1,1,-2.17E-05,-0.000665,-0.00093,0.00021,-0.002472,-3.61E-06,1
0.0007,1,1,0.000657545,0.000395,-0.000198,0.001018,-0.001595,0.000109591,1

There were some changes to the NWR weight initialization in 3.1. I do not believe either the current or original code really have it right. However, the original 3.0 code does seem to work better on some training sets. I can see that the current NWR is not touching every neuron, which it should.

This was discovered looking at #38

It would be useful if log level was saved to config. Also, perhaps INFO would be more convenient default setting than NONE.

This is really just about done, just need to finish HMM persistence. HMM will NOT be hooked into the workbench at this point.

This issue was completed earlier for Encog 3.1, adding for tracking purposes.

Add better support for "signals" in Encog. This is somewhat "open ended", and will be defined further later. Most likely with additional issue records.

When the current error and error improvement are plot during the training, the x-axis shows only a couple of recent iterations (130-290, for example), which does not provide a really good overview of how the training error has evolved since the start.

It would be more convenient to show iterations from 1 up to the current one. Or, there could be an option or checkbox for switching these two views in realtime.

We are very sorry but an unexpected error has occured.
Would you consider sending this information to us?
No personal information will be transmitted, just what you see below.

This information is very useful to us to make Encog a better program.

Encog Version: 3.0.1
Encog Workbench Version: 3.0.1
Java Version: 1.7.0
Java Vendor: Oracle Corporation
OS Name: Linux
OS Arch: amd64
OS Version: 2.6.38-8-generic

Core Count: 8

Exception: java.lang.NullPointerException
org.encog.app.analyst.analyze.PerformAnalysis.process(PerformAnalysis.java:158)
org.encog.app.analyst.EncogAnalyst.analyze(EncogAnalyst.java:173)
org.encog.app.analyst.wizard.AnalystWizard.wizard(AnalystWizard.java:1131)
org.encog.workbench.process.EncogAnalystWizard.createEncogAnalyst(EncogAnalystWizard.java:118)
org.encog.workbench.frames.document.EncogPopupMenus.performPopupMenu(EncogPopupMenus.java:111)
org.encog.workbench.frames.document.EncogPopupMenus.actionPerformed(EncogPopupMenus.java:63)
org.encog.workbench.frames.document.EncogDocumentFrame.actionPerformed(EncogDocumentFrame.java:113)
javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2018)
javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2341)
javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
javax.swing.AbstractButton.doClick(AbstractButton.java:376)
javax.swing.plaf.basic.BasicMenuItemUI.doClick(BasicMenuItemUI.java:833)
javax.swing.plaf.basic.BasicMenuItemUI$Handler.mouseReleased(BasicMenuItemUI.java:877)
java.awt.Component.processMouseEvent(Component.java:6504)
javax.swing.JComponent.processMouseEvent(JComponent.java:3321)
java.awt.Component.processEvent(Component.java:6269)
java.awt.Container.processEvent(Container.java:2229)
java.awt.Component.dispatchEventImpl(Component.java:4860)
java.awt.Container.dispatchEventImpl(Container.java:2287)
java.awt.Component.dispatchEvent(Component.java:4686)
java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4832)
java.awt.LightweightDispatcher.processMouseEvent(Container.java:4492)
java.awt.LightweightDispatcher.dispatchEvent(Container.java:4422)
java.awt.Container.dispatchEventImpl(Container.java:2273)
java.awt.Window.dispatchEventImpl(Window.java:2713)
java.awt.Component.dispatchEvent(Component.java:4686)
java.awt.EventQueue.dispatchEventImpl(EventQueue.java:707)
java.awt.EventQueue.access$000(EventQueue.java:101)
java.awt.EventQueue$3.run(EventQueue.java:666)
java.awt.EventQueue$3.run(EventQueue.java:664)
java.security.AccessController.doPrivileged(Native Method)
java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:87)
java.awt.EventQueue$4.run(EventQueue.java:680)
java.awt.EventQueue$4.run(EventQueue.java:678)
java.security.AccessController.doPrivileged(Native Method)
java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
java.awt.EventQueue.dispatchEvent(EventQueue.java:677)
java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:211)
java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:128)
java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:117)
java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:113)
java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:105)
java.awt.EventDispatchThread.run(EventDispatchThread.java:90)

Attempting a build on my UTF-8 OS-X installation, I get the following errors:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile (default-compile) on project encog-core: Compilation failure: Compilation failure:
[ERROR] /Users/robert/wkdir/encog-java-core/src/main/java/org/encog/ml/hmm/train/bw/BaseBaumWelch.java:[51,49] error: unmappable character for encoding UTF-8
[ERROR] /Users/robert/wkdir/encog-java-core/src/main/java/org/encog/ml/hmm/train/bw/BaseBaumWelch.java:[53,36] error: unmappable character for encoding UTF-8
[ERROR] /Users/robert/wkdir/encog-java-core/src/main/java/org/encog/ml/hmm/HiddenMarkovModel.java:[69,53] error: unmappable character for encoding UTF-8
[ERROR] /Users/robert/wkdir/encog-java-core/src/main/java/org/encog/ml/hmm/train/bw/TrainBaumWelchScaled.java:[48,49] error: unmappable character for encoding UTF-8
[ERROR] /Users/robert/wkdir/encog-java-core/src/main/java/org/encog/ml/hmm/train/bw/TrainBaumWelchScaled.java:[50,36] error: unmappable character for encoding UTF-8
[ERROR] /Users/robert/wkdir/encog-java-core/src/main/java/org/encog/ml/hmm/train/bw/TrainBaumWelch.java:[47,49] error: unmappable character for encoding UTF-8
[ERROR] /Users/robert/wkdir/encog-java-core/src/main/java/org/encog/ml/hmm/train/bw/TrainBaumWelch.java:[49,36] error: unmappable character for encoding UTF-8
[ERROR] /Users/robert/wkdir/encog-java-core/src/main/java/org/encog/ml/hmm/alog/ForwardBackwardScaledCalculator.java:[35,14] error: unmappable character for encoding UTF-8
[ERROR] /Users/robert/wkdir/encog-java-core/src/main/java/org/encog/ml/hmm/alog/ForwardBackwardCalculator.java:[34,14] error: unmappable character for encoding UTF-8
[ERROR] /Users/robert/wkdir/encog-java-core/src/main/java/org/encog/ml/hmm/alog/ViterbiCalculator.java:[40,56] error: unmappable character for encoding UTF-8
[ERROR] /Users/robert/wkdir/encog-java-core/src/main/java/org/encog/ml/hmm/alog/KullbackLeiblerDistanceCalculator.java:[34,47] error: unmappable character for encoding UTF-8
[ERROR] -> [Help 1]

What encoding should be used? Can you put that into the mvn config?

If you do drag more than one file, just one of the files shows up in the workbench.

Am I wrong or there's lack of TrainHopfield class in newest JAVA realse?

The following error is thrown on some GA's when Encog is run under Java 7.

Got this error when I ran the Lunar Lander Example:

Exception in thread "main" java.lang.IllegalArgumentException: Comparison method violates its general contract!
at java.util.ComparableTimSort.mergeLo(Unknown Source)
at java.util.ComparableTimSort.mergeAt(Unknown Source)
at java.util.ComparableTimSort.mergeCollapse(Unknown Source)
at java.util.ComparableTimSort.sort(Unknown Source)
at java.util.ComparableTimSort.sort(Unknown Source)
at java.util.Arrays.sort(Unknown Source)
at java.util.Collections.sort(Unknown Source)
at org.encog.ml.genetic.population.BasicPopulation.sort(BasicPopulation.java:445)
at org.encog.neural.networks.training.genetic.NeuralGeneticAlgorithm.(NeuralGeneticAlgorithm.java:134)
at org.encog.examples.neural.lunar.LunarLander.main(LunarLander.java:71)

Unless gpg (jar signing) is needed for some reason in pom.xml (both in encog-java-core and encog-java-workbench), I suggest to remove maven-gpg-plugin reference from both pom.xml files.

Hi,
it would be useful if multiple target fields could be specified in the Analyst Wizard (Workbench), perhaps comma separated, for example:

Target Field (blank for auto): ideal1, ideal2, ideal3

Currently, only one may be specified and a change requires manual editing of .ega file and running Encog tasks again.

Running Analyst Wizard in Workbench 3.1 on data with no CSV headers may produce an exception:

Exception: java.lang.NullPointerException
org.encog.app.analyst.analyze.PerformAnalysis.process(PerformAnalysis.java:162)
org.encog.app.analyst.EncogAnalyst.analyze(EncogAnalyst.java:174)
org.encog.app.analyst.wizard.AnalystWizard.wizard(AnalystWizard.java:1247)
org.encog.workbench.process.EncogAnalystWizard.createEncogAnalyst(EncogAnalystWizard.java:121)
org.encog.workbench.frames.document.EncogPopupMenus.performPopupMenu(EncogPopupMenus.java:121)
org.encog.workbench.frames.document.EncogPopupMenus.actionPerformed(EncogPopupMenus.java:65)
org.encog.workbench.frames.document.EncogDocumentFrame.actionPerformed(EncogDocumentFrame.java:117)
javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
javax.swing.DefaultButtonModel.setPressed(Unknown Source)
javax.swing.AbstractButton.doClick(Unknown Source)
javax.swing.plaf.basic.BasicMenuItemUI.doClick(Unknown Source)
javax.swing.plaf.basic.BasicMenuItemUI$Handler.mouseReleased(Unknown Source)
java.awt.Component.processMouseEvent(Unknown Source)
javax.swing.JComponent.processMouseEvent(Unknown Source)
java.awt.Component.processEvent(Unknown Source)
java.awt.Container.processEvent(Unknown Source)
java.awt.Component.dispatchEventImpl(Unknown Source)
java.awt.Container.dispatchEventImpl(Unknown Source)
java.awt.Component.dispatchEvent(Unknown Source)
java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
java.awt.Container.dispatchEventImpl(Unknown Source)
java.awt.Window.dispatchEventImpl(Unknown Source)
java.awt.Component.dispatchEvent(Unknown Source)
java.awt.EventQueue.dispatchEventImpl(Unknown Source)
java.awt.EventQueue.access$000(Unknown Source)
java.awt.EventQueue$1.run(Unknown Source)
java.awt.EventQueue$1.run(Unknown Source)
java.security.AccessController.doPrivileged(Native Method)
java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown Source)
java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown Source)
java.awt.EventQueue$2.run(Unknown Source)
java.awt.EventQueue$2.run(Unknown Source)
java.security.AccessController.doPrivileged(Native Method)
java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown Source)
java.awt.EventQueue.dispatchEvent(Unknown Source)
java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
java.awt.EventDispatchThread.pumpEvents(Unknown Source)
java.awt.EventDispatchThread.pumpEvents(Unknown Source)
java.awt.EventDispatchThread.run(Unknown Source)

Test data: use nine.csv from http://dione.zcu.cz/~toman40/encog/encog_visualization.zip

(Note: Workbench 3.0.1 reports "Can't determine target field automatically..." in this case.)

I think that in Encog 3 the notion of population already exists but it seems to be tied to genetic algorithms like the standard GA and Neat. The evolutionary and swarm intelligence approaches contains many other population-based algorithms other than GA. For instance PSO or differential evolution, both of them can be applied to neural network.

It would probably be good to have a more abstract population-based framework to facilitate the addition of such algorithms in the future. One that would offer generic functions to parallelise the training of the population, implement relevant stopping conditions, allow hybrid algorithms to be plugged in etc.

This will allow Encog Objects, as well as data to be more easily stored in SQL.

This will be the "reference implementation" for the Encog Realtime Protocol. I hope that others will help to create similar interfaces for other trading platforms, such as MT4, MT5 and Trade Station.

The architecture of Encog is really well designed with nice separations between the different parts of the networks and the training algorithms. It seems to be built to be very extensible and flexible.

There is a limitiation with RProp implementation however. Due to the strong encapsulation of the parameters (see issue #2) but also to the fact that the updateWeightXXX() functions are not very modular, the only way to introduce modifications to the behaviour of the algorithm is to change the code directly in TrainFlatNetworkResilient.java and apply that same modification to the four upateWeightXXX() functions.

Some improvements of RProp have been proposed (e.g. SARProp or RProp with nonextensive schedule) which only require a slight modification of the calculation of the delta when the sign fo the gradient is changing.

I don't think that it would be possible to implement those changes with a Strategy class as the modification is located in the algorithm itself.

If at least the fields and methods in TrainFlatNetworkResilient were declared as protected rather than private, subclasing would be feasible. Although the subclass would still have to contain nearly the same code as the parent class, which is not ideal.

Not sure if there is a nice solution to this problem but I still mention it in case you had an idea.

Steps to reproduce:

1. Generate .egb file.
2. Doubleclick .egb -> tab opens -> hit "Visualize" button.

This applies both for Workbench 3.0.1 and 3.1.

Secondary issue: unexpected exceptions are not always logged (line 170 in EncogTabManager.java in this case).

This is a feature request.

It would be helpful to expose the parameters of the training algorithms with setters and getters.

Take RProp for instance (TrainFlatNetworkResilient). Some of the parameters can be set via a call to the constructor. But after that there is no way to get their value. And since those parameters are also declared as private, there is no way for a new subclass to change the behaviour of the training algorithm.

Other parameters like RPROPConst.POSITIVE_ETA and RPROPConst.NEGATIVE_ETA are constants. When doing research, it is often important to run many trials with a range of possible values for the heuristics in order to find the best combination. In this case it is only possible to do it by changing the value of the constants in Encog's source file.

Going a step further, it would be even better in a future release to expose those parameters more dynamically. By that I mean a standard interface for obtaining the list of the parameters, their name, description, value and possible range. This would be allow a UI to work more dynamically with the algorithms. It would also allow to log information about a training algorithm and its exact settings to a file so an experiment could be properly documented and reproduced.

How about improving the visualization of .egb file a bit?

1. Alow to rotate/flip axes.
2. Use grayscale instead of black and white.

Sample data: http://dione.zcu.cz/~toman40/encog/encog_visualization.zip

When training is run via Analyst (after clicking on .ega file), there is no graph plotting the current error and error improvement, in contrast to training run from BasicNetwork tab (after clicking on .eg file). I propose to plot the graph in both cases.

I've attached my version of PSO for Encog. It is a standard implementation of the global best version of PSO. It works in multi-threaded mode by default. The implementation is modular so it is easy to extend it to create new variants such as the Constriction Coefficient.

I've worked a lot with it recently and I've tested it with several datasets. Unlike other methods like Rprop, It needs a bit of tuning to work well on a specific problem. w, c1, c2 are important but Vmax seems to be crucial to obtain good convergence rate. The control parameters are documented in the code.

Feel free to reuse it partly or completely if you wish.

package org.encog.neural.networks.training.pso;

import java.util.Random;

import neural.utils.VectorAlgebra;

import org.encog.mathutil.randomize.Randomizer;
import org.encog.ml.MLMethod;
import org.encog.ml.TrainingImplementationType;
import org.encog.ml.train.BasicTraining;
import org.encog.neural.networks.BasicNetwork;
import org.encog.neural.networks.structure.NetworkCODEC;
import org.encog.neural.networks.training.CalculateScore;
import org.encog.neural.networks.training.propagation.TrainingContinuation;
import org.encog.util.concurrency.EngineConcurrency;
import org.encog.util.concurrency.TaskGroup;

/**
 * Iteratively trains a population of neural networks by applying   
 * particle swarm optimisation (PSO).
 * 
 * References: 
 *  James Kennedy and Russell C. Eberhart, Particle swarm optimization, 
 * Proceedings of the IEEE International Conference on Neural Networks, 
 * 1995, pp. 1942-1948
 * 
 * @author Geoffroy Noel
 */
public class NeuralPSO extends BasicTraining {

    protected boolean m_multiThreaded = true;
    protected VectorAlgebra m_va;
    protected CalculateScore m_calculateScore;
    protected Randomizer m_randomizer;

    // Swarm state and memories.
    protected BasicNetwork[] m_networks;
    protected double[][] m_velocities;
    protected double[][] m_bestVectors;
    protected double[] m_bestErrors;
    protected int m_bestVectorIndex;

    // Although this is redundant with m_bestVectors[m_bestVectorIndex],
    // m_bestVectors[m_bestVectorIndex] is not thread safe.
    private double[] m_bestVector;
    BasicNetwork m_bestNetwork = null;

    // Typical range is 20 - 40 for many problems. 
    // More difficult problems may need much higher value. 
    // Must be low enough to keep the training process 
    // computationally efficient.
    protected int m_populationSize = 30;

    // Determines the size of the search space. 
    // The position components of particle will be bounded to 
    // [-maxPos, maxPos]
    // A well chosen range can improve the performance. 
    // -1 is a special value that represents boundless search space. 
    protected double m_maxPosition = -1;

    // Maximum change one particle can take during one iteration.
    // Imposes a limit on the maximum absolute value of the velocity 
    // components of a particle. 
    // Affects the granularity of the search.
    // If too high, particle can fly past optimum solution.
    // If too low, particle can get stuck in local minima.
    // Usually set to a fraction of the dynamic range of the search
    // space (10% was shown to be good for high dimensional problems).
    // -1 is a special value that represents boundless velocities. 
    protected double m_maxVelocity = 2;

    // c1, cognitive learning rate >= 0
    // tendency to return to personal best position
    protected double m_c1 = 2.0;
    // c2, social learning rate >= 0
    // tendency to move towards the swarm best position
    protected double m_c2 = 2.0;

    // w, inertia weight.
    // Controls global (higher value) vs local exploration 
    // of the search space. 
    // Analogous to temperature in simulated annealing.
    // Must be chosen carefully or gradually decreased over time.
    // Value usually between 0 and 1.
    protected double m_inertiaWeight = 0.4;

    // If true, the position of the previous global best position 
    // can be updated *before* the other particles have been modified.
    private boolean m_pseudoAsynchronousUpdate = false;

    /**
     * Constructor.
     * 
     * @param network           an initialised Encog network. 
     *                          The networks in the swarm will be created with 
     *                          the same topology as this network.
     * @param randomizer        any type of Encog network weight initialisation
     *                          object.
     * @param calculateScore    any type of Encog network scoring/fitness object. 
     * @param populationSize    the swarm size.
     */
    public NeuralPSO(final BasicNetwork network,
            final Randomizer randomizer, final CalculateScore calculateScore,
            final int populationSize) {
        super(TrainingImplementationType.Iterative);
        // initialisation of the member variables
        m_populationSize = populationSize;
        m_randomizer = randomizer;
        m_calculateScore = calculateScore;
        m_bestNetwork = network;

        m_networks = new BasicNetwork[m_populationSize];
        m_velocities = null; 
        m_bestVectors = new double[m_populationSize][];
        m_bestErrors = new double[m_populationSize];
        m_bestVectorIndex = -1;

        // get a vector from the network.
        m_bestVector = NetworkCODEC.networkToArray(m_bestNetwork);

        m_va = new VectorAlgebra();
    }

    /**
     * Initialise the particle positions and velocities, 
     * personal and global bests.
     * Only does this if they have not yet been initialised.
     */
    void initPopulation() {
        if (m_velocities == null) {
            int dimensionality = m_bestVector.length;
            m_velocities = new double[m_populationSize][dimensionality];
            // run an initialisation iteration
            iterationPSO(true);
        }
    }

    /**
     * Runs one PSO iteration over the whole population of networks.
     */
    @Override
    public void iteration() {
        initPopulation();

        preIteration();
        iterationPSO(false);
        postIteration();
    }

    /**
     * Internal method for the iteration of the swarm.
     * 
     * @param init  true if this is an initialisation iteration.
     */
    protected void iterationPSO(boolean init) {
        final TaskGroup group = EngineConcurrency.getInstance().createTaskGroup();

        for (int i = 0; i < m_populationSize; i++) {
            NeuralPSOWorker worker = new NeuralPSOWorker(this, i, init);
            if (!init && this.isMultiThreaded()) {
                EngineConcurrency.getInstance().processTask(worker, group);
            } else {
                worker.run();
            }
        }
        if (this.isMultiThreaded()) {
            group.waitForComplete();
        }
        updateGlobalBestPosition();
    }

    /**
     * Update the velocity, position and personal 
     * best position of a particle 
     * 
     * @param particleIndex     index of the particle in the swarm
     * @param init              if true, the position and velocity
     *                          will be initialised. 
     */
    protected void updateParticle(int particleIndex, boolean init) {
        int i = particleIndex;
        double[] particlePosition = null;
        if (init) {
            // Create a new particle with random values.
            // Except the first particle which has the same values 
            // as the network passed to the algorithm.
            if (m_networks[i] == null) {
                m_networks[i] = (BasicNetwork) m_bestNetwork.clone();
                if (i > 0) m_randomizer.randomize(m_networks[i]);
            }
            particlePosition = getNetworkState(i);
            m_bestVectors[i] = particlePosition;

            // randomise the velocity
            m_va.randomise(m_velocities[i], m_maxVelocity);
        } else {
            particlePosition = getNetworkState(i);
            updateVelocity(i, particlePosition);

            // velocity clamping
            m_va.clampComponents(m_velocities[i], m_maxVelocity);

            // new position (Xt = Xt-1 + Vt)
            m_va.add(particlePosition, m_velocities[i]);

            // pin the particle against the boundary of the search space.
            // (only for the components exceeding maxPosition)
            m_va.clampComponents(particlePosition, m_maxPosition);

            setNetworkState(i, particlePosition);
        }
        updatePersonalBestPosition(i, particlePosition);
    }

    /**
     * Update the velocity of a particle 
     * 
     * @param particleIndex     index of the particle in the swarm
     * @param particlePosition  the particle current position vector
     */
    protected void updateVelocity(int particleIndex, double[] particlePosition) {
        int i = particleIndex;
        double[] vtmp = new double[particlePosition.length];

        // Standard PSO formula

        // inertia weight
        m_va.mul(m_velocities[i], m_inertiaWeight);

        // cognitive term
        m_va.copy(vtmp, m_bestVectors[i]);
        m_va.sub(vtmp, particlePosition);
        m_va.mulRand(vtmp, m_c1);
        m_va.add(m_velocities[i], vtmp);

        // social term
        if (i != m_bestVectorIndex) {
            m_va.copy(vtmp, m_pseudoAsynchronousUpdate ? m_bestVectors[m_bestVectorIndex] : m_bestVector);
            m_va.sub(vtmp, particlePosition);
            m_va.mulRand(vtmp, m_c2);
            m_va.add(m_velocities[i], vtmp);
        }
    }

    /**
     * Update the personal best position of a particle.
     * 
     * @param particleIndex     index of the particle in the swarm
     * @param particlePosition  the particle current position vector
     */
    protected void updatePersonalBestPosition(int particleIndex, double[] particlePosition) {
        // set the network weights and biases from the vector
        double score = m_calculateScore.calculateScore(m_networks[particleIndex]);

        // update the best vectors (g and i)
        if ((m_bestErrors[particleIndex] == 0) || isScoreBetter(score, m_bestErrors[particleIndex])) {
            m_bestErrors[particleIndex] = score;
            m_va.copy(m_bestVectors[particleIndex], particlePosition);
        }
    }

    /**
     * Update the swarm's best position
     */
    protected void updateGlobalBestPosition() {
        boolean bestUpdated = false;
        for (int i = 0; i < m_populationSize; i++) {
            if ((m_bestVectorIndex == -1) || isScoreBetter(m_bestErrors[i], m_bestErrors[m_bestVectorIndex])) {
                m_bestVectorIndex = i;
                bestUpdated = true;
            }
        }
        if (bestUpdated) {
            m_va.copy(m_bestVector, m_bestVectors[m_bestVectorIndex]);
            m_bestNetwork.decodeFromArray(m_bestVector);
            setError(m_bestErrors[m_bestVectorIndex]);
        }
    }

    /**
     * Compares two scores.
     * 
     * @param score1    a score
     * @param score2    a score
     * @return  true if score1 is better than score2
     */
    boolean isScoreBetter(double score1, double score2) {
        return ((m_calculateScore.shouldMinimize() && (score1 < score2)) || ((!m_calculateScore.shouldMinimize()) && (score1 > score2)));
    }

    @Override
    public TrainingContinuation pause() {
        return null;
    }

    @Override
    public boolean canContinue() {
        return false;
    }

    @Override
    public void resume(TrainingContinuation state) {
    }

    /**
     * Returns the state of a network in the swarm 
     * 
     * @param particleIndex     index of the network in the swarm
     * @return  an array of weights and biases for the given network
     */
    protected double[] getNetworkState(int particleIndex) { 
        return NetworkCODEC.networkToArray(m_networks[particleIndex]);
    }

    /**
     * Sets the state of the networks in the swarm
     * 
     * @param particleIndex     index of the network in the swarm
     * @param state             an array of weights and biases
     */
    protected void setNetworkState(int particleIndex, double[] state) { 
        NetworkCODEC.arrayToNetwork(state, m_networks[particleIndex]);
    }

    /**
     * Set the swarm size.
     * 
     * @param populationSize    the swarm size
     */
    public void setPopulationSize(int populationSize) {
        m_populationSize = populationSize;
    }

    /**
     * Returns the swarm size.
     * 
     * @return the swarm size.
     */
    public int getPopulationSize() {
        return m_populationSize;
    }

    /**
     * Sets the maximum velocity.
     * 
     * @param maxVelocity   Maximum velocity / Vmax
     */
    public void setMaxVelocity(double maxVelocity) {
        m_maxVelocity = maxVelocity;
    }

    /**
     * Get the maximum velocity (Vmax)
     * 
     * @return  maximum velocity (Vmax)
     */
    public double getMaxVelocity() {
        return m_maxVelocity;
    }

    /**
     * Set the boundary of the search space (Xmax)
     * 
     * @param maxPosition   maximum value for a component (Xmax)
     */
    public void setMaxPosition(double maxPosition) {
        m_maxPosition = maxPosition;
    }

    /**
     * Get the boundary of the search space (Xmax)
     * 
     * @return  the maximum value a component can take (Xmax)
     */
    public double getMaxPosition() {
        return m_maxPosition;
    }

    /**
     * Sets the cognition coefficient (c1).
     * 
     * @param c1    cognition coefficient (c1)
     */
    public void setC1(double c1) {
        m_c1 = c1;
    }

    /**
     * Get the cognition coefficient (c1).
     * 
     * @return  the cognition coefficient (c1)
     */
    public double getC1() {
        return m_c1;
    }

    /**
     * Set the social coefficient (c2).
     * 
     * @param c2    the social coefficient (c2)
     */
    public void setC2(double c2) {
        m_c2 = c2;
    }

    /**
     * Get the social coefficient (c2).
     * 
     * @return  the social coefficient (c2)
     */
    public double getC2() {
        return m_c2;
    }

    /**
     * Set the inertia weight (w)
     * 
     * @param inertiaWeight the inertia weight (w)
     */
    public void setInertiaWeight(double inertiaWeight) {
        m_inertiaWeight = inertiaWeight;
    }

    /**
     * Get the inertia weight (w)
     * 
     * @return the inertia weight (w)
     */
    public double getInertiaWeight() {
        return m_inertiaWeight;
    }

    /**
     * Get a description of all the current settings.
     * 
     * @return a String describing all the current setting in a single line. 
     */
    public String getDescription() { 
        return String.format("pop = %d, w = %.2f, c1 = %.2f, c2 = %.2f, Xmax = %.2f, Vmax = %.2f", 
                m_populationSize, m_inertiaWeight, m_c1, m_c2, m_maxPosition, m_maxVelocity);
    }

    @Override
    /**
     * Returns the most fit network in the swarm
     * 
     *  @return the most fit network in the swarm
     */
    public MLMethod getMethod() {
        return m_bestNetwork;
    }

    /**
     * Keep a reference to the passed population of networks.
     * This population is not copied, it will evolve during training.  
     * 
     * @param initialPopulation
     */
    public void setInitialPopulation(BasicNetwork[] initialPopulation) {
        m_networks = initialPopulation;
    }

    /**
     * Get the multi-threaded mode.
     * 
     * @return true if PSO works in multi-threaded mode
     */
    public boolean isMultiThreaded() {
        return m_multiThreaded;
    }

}

Please note that this problem may just be a bug in my code but I'd rather report this anyway in case someone is able to confirm whether the problem is in Encog or not.

The PROBEN1 report (page 32) claims that they get 0.027 MSE over 101 epochs on average for that dataset. And the classification error is around 17%.

I can't get anything near that with my code based on Encog. So I've implement the same experiment in MATLAB and Encog to objectively compare the behaviours. I stop training if the MSE goes under 0.05 or the number of iterations is 300. I repeat the training process over 100 randomly initialised networks.

In MATLAB 88 times out of 100 the training stops before the 300 limit as it reaches the MSE target of 0.05. And on average it does so in 95 epochs. And the classification accuracies are in line with those described in PROBEN1.

With my code based on Encog and with the same dataset, the same network architecture and, I believe, the same RProp heuristics, the MSE is 0.14 on average. It goes below 0.11 only once out of the 100 trainings.

In practice this difference means that the trained network should be able to correctly classify 83% of the testing samples but instead it is stuck at 65% even when I let the training run up to 3000 epochs. As you can see below the differences in classification error over the training set is even worse.

Here are the results for 10 trials:

MATLAB:

experiment(gene1, 300, [4,2,3], 10, 0.05);
Trial: 1, Epochs: 97, MSE: 0.0489183, CE(trn): 6.4%, CE(tst): 16.4%
Trial: 2, Epochs: 28, MSE: 0.0482678, CE(trn): 7.6%, CE(tst): 18.0%
Trial: 3, Epochs: 50, MSE: 0.0497159, CE(trn): 8.0%, CE(tst): 22.2%
Trial: 4, Epochs: 56, MSE: 0.0497302, CE(trn): 6.5%, CE(tst): 19.2%
Trial: 5, Epochs: 40, MSE: 0.0493543, CE(trn): 6.4%, CE(tst): 18.9%
Trial: 6, Epochs: 54, MSE: 0.0499842, CE(trn): 6.9%, CE(tst): 16.3%
Trial: 7, Epochs: 94, MSE: 0.0497391, CE(trn): 6.2%, CE(tst): 20.9%
Trial: 8, Epochs: 28, MSE: 0.0496143, CE(trn): 6.9%, CE(tst): 15.6%
Trial: 9, Epochs: 44, MSE: 0.0483782, CE(trn): 7.7%, CE(tst): 17.9%
Trial: 10, Epochs: 64, MSE: 0.0497216, CE(trn): 7.7%, CE(tst): 19.4%
Min MSE: 0.0482678, Max MSE: 0.0499842, Avg MSE: 0.0493424
Min epoch: 28, Max epoch: 97, Avg epoch: 55.5
Successes: 10 / 10
4 2 3

My code using Encog:

Trial: 1, Epochs: 300, MSE: 0.130949, CE(trn): 28.5%, CE(tst): 35.2%
Trial: 2, Epochs: 300, MSE: 0.155891, CE(trn): 41.8%, CE(tst): 41.2%
Trial: 3, Epochs: 300, MSE: 0.167867, CE(trn): 46.3%, CE(tst): 46.0%
Trial: 4, Epochs: 300, MSE: 0.149423, CE(trn): 34.5%, CE(tst): 38.5%
Trial: 5, Epochs: 300, MSE: 0.129148, CE(trn): 28.0%, CE(tst): 34.3%
Trial: 6, Epochs: 300, MSE: 0.167622, CE(trn): 46.0%, CE(tst): 46.5%
Trial: 7, Epochs: 300, MSE: 0.153942, CE(trn): 36.5%, CE(tst): 40.6%
Trial: 8, Epochs: 300, MSE: 0.136173, CE(trn): 29.3%, CE(tst): 32.7%
Trial: 9, Epochs: 300, MSE: 0.170760, CE(trn): 47.5%, CE(tst): 47.2%
Trial: 10, Epochs: 300, MSE: 0.125813, CE(trn): 26.8%, CE(tst): 34.9%
Min MSE: 0.125813, max MSE: 0.170760, Avg MSE: 0.148759
Min epoch: 300, max epoch: 300, Avg epoch: 300.000
Successes: 0 / 10
Architecture: ?:B->SIGMOID->4:B->SIGMOID->2:B->LINEAR->?

Hi,

I am a happy user of Encog Framework. (version 3.0 beta)
I have a request for a minor enhancement, in EncogPersistor interface you use Inputstream and OutputStream in read and save Method.
I read the code and you then use those stream to create a Reader (BufferedReader) and a Writer (PrintWriter) in the Helper classes.

I am saving a encog network in a model file with many other informations for our own tool.
I am also using Reader and Writer object because they are easier than InputStream/OutputStream.

The request is to add a save(Writer w, Object o) and read(Reader r) method to the EncogPersistor interface so i can use the same reader.
I think you will only have to add another constructor to the helper classes and then both read method will use EncogReadHelper object created differently.

I think you get the idea.

I know this is a very specific request, if you don't have time to do it i will try to create and submit a patch but i am not familiar with github yet.

thank you for your great work on this framework.

Julien Blaize