Ver. 1.0
Intro video (11 min): https://www.youtube.com/watch?v=yB43jj-wv8Q
-
Optimized for 2D image data -- input data can be read from .bmp image files -
Neuron layers can be abstracted as 1D or 2D arrangements of neurons -
Network topology is defined in a text file -
Neurons in layers can be fully or sparsely connected -
Selectable transfer function per layer -
Adjustable or automatic training rate (eta) -
Optional momentum (alpha) and regularization (lambda) -
Any layer(s) can be configured as convolution filters -
Standalone console program -
Simple, heavily-commented code, < 3000 lines, suitable for prototyping, learning, and experimentation -
Optional web-browser-based GUI controller
Requirements
How to run the digits demo
How to run the XOR example
GUI interface
How to use your own data
The 2D in neural2d
Convolution filtering
Topology config file format
Topology config file examples
How-do-I X?
- How do I get, build, and install the command-line neural2d program?
- How do I run the command-line program?
- How do I run the GUI interface?
- How do I disable the GUI interface?
- How do I use my own data instead of the digits images?
- How do I use a trained net on new data?
- How do I train on the MNIST handlwritten digits data set?
- How do I change the learning rate parameter?
- Are the output neurons binary or floating point?
- How do I use a different transfer function?
- How do I define a convolution filter?
- How do the color image pixels get converted to floating point for the input layer?
- How can I use .jpg and .png images as inputs to the net?
- Why does the net error rate stay high? Why doesn't my net learn?
- What other parameters do I need to know about?
Also see the wiki for more information.
- C++-11 compiler (e.g., g++ on Linux)
- POSIX sockets (e.g., Cygwin on Windows)
- Compiles and runs on Linux, Windows, and probably Mac
Place all the files, maintaining the relative directory structure, into a convenient directory.
In the images/digits/ subdirectory, extract the image files from the archive into the same directory.
To compile neural2d, cd to the directory containing the Makefile and execute the default make target:
make
This will use g++ to compile neural2d.cpp and neural2d-core.cpp and result in an executable named neural2d.
To run the demo, execute:
make test
In this demo, we train the neural net to recognize digits. The input data, or "training set", consists of a few thousand images of numeric digits. The first 50 look like these:
The images are 32x32 pixels each, stored in .bmp format. In this demo, the neural net is configured to have 32x32 input neurons, and 10 output neurons. The net is trained to classify the digits in the images and to indicate the answer by driving the corresponding output neuron to a high level.
Once the net is sufficiently trained, all the connection weights are saved in a file named "weights.txt".
In the top level directory where the Makefile lives, execute:
make test-xor
For more information about the XOR example, see this wiki page.
First, launch the neural2d console program in a command window with the -p option:
./neural2d topology.txt inputData.txt weights.txt -p
The -p option causes the neural2d program to wait for a command before starting the training. The screen will look something like this:
At this point, the neural2d console program is paused and waiting for a command to continue. Using any web browser, open:
http://localhost:24080
A GUI interface will appear that looks like:
Press Resume to start the neural net training. It will automatically pause when the average error rate falls below a certain threshold (or when you press Pause). You now have a trained net. You can press Save Weights to save the weights for later use.
If you are inputting data from image files, you'll need to prepare a set of BMP image files and an input data config file. The config file (named inputData.txt by default) is a list of image filenames to use as inputs to the neural net, and optionally the target output values for each image. The format looks like this example:
images/thumbnails/test-918.bmp -1 1 -1 -1 -1 -1 -1 -1 -1 -1
images/thumbnails/test-919.bmp -1 -1 -1 -1 -1 -1 -1 -1 1 -1
images/thumbnails/test-920.bmp -1 -1 -1 -1 -1 -1 1 -1 -1 -1
images/thumbnails/test-921.bmp -1 -1 -1 -1 -1 1 -1 -1 -1 -1
The path and filename cannot contain any spaces.
If you are not using image files for input, you'll need to prepare an input config file (named inputData.txt by default) similar to the above but with the literal input values inside curly braces. For example, for a net with eight inputs and two outputs, the format is like this:
{ 0.32 0.98 0.12 0.44 0.98 1.2 1 -1 } -1 1
You'll also need a topology config file (named topology.txt by default). It contains a specification of the neural net topology (the number and arrangement of neurons and connections). Its format is described in a later section. A typical one looks something like this:
input size 32x32
layer1 size 32x32 from input radius 8x8
layer2 size 16x16 from layer1
output size 1x10 from layer2
Then run neuron2d (optionally with the web browser interface) and experiment with the parameters until the net is adequately trained, then save the weights in a file for later use.
If you run the web interface, you can change the global parameters from the GUI while the neural2d program is running. If you run the neural2d console program without the GUI interface, there is no way to interact with it while running. Instead, you'll need to examine and modify the parameters in the code at the top of the files neural2d.cpp and neural2d-core.cpp.
In a simple traditional neural net model, the neurons are arranged in a column in each layer:
In neural2d, you can specify a rectangular arrangement of neurons in each layer, such as:
The neurons can be sparsely connected to mimic how retinal neurons are connected in biological brains. For example, if a radius of "1x1" is specified in the topology config file, each neuron on the right (destination) layer will connect to a circular patch of neurons in the left (source) layer as shown here (only a single neuron on the right side is shown connected in this picture so you can see what's going on, but imagine all of them connected in the same pattern):
The pattern that is projected onto the source layer is elliptical. (Layers configured as convolution filters work slightly differently; see the later section about convolution filtering.)
Here are some projected connection patterns for various radii:
radius 0x0
radius 1x1
radius 2x2
radius 3x1
Any layer other than the input layer can be configured as a convolution filter layer by specifying a convolve-matrix specification for the layer in the topology config file. The neurons are still called neurons, but their operation differs in the following ways:
-
The connection pattern to the source layer is defined by the convolution matrix (kernel) dimensions (not by a radius parameter)
-
The connection weights are initialized from the convolution matrix, and are constant throughout the life of the net.
-
The transfer function is automatically set to the identity function (not by a tf parameter).
For example, the following line in the topology config file defines a 3x3 convolution matrix for shrarpening the source layer:
layerConv1 size 64x64 from input convolve {{0,-1,0},{-1,5,-1},{0,-1,0}}
When a convolution matrix is specified for a layer, you cannot also specify a radius parameter for that layer, as the convolution matrix size determines the size and shape of the rectangle of neurons in the source layer. You also cannot also specify a tf parameter, because the transfer function on a convolution layer is automatically set to be the identity function.
The elements of the convolution matrix are stored as connection weights to the source neurons. Connection weights on convolution layers are not updated by the back propagation algorithm, so they remain constant for the life of the net.
The results are undefined if a layer is defined as both a convolution layer and a regular layer.
For illustrations of various convolution kernels, see this Wikipedia article
In the following illustration, the topology config file defines a convolution filter with a 2x2 kernel that is applied to the input layer, then the results are combined with a reduced-resolution fully-connected pathway. The blue connections in the picture are the convolution connections; the green connections are regular neural connections:
input size 8x8
layerConvolve size 8x8 from input convolve {{-1,2},{-1,2}}
layerReducedRes size 4x4 from input
output size 2 from layerConvolve
output size 2 from layerReducedRes
The topology config file contains lines of the following format:
layer-definition-line := layer-name size size-spec [from layer-name] [channel channel-name] [radius size-spec] [tf transfer-function] [convolve convolve-matrix]
where
layer-name := "input", "output", or a string that begins with "layer"
size-spec := integer ["x" integer]
channel-name := R, G, B, or BW
transfer-function := "tanh", "logistic", "linear", "ramp", or "gaussian"
convolve-matrix := same {{,},{,}} syntax used for array initialization in C, C#, VB, Java, etc.
Rules:
-
Comment lines that begin with "#" and blank lines are ignored.
-
The first layer defined must be named "input".
-
The last layer defined must be named "output".
-
The hidden layers can be named anything beginning with "layer".
-
The argument for "from" must be a layer already defined.
-
The color channel parameter can be specified only on the input layer.
-
A convolution matrix specification cannot appear on the same layer with a radius or tf spec.
-
The same layer name can be defined multiple times with different "from" parameters. This allows source neurons from more than one layer to be combined in one destination layer. The source layers can be any size. When a destination layer is defined more than once, the first definition must have a size parameter. The size parameter is optional on the repeated lines; if it appears, it must be the same size as defined initially. For example, in the following, layerCombined is size 8x8:
input size 128x128
layerVertical size 32x32 from input radius 1x8
layerHorizontal size 16x16 from input radius 8x1
layerCombined size 8x8 from layerVertical
layerCombined from layerHorizontal
output size 1 from layerCombined -
The size-spec can specify two dimensions, or one. Spaces are not allowed in the size spec. If only one dimension is given, the other is assumed to be 1. For example:
- "8x8" means 64 neurons in an 8 x 8 arrangement.
- "8x1" means a row of 8 neurons
- "1x8" means a column of 8 neurons.
- "8" means the same as "8x1"
Here are a few complete topology config files and the nets they specify.
input size 4x4
layer1 size 3x3 from input
layer2 size 2x2 from layer1
output size 1 from layer2
input size 4x4
layer1 size 1x4 from input
layer2 size 3x1 from layer1
output size 1 from layer2
input size 4x4
output size 4x1 from input radius 0x2
input size 16x16
layer1 size 4x4 from input radius 1x1
output size 7x1 from layer1
# In the picture that follows, layerVertical is the set of 4 neurons
# in the upper part of the picture, and layerHorizontal is the lower
# set of 4 neurons.
input size 6x6
layerHorizontal size 2x2 from input radius 2x0
layerVertical size 2x2 from input radius 0x2
output size 1 from layerHorizontal
output size 1 from layerVertical
# This example shows how vertical and horizontal image features can be
# extracted through separate paths and combined in a subsequent layer.
input size 4x4
layerH1 size 1x4 from input radius 4x0
layerH2 size 1x4 from layerH1
layerH3 size 1x4 from layerH2
layerV1 size 4x1 from input radius 0x4
layerV2 size 4x1 from layerV1
layerV3 size 4x1 from layerV2
output size 2 from layerV3
output size 2 from layerH3
How do I get, build, and install the command-line neural2d program?
Get the files from:
https://github.com/davidrmiller/neural2d
Put those into a directory. Expand the images files in the images/digits/ subdirectory. In the top level directory (where the Makefile lives), build the program with:
make
That will produce an executable named neuron2d in the same directory.
To test the installation, run:
make test
If it succeeds, it will create a weights.txt file of non-zero size.
How do I run the command-line program?
./neural2d topology.txt inputData.txt weights.txt
How do I run the GUI interface?
First launch the neural2d program with the -p option:
./neural2d topology.txt inputData.txt weights.txt -p
Then open a web browser and point it at http://localhost:24080 .
If your firewall complains, you may need to allow access to TCP port 24080.
How do I disable the GUI interface?
The easiest way is to add -DDISABLE_WEBSERVER to the g++ command line in the Makefile. Alternatively, you can undefine the macro ENABLE_WEBSERVER in neural2d.h.
When the web server is disabled, there is no remaining dependency on POSIX sockets.
How do I use my own data instead of the digits images?
Create your own directory of BMP images, and a config file that follows the same format as shown in the provided default inputData.txt. Then define a topology config file with the appropriate number of network inputs and outputs, then run the neural2d program.
Or if you don't want to use image files for input, make an input config file containing all the literal input values and the target output values. The format is described in an earlier section.
How do I use a trained net on new data?
It's all about the weights file. After the net has been successfully trained, save the internal connection weights in a weights file. That's typically done in neural2d.cpp by calling the member function saveWeights(filename).
The weights you saved can be loaded back into a neural net of the same topology using the member function loadWeights(filename). Once the net has been loaded with weights, it can be used applied to new data by calling feedForward(). Prior to calling feedForward(), you'll want to set a couple of parameters:
myNet.repeatInputSamples = false;
myNet.reportEveryNth = 1;
This is normally done in neural2d.cpp.
You'll need to prepare a new input data config file (default name inputData.txt) that contains a list of only those new input data images that you want the net to process.
How do I train on the MNIST handlwritten digits data set?
See the instructions in the wiki.
How do I change the learning rate parameter?
In the command-line program, you can set the eta parameter or change it by directly setting the eta member of the Net object, like this:
myNet.eta = 0.1;
When using the web interface, you can change the eta parameter (and other parameters) in the GUI at any time, even while the network is busy processing input data.
Also see the Parameter List in the wiki.
Are the output neurons binary or floating point?
They are interpreted in whatever manner you train them to be, but you can only train the outputs to take values in the range that the transfer function is capable of producing.
If you're training a net to output binary values, it's best if you use the maxima of the transfer function to represent the two binary values. For example, when using the default tanh() transfer function, train the outputs to be -1 and +1 for false and true. When using the logistic transfer function, train the outputs to be 0 and 1.
How do I use a different transfer function?
You can add a "tf" parameter to any layer definition line in the topology config file. The argument to tf can be "tanh", "logistic", "linear", "ramp", or "gaussian". The transfer function you specify will be used by all the neurons in that layer. See neural2d-core.cpp for more information.
In the topology config file, the tf parameter is specified as in this example:
layerHidden1 size 64x64 from input radius 3x3 tf linear
You can add new transfer functions by following the examples in neural2d-core.cpp. There are two places to change: first find where transferFunctionTanh() is defined and add your new transfer function and its derivative there. Next, locate the constructor for class Neuron and add a new else-if clause there, following the examples.
How do I define a convolution filter?
In the topology config file, any layer defined with a convolve parameter will operate as a convolution filter applied to the source layer. The syntax is of the form:
layer2 size 64x64 from input convolve {{1,0,-1},{0,0,0},{-1,0,1}}
How do the color image pixels get converted to floating point for the input layer?
That's in the ReadBMP() function in neural2d-core.cpp. The default version of ReadBMP() converts each RGB pixel to a single floating point value in the range 0.0 to 1.0.
By default, the RGB color pixels are converted to monochrome and normalized to the range 0.0 to 1.0. That can be changed at runtime by setting the colorChannel member of the Net object to R, G, B, or BW prior to calling feedForward(). E.g., to use only the green color channel of the images, use:
myNet.colorChannel = NNet::G;
The color conversion can also be specified in the topology config file on the line that defines the input layer by setting the "channel" parameter to R, G, B, or BW, e.g.:
input size 64x64 channel G
How can I use .jpg and .png images as inputs to the net?
Currently only .bmp images are supported. This is because the uncompressed BMP format is so simple that we can use simple, standard C/C++ to read the image data without any dependencies on third-party image libraries. To add an adapter for other image formats, follow the example of the ReadBMP() function and write a new adapter such as ReadJPG(), ReadPNG(), etc., using your favorite image library, then replace the call to ReadBMP() with your new function.
Why does the net error rate stay high? Why doesn't my net learn?
Neural nets are finicky. Try different network topologies. Try starting with a larger eta values and reduce it incrementally. It could also be due to redundancy in the input data, or mislabeled target output values. Or you may need more training samples.
What other parameters do I need to know about?
Check out the list of parameters in the wiki.
The neural2d program and its documentation are copyrighted and licensed under the terms of the MIT license.
The set of digits images in the images/digits/ subdirectory are released to the public domain.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent