jorgenkg / python-neural-network

This is an efficient implementation of a fully connected neural network in NumPy. The network can be trained by a variety of learning algorithms: backpropagation, resilient backpropagation and scaled conjugate gradient learning. The network has been developed with PYPY in mind.

Neural network written in Python (NumPy)

This is an implementation of a fully connected neural network in NumPy. By using the matrix approach to neural networks, this NumPy implementation is able to harvest the power of the BLAS library and efficiently perform the required calculations. The network can be trained by a wide range of learning algorithms.

Visit the project page or Read the documentation.

The code has been tested.

Implemented learning algorithms:

• Vanilla Backpropagation
• Backpropagation with classical momentum
• Backpropagation with Nesterov momentum
• RMSprop
• Adagrad
• Adam
• Resilient Backpropagation
• Scaled Conjugate Gradient
• SciPy’s Optimize

Installation

pip install nimblenet

Requirements

• Python
• NumPy
• Optionally: SciPy

This script has been written with PYPY in mind. Use their jit-compiler to run this code blazingly fast.

Features:

• Implemented with matrix operations to ensure high performance.
• Dropout regularization is available to reduce overfitting. Implemented as desribed here.
• Martin Møller's Scaled Conjugate Gradient for Fast Supervised Learning as published here.
• PYPY friendly (requires pypy-numpy).
• Features a selection of cost functions (error functions) and activation functions

Example Usage

    from nimblenet.activation_functions import sigmoid_function
    from nimblenet.cost_functions import cross_entropy_cost
    from nimblenet.learning_algorithms import RMSprop
    from nimblenet.data_structures import Instance
    from nimblenet.neuralnet import NeuralNet


    dataset        = [
        Instance( [0,0], [0] ), Instance( [1,0], [1] ), Instance( [0,1], [1] ), Instance( [1,1], [0] )
    ]

    settings       = {
        "n_inputs" : 2,
        "layers"   : [  (2, sigmoid_function), (1, sigmoid_function) ]
    }

    network        = NeuralNet( settings )
    training_set   = dataset
    test_set       = dataset
    cost_function  = cross_entropy_cost


    RMSprop(
            network,           # the network to train
            training_set,      # specify the training set
            test_set,          # specify the test set
            cost_function,     # specify the cost function to calculate error
        )

python-neural-network's Issues

Theoretical background links in documentation

It would be very useful, especially for newcomers to neural networks, to have links pointing to the theoretical background for some of the concepts in the documentation, e.g. each of the activation functions, each of the learning algorithms, etc. Right now the documentation is very well written but more focused on how to use the functions available without giving some primer of what the concepts mean and differences between them

Attempting tranpose on int returned from linear_function

Traceback (most recent call last):
File "/neuralnet.py", line 101, in backpropagation
input_signals, derivatives = self.update( training_data, trace=True )
File "/neuralnet.py", line 436, in update
derivatives.append( self.layers[i][1](signal, derivative = True).T )
AttributeError: 'int' object has no attribute 'T'

The issue in this case is linear_function, which returns 1 if derivative = True, and obviously this cannot be transposed.

Does not work on machine

Tried pip install nimblenet, got this error:

Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/private/var/folders/84/6yjtvl8s52jc3k8lf41chkn40000gn/T/pip-build-rLoiFl/nimblenet/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/84/6yjtvl8s52jc3k8lf41chkn40000gn/T/pip-bVsSSv-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/84/6yjtvl8s52jc3k8lf41chkn40000gn/T/pip-build-rLoiFl/nimblenet

Error shapes not alligned when creating Instance with numpy array

It is quite common in data science apps to use numpy arrays and pandas dataframes. Unfortunately when you try to create Instance using numpy array of shape (1,x) your code creates in one of the places array of shape (nobs, x, 1) which does not work in dot when multiplied by array of shape (x, nlayers). It could be easily checked with the verify_dataset_shape_and_modify. Here's some snippet:

def mult(x):
     res = x[0]
     for a in x[1:]:
         res *= a
     return res
[...]
def verify_dataset_shape_and_modify(...)
    [...]
    data              = np.array( [instance.features.reshape((mult(instance.features.shape),) \
                                         if type(instance.features) == np.ndarray\ #more generalization perhaps?
                                         else instance.features\
                                         for instance in dataset ] )

whith this snippet even accidentally passing arrays of shape (x,1) will work.

Can't figure out how to use with csv files/numpy arrays.

How to measure accuracy?

Is there a function to measure accuracy?

Softmax Derivative

ImportError: No module named learning_algorithms

after installing nimblenet via pip or github and trying to run the example code, I got these:
Traceback (most recent call last): File "ann_nimble.py", line 3, in <module> from nimblenet.learning_algorithms import RMSprop ImportError: No module named learning_algorithms

Why?

Your package is py2.x and it installs on py3.x

Your package has syntax of standard python 2.x version, but probably has a fault in pypi configuration, since it downloads on python3.x, installs, and fails ungraciously.

The numeric gradient check failed!

Hello I'm checking the library and I got a problem when I set my own data.

I'm creating a network with 48inputs, 30hidden layes and 2 for output. My inputs consist on values between 0-100. When I run the examples with my training data, the programs stops working and I get the error: The numeric gradient check failed!. I printed the diferrents values on gradient() and check_gradient() and I got nan on many values.

Hope you can help me.

ImportError: No module named 'scaled_conjugate_gradient'

Using python 3 i get the following error:

Traceback (most recent call last):
File "example.py", line 12, in
from nimblenet.learning_algorithms import *
File "python-neural-network/nimblenet/learning_algorithms/init.py", line 2, in
from scaled_conjugate_gradient import scaled_conjugate_gradient
ImportError: No module named 'scaled_conjugate_gradient'

I got same error on my desktop but i don't know what i changed that it vanished but now on my laptop after installing i have same issue.

Copy-pasted example code from README

After 150.000 epochs I understood that this is not working as expected -- it doesn't diminish the error.

Support for python 3.+

I got the following:

Traceback (most recent call last):
  File "C:/Users/josea/Dropbox/ITBA/Redes Neuronales/ECG/src/main.py", line 5, in <module>
    from nimblenet.neuralnet import NeuralNet
  File "C:\Anaconda3\lib\site-packages\nimblenet\neuralnet.py", line 126
    print "[gradient check] Running gradient check..."
                                                     ^
SyntaxError: Missing parentheses in call to 'print'

This means of course that the program is for python 2.+ only.. will it be a version for python 3.+?

unable to save a model

When you try to save trained weights, it gives error as some parameters are not initialized.

Overfitting

First of all I take this opportunity to thank you about your library. It's been great and of great help!

2nd. I'm getting overfiitting. Around 5000 epochs the Current Error starts to rise so training the net is not useful anymore and unless I put a max iterations it will keep on forever.

My question is... is there an easy way of using early stop or another anti-overfitting/regularization technique? (Although the library is very well documented I cannot find anything of the sort).

Thank you again.

Example tests on training set

The example tests on the same dataset that it trains on. This makes it liable to overfitting.

There's no way around it for this specific example, unfortunately; perhaps the example could be changed to make it possible to demonstrate without having this flaw?

Support of MLP Network with 2 or >2 hidden layers

Good afternoon, dear respective jorgenkg!

If is it possible may I ask following, would it be possible to set MLP with more than 2 or >2 hidden layers in your code. For example if I need to create MLP with 4 layers (1 input layer, 2 hidden layers, 1 output layer), each layer have 3 neurons, 3 inputs at input layer, 3 outputs on the output layer. In main.py I should set following:
"n_inputs" : 3, # Number of network input signals
"layers" : [ (3, tanh_function), (3, tanh_function),(1, sigmoid_function) ], ?

Could you please say would it be right?

Could you please say in your question at http://stackoverflow.com/questions/17049321/mlp-neural-network-calculating-the-gradient-matrices you have enclosed Code which computes Error Gradient for MLP's with n hidden layers with n neurons, may I ask you how is it possible to launch it? I have seen there 3 tabs "My Python code (derived from Chuck Anderson's snippet):", "The NPNeuralNet class", "The final implementation", what should I do in order to launch it, I should perceive "My Python code (derived from Chuck Anderson's snippet):" as main? But in "The final implementation" there is no Class declaration, only Functions definitions (def forward(), def backprop(), def errorFunction(), def errorGradient() ), maybe I should transfer them in "The NPNeuralNet class", but then I do not see calling of all this functions in "My Python code (derived from Chuck Anderson's snippet):". Could you please say how can I launch program described in your question? Currently I am studying calculations of Error gradient with appliance of Partial derivatives I have done couple of calculations manually and wanted to check it with some program but on the internet there was no any frameworks for MLP's with support of n hidden layers, except your Code which I have found on stackoverflow, as in "The NPNeuralNet class" I have seen that it is possible to set n numbers of hidden layers.

self.n_inputs = n_inputs
self.n_hiddens = n_hiddens
self.n_outputs = n_outputs
self.n_hidden_layers = n_hidden_layers

Thank you in advance for your reply!

Sincerely

There are so many problems

1.are u sure u are batch training?
2.look at at ur backpropagation,ur inputs actually are the outputs after every activations.but u regard them as the inputs again.it's wrong.
3.ur delta actually is dL/dx,but if u use L2 loss,the delta is y(1-y)*(t-y).if u use softmax as output,then the delta is out-targets.

Getting "nan" error

I am trying to implement a simple example with scaled conjugate gradient. This is my code

`
dataset = [Instance( [0,0], [0] ), Instance( [1,0], [1] ), Instance( [0,1], [1] ), Instance( [1,1], [0] )]
settings = {
"n_inputs" : 2,
"layers" : [ (2, sigmoid_function), (1, sigmoid_function) ]
}

network        = NeuralNet( settings )
training_set   = dataset
test_set       = dataset
cost_function  = cross_entropy_cost

scaled_conjugate_gradient(
# Required parameters
network,                     # the neural network instance to train
training_set,                # the training dataset
test_set,                    # the test dataset
cost_function,               # the cost function to optimize

# Optional parameters
ERROR_LIMIT          = 1e-3, # Error tolerance when terminating the learning
max_iterations       = (),   # Regardless of the achieved error, terminate after max_iterations epochs. Default: infinite
print_rate           = 1000, # The epoch interval to print progression statistics
save_trained_network = False # Whether to ask the user if they would like to save the network after training
)

`

The output at each epoch looks something like this:

[training] Current error: nan Epoch: 1000

It never changed from nan and I can't figure out why

Neural Network Support for python 3.5?

Does this program have support for Python 3.5?

If not, I will be pushing new commits with version changes needed.

can be use with integer/float > 1 ?

Hello, it's possible to use your neural network for addition of integer/float with result > 1 ?
like this :

training_set_inputs = array([[2,1],[3,2],[1,4],[4,3],[2,4]])
training_set_outputs = array([[3,5,5,7,6]]).T

when i search 1+3:
neural_network.forward_pass(array([1,3]))

i have in output 0.99, what do I have to change in the code to do that?

How do I import a CSV file

Sorry I can't seem to find in the documentation...

Wrong error assignment in resilient backpropagation

On line 230 in resilient_backpropagation you assign,
error = error, I believe it should be prev_error = error