bicv / sparsehebbianlearning Goto Github PK

be careful
the " +1 " in

ceil = P_cum.ravel()[indices + 1 - 2 * (p_c == 1) + stick]

generates an error so that when indices[-1]=511 you get the error

IndexError: index 51200 is out of bounds for axis 1 with size 51200

it also means that when indices[0]=511 then ceil[0]=0 instead of ceil[0]=1

The Adam method adapts the step size depending on the amplitude of the gradient:

https://arxiv.org/pdf/1412.6980.pdf

proves to be faster the traditional fixed-step on-line stochastic gradient descent

Is it possible in our current implementation to implement the heuristic behind Equi-probable MP by using a binary vector gain = (P < p) where

This basically blocks those which are "too rich" and lets the other learn. For a given input, the equilibrium probability should be p_eq = L0_sparseness / M such that we may use something like p = p_eq * 1.1

the get_data function is present in the main scripts and in the tools scripts. We should just keep one... I suggest the one in the tools script.

the homeostasis using histogram equalization works and is great mathematically, but it slows down things... we could as well do a gradient descent of an homeostatic cost - hence have something similar as the implementation of the homeostasis in Olshausen, but on the probability of activation instead of the variance.

• one possible implementation is along that in Sandin et al 2016

• perhaps doable there in one line : https://github.com/bicv/SHL_scripts/blob/master/shl_scripts/shl_learn.py#L442

• then we need a way to tell to the algo to chose one method or the other (without making the code even more baroque, sigh...)

it makes no sense of coding for DC componenets: filter everithing below a certain frequency given by the patches' size

potential source for novel equations = "A tutorial on the free-energy framework for modelling perception and learning" by Rafal Bogacz, http://www.sciencedirect.com/science/article/pii/S0022249615000759

to avoid problems on the corners, use a disk to remove that pixels.

in https://github.com/bicv/SLIP/blob/master/SLIP/SLIP.py#L219

we have an example of how to do it:

        self.x, self.y = np.mgrid[-1:1:1j*self.pe.N_X, -1:1:1j*self.pe.N_Y]
        self.R = np.sqrt(self.x**2 + self.y**2)
        if not 'mask_exponent' in self.pe.keys(): self.pe.mask_exponent = mask_exponent
        self.mask = ((np.cos(np.pi*self.R)+1)/2 *(self.R < 1.))**(1./self.pe.mask_exponent)
        

In the original SparseNet algorithm, the learning rule for the gain vector follows an heuristics which could be formulated in a classical formulation using an "homeostatic cost function".

The advantage would be that it would generate the different rules for coding, learning and homeostatic gain combined from one generic formulation of the gradient descent.

We need to find a more generic and large database than serre07_* or kodakdb - we should use that databases that are provided by machine learning , for instance:

• STL10: http://pytorch.org/docs/master/torchvision/datasets.html#stl10

may be we should include some part of the https://github.com/mttk/STL10/blob/master/stl10_input.py file into our scripts ?

best would be even more generic images from instagram etc...

Show that changing the gain changes the probability of firing

P_i versus p(a_i)

Make a theorem to predict outcome of the algorithm assuming existence of a solution

This happened with all the training I left running this weekend

Did you get anything like that? Do you think it was because of the Fast Pcum function?

We should concentrate on a few main points:

• a global unsupervised learning cost, and how we derive three costs (coding, learning, homeostasis)
• how we minimise the homeostatic cost (exact, gradient descent)
• how we do the learning and how this is relatively independent from the coding (lasso, MP, ...)
• quantification of the learning (not just the beauty of the filters)

So we should at the moment avoid talking about:

• dynamic streams,
• convolutions
• hierarchies

The learning appears to have some analogies with a complex dynamical system with different phases as a function of the parameters. We need to trace the evolution of some parameters during the learning.

One solution is to introduce a record_each parameter to the learn_dico function - if set to 0, it does nothing, else it records every record_each step the statistics during the learning phase (variance and kurtosis of coefficients to start off).

To get to that branch, use

git checkout fast_homeo

Then, we can experiment any different crazy idea

Some things to check, and which can be different separate issues:

• benchmark with respect to the full homeostasis - potentially with different codes
• benchmark to find optimal learning parameters
• derive the "learning rule" from theory
• compare results (dico) quantitatively

The code just works but could be improved by:

• vectorizing the MP
• simplifying learning
• profiling weak points
• porting numerical operations to GPUs using theano etc...

Some sources of inspiration:

• https://blog.keras.io/building-autoencoders-in-keras.html

https://en.wikipedia.org/wiki/Mutual_coherence_(linear_algebra)

to show the robustness of the learning we should iterate the learning multiple times in different epochs

1/f stats comes from averaging many images

one possibility is to average, something like
dictionary = np.array_split(X_train, n_dictionary).mean(axis=??)
around line https://github.com/bicv/SHL_scripts/blob/master/shl_scripts/shl_learn.py#L300

validate with a notebook in probe

• try it with no learning
• include it in code and run a basic learning

to re-generate figure 14.1 from https://www.invibe.net/LaurentPerrinet/Publications/Perrinet15bicv

• test with shape = (1, 256)
• make the plot function more intelligent