Resulted CodeBook Re-use as Initialization for Next Training Cycle about somoclu

[peterwittek]

Probably not, unless you are extremely careful about all of the parameters that control learning (starting and finishing learning rate, starting and finishing radius).

[peterwittek]

Another thing to keep in mind is that you must have a deterministic initialization of the initial codebook; the default being random. Furthermore, for the radius parameters, only integer values are accepted. If you choose linear cooling, at low level, this is how the value is calculated (this is for both the learning rate and the radius):

float linearCooling(float start, float end, float nEpoch, float epoch) {
    float diff = (start - end) / (nEpoch - 1);
    return start - (epoch * diff);
}

Here start is the starting value, end is the final value, nEpoch is the total number of training epochs requested, and epoch is the current epoch.

[sunbc0120]

Awesome.

Gonna try on this and report my result.

[peterwittek]

Actually, I have no idea why the denominator had nEpoch-1: this will make the training overshoot. Commit f21cf01 fixes this. With this, the following Python code gives identical codebooks:

import somoclu
import numpy as np

data = np.float32(np.random.rand(50, 2))
n_rows, n_columns = 30, 50
som_a = somoclu.Somoclu(n_columns, n_rows, data=data, initialization="pca")
som_a.train(epochs=10, radius0=10, radiusN=1, scale0=0.1, scaleN=0.01)
som_b = somoclu.Somoclu(n_columns, n_rows, data=data, initialization="pca")
som_b.train(epochs=5, radius0=10, radiusN=6, scale0=0.1, scaleN=0.064)
som_b.train(epochs=5, radius0=5, radiusN=1, scale0=0.055, scaleN=0.01)
print(np.any(som_a.codebook != som_b.codebook))

At least most of the time. The single-precision floats allow for some uncertainty to creep in, but this is by design: SOM is a qualitative method.

[sunbc0120]

What is wrong that I did? and/or How to improve this?

import somoclu
import numpy as np

data = np.float32(np.random.rand(900, 3))
n_rows, n_columns = 30, 50

step_total = 20
checking_point = [3,5,7,13,17,19,step_total]

codebook_trajectory = []
count = 0

som_a = somoclu.Somoclu(n_columns, n_rows, data=data, initialization="pca")
som_a.train(epochs=step_total, radius0=0, radiusN=1, scale0=0.1, scaleN=0.01)
ref = som_a.codebook

def linear_cooling_rate(epoch, start=[np.round(np.minimum(n_rows, n_columns)/2),0.1], end=[1,.01],nEpoch=step_total):
    diff = np.subtract(start, end)/nEpoch
    new = start - (epoch * diff)
    new[0] = np.round(new[0])
    return new


for index,interruption in enumerate(checking_point):

    if index > 0:
        last_checkpoint = checking_point[index-1]
        little_steps = checking_point[index]-last_checkpoint
        som_b = somoclu.Somoclu(n_columns, n_rows, data=data, initialcodebook=som_b.codebook)
    else:
        little_steps = interruption
        last_checkpoint = 0
        som_b = somoclu.Somoclu(n_columns, n_rows, data=data, initialization="pca",  initialcodebook=None)

    pars_linear = linear_cooling_rate(interruption)
    pars_last = linear_cooling_rate(last_checkpoint)

    som_b.train(epochs=little_steps, radius0=int(pars_last[0]), radiusN=int(pars_linear[0]), scale0=pars_last[1], scaleN=pars_linear[1])
    codebook_trajectory.append(som_b.codebook)

print(np.any(np.around(som_a.codebook.astype(float),decimals=3) != np.around(som_b.codebook.astype(float),decimals=3)))

No much changes:

for codebook in codebook_trajectory:
    plt.imshow(codebook[:,:,2])
    plt.show()

(Much?) difference:

def rsquare(*vec):
    return np.sum(np.power(vec,2))

diff = som_a.codebook-som_b.codebook
r2 = np.array(map(rsquare,diff.reshape(n_rows*n_columns,3)))

plt.imshow(r2.reshape(n_rows,n_columns))
plt.show()

[peterwittek]

One obvious thing is that in the next round, you should have the starting radius and learning rate calculated at last step + 1 (see my example). I am uncertain whether the rounding of the radii will affect the result. Also, after the first iteration, it is unnecessary to create the som_b object again and again, although in principle this should have no bearing on the result.

[sunbc0120]

Really got confused with the number:

• radius0=10, radiusN=6 (difference is 4) vs. radius0=5, radiusN=1 (difference is 4). These two differences are consistent as each SOM is trained with 5 epochs.
• scale0=0.1, scaleN=0.064 (difference is 0.036) vs. scale0=0.055, scaleN=0.01 (difference is 0.045). This is a really weird "linear" to me. How you compute 0.064 and 0.055? Even more weird, SOMs match with each other under these 2 numbers.

I feel nEpoch-1 is correct as it satisfies the boundary condition 10 to 1, and 0.1 to 0.01.

def linear_cooling_rate(epoch, start=[10,.1], end=[1,.01],nEpoch=step_total):
    diff = np.subtract(start, end)/(nEpoch-1)
    new = start - (epoch * diff)
    new[0] = int(new[0])
    return new

for i in range(10):
    print i,linear_cooling_rate(i)

0 [ 10.    0.1]
1 [ 9.    0.09]
2 [ 8.    0.08]
3 [ 7.    0.07]
4 [ 6.    0.06]
5 [ 5.    0.05]
6 [ 4.    0.04]
7 [ 3.    0.03]
8 [ 2.    0.02]
9 [ 1.    0.01]

But this will not give 0.064 and 0.055.

[peterwittek]

You are right, the denominator should be nEpochs-1, I reverted that. Then I tried this:

som_b = somoclu.Somoclu(n_columns, n_rows, data=data, initialization="pca")
som_b.train(epochs=5, radius0=10, radiusN=6, scale0=0.1, scaleN=0.06)
som_b.train(epochs=5, radius0=5, radiusN=1, scale0=0.05, scaleN=0.01)

So this gives you the right step size (1 and 0.01 for the radius and the learning rate, respectively). I did a few runs, and som_a and som_b seem to be equivalent.

[sunbc0120]

All right, thanks very much.