Alternative SOM computation methods. about somoclu

[peterwittek]

Somoclu has a long history. Its immediate predecessor was developed in 2011, but it was based on a previous distributed implementation. Back then Theano wasn't that visible.

Somoclu is both parallel and distributed, and in that sense, it is more scalable than Theano. I haven't looked into TensorFlow yet, but as far as I understand, it also focuses on single-node parallelism. Nevertheless, it would be great to see a performance comparison on a single node, especially on emergent maps, which are Somoclu's forte.

You have a point that we could introduce Theano or TensorFlow as a dependency, but then it would become very complex to use the non-Python interfaces. Thrust is definitely a performance bottleneck in the GPU kernel, but the effort expended on rewriting the kernel is minimal: if we need more power, we can simply run the computation on a larger GPU cluster.

Tensorflow is in use by Google in their datacenters but they haven't released the code that allows it to scale across multiple machines, they plan to do so with an opensource variant. So it will be distributed in the coming months.

You said that 'effort expended', does this mean that you already got rid of Thrust? I'm not able to find that anywhere and the cuda branch seems dated and still uses Thrust.

[peterwittek]

Good to know that the multinode flavour of TensorFlow is coming soon.

The current CUDA branch is what I described as minimal effort. We started working on it, but never finished it. So that branch is half-CUDA, half-Thrust.

The immediate predecessor of Somoclu, MR-MPI-SOM-GPU used a pure CUDA kernel, but it had a number of issues. We wanted to salvage what we could, but then Somoclu was already scalable enough for everything we wanted to do, so we lost the movivation. Feel free to dive into it. Since the actual calculations are quite simple, it can't be too hard. The biggest chunk of calculations is done by a CuBLAS call, so I doubt if that could be faster. This leaves you with only two functions that use Thrust and the reason they are slow is because they use an iterator that slices up Thrust's row-oriented processing to a matrix operation.

One of the reasons I asked is because I actually want to implement a SOM in a unique way and I did not want to learn CUDA to do it.

It is common practice in data precprocessing for neural nets to sometimes use PCA to reduce the dimensionality by some factor. SOM however always reduces the dimensionaity to 2. SOM is vastly superior to PCA in that domain (2D).

Would you have any comment on implementing SOM such that the dimensionality of the output is arbitrary? What would be the challenge of a 100D SOM other than memory?

[peterwittek]

The highest dimensional SOM I heard of was 3D. The problem is that computational and memory requirements grow really fast. Remember that we store a weight vector for each node of the same dimension as the original space. Let's simplify things and make our SOM a hypercube of 100 nodes in each direction. Since you want something in the range of a hundred dimension, assume that your original space has a lot more dimensions, say, 10000. Also assume that the space is dense.

Then, for 2D, you need 10010010000*4 bytes, assuming 32-bit floats to save space. That is still okay, we are below 400 Mbytes. 3D would give you 38 GBytes, plus you generally lose your ability to visualize what you got. You see where this goes. Then finding of the best matching unit for each data instance is also exponential in computational time in the number of dimensions.

Your point is valid, SOM is generally superior in 2D than PCA. (A side remark: it is especially superior if you initialize the codebook with PCA, and then train the map). This is because you capture the local topology of the points. SOM was conceived with visualization in mind, but in a sense you want less: you only need the local-topology preserving quality. Wouldn't Isomap or some form kernel PCA do that for you?