fklepel / thesis_capsnet_perception-imagery Goto Github PK

This folder contains all the relevant code and data for the master's thesis of Florentine Klepel about 
"Learning Equivariant Object Recognition, and its Reverse Application to Imagery".

Abstract:
This thesis aims to model the visual ventral stream during perception and imagery with the help of capsule networks. The proposed network consists of V1 and V2 from CorNetZ, as well as the Capsule Network architecture with the routing by agreement algorithm. The decoder then reverses this architecture to model the feedback activation patterns of the visual ventral stream. While high classification performance is reached, generalisation performance to different sizes, positions, and rotations is maintained. Due to high variability, reconstruction quality was restricted in some conditions. Surrounding information was used in the feedback path for reconstructions so that reconstructions could be correctly classified. Additionally, a pre-trained network was used to reconstruct remapped fMRI activation patterns from higher visual areas. Reconstructions of single-trial imagery data showed significant correlations to physical letter stimuli. The fMRI activation patterns of V1 and V2 and their reconstructions with population receptive field mapping and an autoencoder were related to activation patterns of the network to test biological plausibility. Representational Similarity Analysis and spatial correlations indicated an overlap of information content between the capsule network and the fMRI activations. Overall, this network sets a promising path for increased generalisation ability because of its focus on figure-part relationships while maintaining biological validity. Due to the capsule networks' high generalisation performance and the implemented feedback connections, the proposed network is a promising approach to improve current modelling efforts of perception and imagery. Further research is needed to compare the presented network to established networks that model the visual ventral stream as well as show that it is crucial to model the human brains' generalisation ability in order to model perception and imagery processes.

Unfortunately, some of the files are very large. That is why some versions of the files had to be zipped.
In case you encounter any issues using the code, email me at: [email protected]