stefanches7 / deepvisual

Reconstruct visual stimuli from EEG data using Deep Learning

DeepVisual - EEG-based picture reconstruction

Connect

Goals

Tier 1 goal is to predict seen image category based on EEG time series produced during visual perception of the former. The ultimate goal is try to reconstruct whole image (i.e., pixel to pixel) from EEG time series. Here, we are using Deep Learning approach.

Data

Data suitable for the first goal is e.g. from Rashkov G. et al. (2019) https://doi.org/10.1101/787101. Dataset is openly accessible at https://data.mendeley.com/datasets/s2dxrv45fr/1. Pitily, the researchers couldn't include whole shown video clips because of copyright issues, only screenshots - so the second goal is only achievable after some data transformation. The second goal affine dataset should at some point be uploaded by instigators of Brainhack Geneva 2019 project (see Credentials) and be made publically available. We are welcoming any clues about further suitable dataset - don't hesitate to drop a message on Mattermost channel, or open an issue here!

Methods

Year 2019 appears to be bursting even with review articles on Deep Learning using EEG data. One of the reviews finds CNN, Deep Belief Networks and Multilayer Perceptrons to be the most often used models for learning on EEG data. If we consider "emotion recognition" task setup to naturally correlate with at least the first task setting (classification of seen image), it is indeed interesting to try Deep Belief Networks on our data.

Credits

This project was begun during Brainhack Geneva 2019. Credits for the scientific idea go to the Team 10 of instigators of Brainhack Geneva 2019 (Konstantinos S.-T., Timokleia K., Florian L. and Carolina L.). Technical idea and way of implementation is of the owner of this repository, Stefan Dvoretskii.

deepvisual's Issues

Achieve image classes reconstruction based on EEG signal

Data from the study by Rashkov G. et al. (2019) is accessible at https://data.mendeley.com/datasets/s2dxrv45fr/1. In it, EEG signals are annotated with information about the one of the classes of visual stimulus shown to the participant (animal movement, abstract figures etc. - refer to study for more details).
The first and most simple goal is to predict seen image class based on EEG signal during the viewing.

TODO:

• Extract sequences, assign session and seen image class
• Signal aggregation - e.g. Discrete Wavelet Transformation
• Deep Learning model
  Currently MLP, but can try other e.g. Deep Belief Networks
• Test/train split
• Prediction - does not finish on laptop, have to scale
• Evaluation of results

Hello!

I'm glad to see you as collaborator in this scientific effort to recover this visual data from EEG signal. Here I suggest the list of things to-do for better understanding of the project:

• Read https://journals.plos.org/ploscompbiol/article?id=10.1371%2Fjournal.pcbi.1006633 - main idea
• Read https://journals.plos.org/ploscompbiol/article?id=10.1371%2Fjournal.pcbi.1006633 - data and method

EEG signal transformation

At the end, EEG signal can be represented as n*m matrix, where n is the number of channels in EEG helmet (64 or 128) and m is the number of taken discrete time points.
Usually, signals are being transformed to add information quotient/distropy to each feature (Physics specialists are welcome to tell more!). We might want to also pick one transformation method and see whether it helps our pipeline.
TODO

• (newcomer suggested) Read https://doi.org/10.1101/787101 on tranformation before ICA
• (newcomer suggested) Read about Discrete wavelet transformation
• PIck transformation method
• Evaluate compared to baseline (no transformation)