Investigating the difference in robustness between decoded brain (fMRI) representations versus those learned by a convolutional neural network using methods from computational neuroscience and deep learning.
Clone the repository.
git clone https://github.com/theairbend3r/decoding-thoughts.git
Create environment.
conda env create -f environment.yml
Install dependencies.
pip install -r requirements.txt
Install package.
pip install -e . # or make install
The main package resides in ./src/
.
Jupyter notebooks that use modules from the above package are in ./notebooks/
.
- Exploratory Analysis
- Stimulus Classifier
- fMRI Classifier
- Representational Similarity Matrix and Robustness
- Study the robustness of CNN models to increasing noise and compare the similarity of its learned embeddings to brain's representation of the same visual stimulus.
- Latent representations generated by the most robust CNN (Stimulus) model may be closest to the one generated by the DNN (fMRI/Brain) model.
- The kay dataset (Kay et. al., 2008) contains images and the corresponding brain fMRI readings collected from 2 human subjects who were shown a set of natural images while their brain activity was being recorded.
- Data shapes:
- fMRI: 8428 voxels
- Images: (128 x 128) pixels
- Train a 3 layer DNN to predict output labels using fMRI data.
- Finetune 5 CNNs (pretrained on ImageNet) to predict output labels using images.
- Extract embeddings for all images in test set from the trained models (CNN and DNN).
- Perform representation similarity analysis using cosine-similarity metric for each model using its learned embeddings.
- Add increasing levels of gaussian blurring to the images as they are passed through these networks and capture the accuracy.
- Compute the correlation between the representational similary matrix (RSM) (computed above) for fMRI (DNN) vs Stimulus (CNN1, CNN2, ..., CNN5) models.
- Observe the relationship between correlation coefficients calculated betwee RSMs (fMRI vs Stimulus) and the CNN model (Stimulus) that drops the most accuracy for noisy input.
- See notebooks in the
./notebooks/
directory.
Akshaj Verma – @theairbend3r
Distributed under the GNU GPL-V3 license. See LICENSE for more information.