This study brings major advances in encoding and decoding cortical activity that supports human natural vision. For encoding, it demonstrates the unique promise of using deep learning to model and visualize the functional representations at the level of single cortical locations along the entire visual pathway, and to create a computational workbench for high-throughput vision research. For decoding, the study presents a stand-alone, efficient, reliable, and generalizable strategy to decode cortical fMRI activity to directly reconstruct the visual and semantic experiences during natural vision. These unique capabilities highlight a promising emerging direction of using the artificial brain to under-stand the biological brain.

The Laboratory of Integrated Brain Imaging (LIBI) at Purdue University acquired 3T fMRI responses from three subjects during watching natural movies (Wen et al., 2017). The movie stimuli contain diverse yet representative of real-life visual experiences, e.g. people in action, moving animals, nature scenes, outdoor or indoor scenes etc. The stimuli include two sets of movie segments: 1) 18 training movie segments, and 2) 5 testing movie segments. The duration of each segment is 8 minutes. During each fMRI scanning session, one segment was presented to the subjects. For each subject, the training movie segments were presented twice and the testing movie segments were presented ten times. In total, there are 11.47 hours of fMRI responses to 3.07 hours of movie stimuli for each subject.

This is publication series that contains data from three subjects, datasets with the stimuli and source code.

Wen, H., Shi, J., Zhang, Y., Lu, KH., Cao JY. & Liu, Z. (2017). Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision. Cerebral cortex. In press.

• Data for Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision Tests - Stimuli (v1.0)

  This is a video-fMRI dataset contains the videos with stimuli acquired by the Laboratory of Integrated Brain Imaging (LIBI).

• Data for Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision Tests - Subject 1 (v1.0)

  This is a video-fMRI dataset for subject 1 (out of three) acquired by the Laboratory of Integrated Brain Imaging (LIBI),

• Data for Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision Tests - Subject 2 (v1.0)

  This is a video-fMRI dataset for subject 2 (out of three) acquired by the Laboratory of Integrated Brain Imaging (LIBI).

• Data for Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision Tests - Subject 3 (v1.0)

  This is a video-fMRI dataset for subject 3 (out of three) acquired by the Laboratory of Integrated Brain Imaging (LIBI).

• Source code for Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision (v2.0)

  This document includes the main source code (Matlab or Python) related to our study.

Researchers should cite this work as follows:

• Wen, H.; Shi, J.; Zhang, Y.; Lu, K.; Cao, J.; Liu, Z. (2017). Data for Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision Tests. Purdue University Research Repository. doi:10.4231/R7SF2TCW

  BibTex | EndNote

1. Biomedical Engineering
2. Biomedical Engineering (Dept)
3. deep learning
4. fMRI
5. LIBI
6. natural vision
7. neural decoding
8. Neural encoding
9. Neuroscience