Dynamics of Scene Representations in the Human Brain revealed by MEG and Deep Neural Networks

Radoslaw M. Cichy
Aditya Khosla
Dimitrios Pantazis
Aude Oliva
MIT/Free University Berlin
MIT
MIT
MIT

Abstract

Human scene recognition is a rapid multistep process evolving over time from single scene image to spatial layout processing. We used multivariate pattern analyses on magnetoencephalography (MEG) data to unravel the time course of this cortical process. Following an early signal for lower-level visual analysis of single scenes at ~100ms, we found a marker of real-world scene size, i.e., spatial layout processing, at ~250ms indexing neural representations robust to changes in unrelated scene properties and viewing conditions. For a quantitative explanation that captures the complexity of scene recognition, we compared MEG data to a deep neural network model trained on scene classification. Representations of scene size emerged intrinsically in the model, and resolved emerging neural scene size representation. Together our data provide a first description of an electrophysiological signal for layout processing in humans, and a novel quantitative model of how spatial layout representations may emerge in the human brain.


[paper] [bibtex]
[supplementary materials]

MEG decoding accuracy matrices over time


The movie shows the MEG decoding accuracy matrices over time in 1 millisecond steps, accompanied with a two-dimensional scaling solution (criterion: metric stress). Experimental conditions (images) are plotted in the two dimensional space, color- and shape-coded to indicate scene size and clutter level.

Reference

Dynamics of Scene Representations in the Human Brain revealed by MEG and Deep Neural Networks
Radoslaw M. Cichy, Aditya Khosla, Dimitrios Pantazis and Aude Oliva
Neuroimage(in press), doi:10.1016/j.neuroimage.2016.03.063.
[paper] [bibtex] [biorxiv] [supplementary materials]