5 What does multivariate decoding reveal about NCCCs?
The importance of information theory for understanding the neural correlates of consciousness has been stressed repeatedly, most notably by Giulio Tononi (2005). His information integration theory focuses on the information-based properties of neural processes and uses these special properties to provide a general explanation of consciousness. In contrast, the multivariate decoding account presented here attempts to solve the much more basic question of which neural populations provide the best account for which visual experiences. As mentioned above, this can be thought of as a search for the core neural correlates of the contents of consciousness (NCCCs), which have been postulated in similar forms by previous authors (Chalmers 2000; Block 2007; Koch 2004). While these proposals for core NCCCs have been influential in theoretical discussions on consciousness, they have only rarely been directly linked to neuroscience research, which requires spelling out how the NCCC can be established in empirical data. In the following, various studies of multivariate decoding from our lab will be presented that have implications for identifying NCCCs.
5.1 Example 1: Encapsulated information in V1
There has long been a debate as to whether the primary visual cortex (V1) is a neural correlate of visual consciousness. Crick & Koch (1995) postulated that V1 does not encode visual experiences for several reasons. First, V1 does not have the anatomical projections to the prefrontal cortex that would allow for a direct read-out of information in V1. This would be required to explain a key distinguishing feature of conscious experiences: that we can voluntarily act upon them. A second reason is that V1 encodes information of which we are not aware. Psychophysical experiments, for example, show that V1 can encode orientation information of which we are not aware (He et al. 1996). We thus directly assessed the link between information encoding in V1 and visual awareness (Haynes & Rees 2005). Specifically, we investigated the effects crossing the threshold to awareness has on the neural coding of simple visual features. Participants viewed oriented “grating” images (Figure 5) and had to tell whether they were tilted to the left or to the right. In one condition the images were clearly visible, in the other condition they were rendered invisible by rapidly alternating the orientation stimulus with a mask. In this condition participants were not able to tell the difference between the two orientation stimuli (Figure 5).
We then applied a classifier to fMRI-signals from early visual regions V1, V2, and V3 to see if it would be possible to decode the orientation of stimuli. We found that orientation for the visible stimuli could be decoded from all early visual regions, V1, V2, and V3 (Figure 5, right). This is in line with previous research on encoding of orientation information in early visual areas (Bartfeld & Grinvald 1992). Interestingly, we were able to decode the orientation from V1 even for invisible stimuli. This means that V1 presumably continues to carry low-level feature information even when a participant can’t access this information. V2 and V3, however, only had information for visible stimuli, not for invisible stimuli. Please note that an alternative interpretation could be that subjects perceive the subtle differences between masked stimuli, but they cannot report or reason about them. However, in psychophysics an absence of discriminability is typically considered a strong criterion for absence of awareness. This finding is interesting for several reasons. First, it demonstrates that information can be encapsulated in a person’s early visual cortex, without them being able to access this information. This suggests that V1 is not an NCCC for conscious orientation perception. Second, it shows that one explanation why stimuli are rendered invisible by visual masking is that the information that is available at early stages of processing (V1) is not passed on to the next stages of processing in V2 and V3. Similar encapsulation of information has also been observed for parietal extinction patients in higher-level visual areas with more conventional neuroimaging approaches (Rees et al. 2000).
Figure 5: Decoding the orientation of invisible grating stimuli from patterns of activity in early visual areas. Target stimuli were line patterns that were either tilted top left to bottom right, or top right to bottom left. They were rapidly alternated with mask stimuli so that participants were unable to identify the target orientation. The classification accuracy for these “invisible” gratings was above chance in area V1, but not in V2 or V3. For visible orientation stimuli the classification was above chance in all three early visual areas (figure taken from Haynes & Rees 2005).
5.2 Example 2: Imagery and perception
There has also been a long debate on the neural mechanisms underlying visual imagery. One important question is whether the NCCCs underlying imagery are the same—or at least overlapping—with those for veridical perception. One study (Kosslyn et al. 1995) found that imagery activated even very early stages of the visual cortex. This fits with a mechanism that encodes visual images as a replay of representations of veridical percepts. However, this does not reveal whether the activation of the early visual cortex really participates in encoding the imagined contents. Instead, these regions might be involved in ensuring the correct spatial distribution of attention across the visual field (Tootell et al. 1998). The question of whether the neural representations of veridical percepts are the same as those for visual imaginations needs to be established in addition.
Figure 6: Visual imagery. (a) Visual stimuli used in the experiment consisted of three selections from four categories. (b) In different trials participants either saw the images to the left or right of fixation or they received an auditory instruction to imagine a specific image. (c) A classifier trained on the brain responses of different imagined images could be used able to correctly cross-classify which image a person was currently seeing on the screen in the perception condition. Information was higher for the images “preferred” by a visual area, but there was still information, esp. in FBA, about the non-preferred categories (FFA=fusiform face area; OFA=occipital face area; FBA=fusiform body area; EBA=extrastriate body area; PPA=parahippocampal place area; TOS=transverse occipital sulcus)(figure from Cichy et al. 2012).
We conducted a study to directly address the overlap of NCCCs for veridical perception and imagery (Cichy et al. 2012). Participants were positioned inside an MRI scanner and had to perform one of two tasks: Either they were asked to observe visual images presented to the left or right of fixation (Figure 6), or they were asked to imagine visual images in the same locations. Twelve different images from four categories were used: three objects, three visual scenes, three body parts, and three faces. We found that multiple higher-level visual regions had information about the images. Furthermore, it was possible to decode seen visual images using a classifier that had only been trained on imagined visual images. This suggests that imagery and veridical perception share similar neural representations for perceptual contents, at least in high-level visual regions. Please note, however, that the cross-classification between veridical perception and imagery is not perfect. It is currently unclear whether this reflects imperfections in the measurement of brain signals with fMRI, or whether it reflects residual differences in the contents of consciousness between imagery and veridical perception, for example the higher vividness of perception based on external visual stimuli (Perkey 1910).
5.3 Example 3: Perceptual learning
Another interesting riddle of sensory awareness is perceptual learning (Sagi 2011; see also Lamme this collection). When we are first exposed to a novel class of sensory stimuli our ability to differentiate between nuances is highly limited. When one tastes the first glass of wine, all wines taste the same. But with increased exposure and experience we learn to distinguish even subtle differences between different wines. The interesting question here is whether the sensory information was there all along, and we just failed to notice it, or whether the sensory representation of the wines actually improves (see Dennett 1991).
We addressed this question, but with visual grating stimuli instead of different wines (Kahnt et al. 2011). Participants performed two fMRI sessions, where they had to distinguish small differences in the orientation of lines presented on the screen. They had to tell whether they were rotated clockwise or counter-clockwise with respect to a template. During the first fMRI session their ability to distinguish between the line patterns was quite limited. Afterwards we trained them in two sessions outside the MRI scanner on the same line patterns, and their performance continually improved. In a final second fMRI session they had then substantially improved their ability to tell even subtle differences between the orientations apart. But what explains this improvement: Better sensory coding, or better interpretation of the information that was there all along?
To address this question we first looked into the responses in the early visual cortex to the different line stimuli. As expected from our abovementioned study on orientation coding (Haynes & Rees 2005), it was possible to decode the orientation of the line elements from signals in early visual areas. It is well established that these areas have information about such simple visual features (Bartfeld & Grinvald 1992). However, we found no improvement in our ability to decode the orientation of the stimuli with learning. There is some divergence in the literature with some studies finding effects of learning in early sensory areas (see Sasaki et al. 2010). Other recent findings in monkeys are in line with our findings and do not find improved information coding in sensory areas (e.g., Law & Gold 2009). In our case, it seems as if the sensory representation of orientation remains unchanged and that some other mechanism has to be responsible for the improvement in perceptual discrimination. We found a region in the medial prefrontal cortex where signals followed the learning curve, thus suggesting that the improvement was not so much a question of stimulus coding but of the read-out of information from the sensory system. This study suggests that representation of a feature in an NCCC might not automatically guarantee it enters visual awareness.
5.4 Example 4: Invariance in human higher-level visual cortex
As mentioned above, one important challenge to the idea that the contents of visual awareness are encoded exclusively late in the visual system is the invariance of responses to low-level visual features (Sáry et al. 1993). We directly investigated the invariance of fMRI responses in the regions lateral occipital (LO) and fusiform gyrus (FUS) of the higher-level object-selective visual cortex (Malach et al. 1995; Grill-Spector et al. 2001). In this study (Cichy et al. 2011) participants viewed objects presented either to the left or the right of the fixation spot (Figure 7). These objects consisted of three different exemplars from four different categories (animals, planes, cars, and chairs). For example, the category “animal” contained images of a frog, a tortoise, and a cow. With these data we were able to explore two different aspects of invariance. First, we wanted to know whether object representations are invariant to changes in spatial location. This is important because a low-level visual representation that focuses exclusively on the distribution of light in the visual field would not be able to generalize from one position to another. So we assessed whether a classifier trained to recognize an object at one position in the visual field would be able to generalize to another position in the visual field. We found that a classifier was able to generalize to a different position, however with reduced accuracy. This indicates that the representations were at least partially invariant with respect to low-level visual features. Next, we investigated whether the representations would generalize from one exemplar to another. This goes even further in testing for the level of abstraction of the representation. A classifier that can generalize not only to a different location but even to a different exemplar (say from a frog to a cow) needs to operate at a higher level of abstraction that is largely independent from low-level visual features. Again we found that the classifier was able to generalize between exemplars of the same category, further supporting the abstraction of representations in the higher visual regions LO and FUS (Figure 7). This makes it again less plausible that the contents of visual awareness are encoded exclusively in the higher-level visual cortex. Encoding in these regions is invariant (or at least tolerant) to low-level feature changes, and thus this level of perceptual experience has to be encoded at a different, presumably lower, level of visual processing.
Figure 7: FMRI evidence for invariance of object-representations in the high-level visual regions lateral occipital (LO) and fusiform gyrus (FUS) as compared to early visual cortex (EV; figure from Cichy et al. 2011).
5.5 Example 5: No sensory information in PFC
A further case where multivariate decoding might inform theories of visual awareness becomes apparent when we confront the question of whether sensory information is distributed throughout the brain when a stimulus crosses the threshold of awareness. The global neuronal workspace theory (e.g., Dehaene & Naccache 2001; see also Baars 2002) posits that sensory signals are made globally available across large-scale brain networks, especially in the prefrontal and parietal cortices, when they reach awareness. An interesting and open question is whether this global availability of sensory information means that the sensory information about a stimulus can be actually decoded from these prefrontal and parietal brain regions to which the information is made available. In theory, one might be able to distinguish between a “streaming model” of global availability, where information is broadcast throughout the brain (e.g., Baars 1988), and which should thus be decodable from multiple brain regions; an alternative would be an “on demand” model of global availability, where sensory signals are only propagated into prefrontal and parietal cortex when selected by attention (e.g., Dehaene & Naccache 2001).
We performed three fMRI studies to test this question (Bode et al. 2012; Bode et al. 2013; Hebart et al. 2012). In the first study (Bode et al. 2012), participants were briefly shown images of pianos and chairs that were temporally embedded in scrambled mask stimuli. There were two conditions. In one condition, the timing of visual stimuli was chosen such that the target stimuli were clearly visible. In the other condition, the timing of scrambled masks and targets was such that the targets were effectively rendered invisible. We attempted to decode the sensory information about the presented objects. Under high visibility we were able to decode which image was being shown from fMRI signals in the so-called lateral occipital regions of the human brain, where complex object recognition takes place. Under low visibility, there was no information in these brain regions. This suggests a possible mechanism for explaining why the stimuli failed to reach awareness. Presumably their sensory representations were already cancelled out at the visual processing stages. The “streaming model” mentioned above would mean that sensory information about the object category is distributed into parietal and prefrontal brain regions when the stimulus crosses the threshold of awareness. However, we found no information in the prefrontal cortex—under either high or low visibility (Bode et al. 2012). This finding was repeated in two different studies, one also using objects as stimuli (Bode et al. 2013) and one using drifting motion stimuli (Hebart et al. 2012). In contrast, in animal studies sensory information has been found in the prefrontal cortex (Pasternak & Greenlee 2005). It is currently unclear whether this reflects a species-difference or whether it is due to limitations in the resolution of human neuroimaging techniques.
5.6 Example 6: Unconscious processing of preferences
It is well known that unattended and even invisible visual stimuli can undergo substantial processing. We investigated whether information about high-level, more interpretative and subjective properties of visual stimuli would also be traceable using decoding techniques. For this we aimed to decode the degree to which preferences for certain visually presented images of cars can be decoded, even when these stimuli were unattended and were not task-relevant (Tusche et al. 2010).
For this experiment we carefully pre-selected our participants, who were self-reporting car-enthusiasts. Then we ensured that we chose stimuli where different participants had maximally-divergent opinions as to which car they preferred. This was necessary in order to de-correlate the classification of the preference from the classification of the specific vehicles. Subjects were divided into two groups. Participants from the first group were presented with the car images in the scanner and had to actively evaluate whether they liked them. The second group was also presented with the car images, but they were distracted from them. They were required to solve a very difficult task that required them to focus their attention elsewhere in the visual field, on fixation. The car stimuli were thus task-irrelevant and presented outside of the attentional focus. This group of subjects could not recall which cars had been shown during the experiment, suggesting that they were indeed not actively deliberating about the cars. After the experiment, participants from both groups were asked to rate how much they would like to buy each car. This served as a gold standard for their preference.
We then tried to decode whether individual subjects liked the cars or not. For this, we looked into patterns of brain activity throughout the brain, to see where there might be information regarding preferences. This was done in order to reduce the bias when only looking into pre-specified brain regions. We found that it was possible to decode the preferred cars with 75% accuracy from brain regions far outside the visual system, in the medial prefrontal and in the insular cortex. This was true for the subjects who had been actively deliberating about their preferences for the cars, but also for the participants who been distracted from thinking about them. Presumably, this means that the brain automatically processes the car-images all the way up to the stage of encoding preferences, even in the absence of visual attention. Please note that this finding of preference information in the prefrontal cortex is quite different to that in the previous experiment, where there was no sensory information in PFC. Here, in contrast, there is information in PFC, but (a) not about a sensory property and (b) even for unattended stimuli. Thus, it appears that the informational dividing line between sensory and prefrontal brain regions is not one of awareness, but rather one of the type of information coded.