2 Consciousness as an inferential process and the consequences for the neural mechanism of conscious perception

One central and characteristic feature of conscious perception is its constructive nature. In contrast to unconscious cognition, which is directly driven by sensory stimulation, the images that reach consciousness often bear little resemblance to reality. Indeed, percepts in our mind can be understood as useful distortions of reality in which only specific parts of the physical input are represented while being enriched with a model of the world that has been learned and that provides context to the current moment. In the words of Heinz von Foerster (1984), “the world, as we perceive it, is our own invention”. To provide a striking example of this, consider the image on the right (Figure 1) and try to figure out what it shows. Most people at first see a collection of black and white blobs, much like the input that strikes our retina—a raw, uninterpreted signal. Now, rotate the page upside down. Voila! You will clearly see a face (do you recognize whose face it is?). Remarkably, you can turn the page back and you will continue seeing the face. Once you have recognized the image, the visual system has created a prior, an expectation that enriches perception. This example is not mere curiosity. Most of our behaviour and perceptions are based on predictions: we do not wait for visual input to impinge our eyes, we actively look for it. We cannot, however, initiate a rational search for an object without making predictions about “what” it is, “where” it is likely to be, and even “when” it is likely to be there. The brain’s ability to make predictions and to mould its data gathering accordingly is thus essential for its ability to evaluate options, make life-critical decisions, and generate adaptive behaviour.

Figure 1: Can you recognize what this is? If not, rotate the image. Note that once you turn it back around the object is now clear.

While the constructive nature of perception is undeniable and may even appear as one of its defining features, surprisingly little research has been carried out to understand how previous experience interacts with consciousness. Most importantly, the scientific community has not embraced an understanding of consciousness in the context of a flow of experience in which every moment is integrated with past moments and interfaced with expectations about what will happen in the future (but see Varela 1999). A possible reason for neglecting the contribution of previous experience is that this integration of past with present moments has been understood as a process of “unconscious inference” (following von Helmholtz 1866/1962), or, in Victor Lamme’s words, in the context of the “automaticity of the many expectation effects.” However, this inferential process is carried out in the backstage of consciousness, and it is only the result that we consciously experience. This bears resemblance to syntactic analysis, which is also carried out automatically and unconsciously, but is paramount to conscious access to meaning. Without unconscious syntactic analysis we would not be able to “consciously” understand text; nor is its automatic activation under our control. In the same vein, our conscious perception would be totally different if prior knowledge did not help us enrich or even construct our experience, endowing it with meaning. In fact, it has been proposed that alterations in perception, i.e., the defragmented sensory experience observed in schizophrenics and autistic people can be the result of a deficit in this inferential process (Jardri & Deneve 2013; Pellicano & Burr 2012), underscoring the fundamental role that perceptual inference plays in conscious perception.

One promising framework within which the influence of previous experience through unconscious inference can be understood is the Bayesian framework. When applied to perception, each mathematically-formulated ingredient of this framework can be assigned a perceptual counterpart, with previous experience referring to the prior, the current moment referring to the likelihood, unconscious inference referring to Bayes rule (which combines the prior with the likelihood in an optimal way), and the result—our perception—referring to (the peak of) the posterior distribution. This idea has recently proven to be a powerful tool for understanding perception not only in terms of modelling behaviour, but also as a theoretical framework for understanding how perception arises in the brain. A prominent implementation of the latter is Predictive Coding (Friston 2010). This theory postulates that the brain builds models (priors) of the world based on previous experience, which are used to explain the current inputs. This occurs iteratively across all levels in the cortical hierarchy with the goal of minimising predictions errors, i.e., the difference between what is expected and the incoming sensory input, which are energetically costly. This minimization process can either be achieved by changing the way the system samples its environment, or by changing its models. Relevant for this discussion is the idea that perceptual inference, in the Predictive Coding framework, implies that all levels in the hierarchy reach an agreement, i.e., minimise all prediction errors, much like the idea of a unified/integrative moment as proposed by Victor Lamme and others (Dehaene 2014; Edelman & Tononi 2000; Melloni & Singer 2010). While Predictive Coding by itself is currently agnostic as to whether such unified agreement represents a conscious state, the central tenet that integration across all levels is what the system strives for still holds. This allows for the formulation of interesting, testable predictions about the Neural Correlates of Consciousness (NCC).

In recent years research in my lab has focused on understanding how previous experience enriches perception, how expectations alter the NCC, and how this can be understood within the Predictive Coding framework. The central idea that motivated these studies was to test whether or not the NCC are context independent, i.e., impervious to the influence of expectations, as many theories implicitly postulate. To test this hypothesis we presented subjects with illusory letters, that is letters whose borders where not explicitly defined but instead required the activation of figure–ground segregation cues. We reasoned that providing subjects with a prior, i.e., knowing which letter would be presented next, would facilitate the figure–ground segregation process, making an initially invisible letter clearly visible. In line with our expectations, we observed that the threshold of conscious perception is not fixed but instead changes depending on the availability of previous knowledge: subjects are able to perceive a stimulus on the basis of minimal sensory information when they have a clear expectation. We were able to confirm this result in a series of different paradigms in which expectations could be generated online from recent experience as in the example of the letter given above (Melloni et al. 2011; Schwiedrzik et al. 2014), drawn from memory based on prior exposure to clearly visible natural images (Aru et al. 2012), stem from a life-long history of association between letters and colour as in grapheme-colour synaesthesia (van Leeuwen et al. 2013), or result from systematic training as in perceptual learning (Schwiedrzik et al. 2009, 2011). These studies allowed us to test not only whether the behavioural threshold of conscious perception is fixed, but also how previous knowledge would affect the neural “construction” of conscious percepts.

A first hypothesis we derived from the Predictive Coding framework was that the presence of strong priors should have an effect of how quickly content reaches awareness. If conscious perception is the result of a process that iterates until information is consistent between the different levels of the hierarchy (Di Lollo et al. 2000), i.e., until all prediction errors are minimised, then having a better model of the input based on prior knowledge may speed up this process. Indeed and contrary to the common belief that information processing in the brain has a fixed latency, we observed that the NCC shifts in time when a prior is available. While the electrophysiological difference between seen and unseen letters occurred around 300ms when it exclusively depended on sensory evidence, it occurred as early as 200ms when priors were available (Melloni et al. 2011). Thus, priors sped up information processing by 100ms. These results have important implications for the search for the NCC as they show that conscious processing is not bound to a particular time, but can flexibly adjust its timing depending on the task at hand, the readiness of the system, or the presence of expectations. They also pose a challenge to theories that postulate that the NCC always occur late, as proposed by Victor Lamme (this collection) or Stanislas Dehaene (2014).

A second prediction that follows from the principle of minimising prediction errors is that in the presence of priors, activity in lower areas can be “explained away” by priors in higher brain areas (Murray et al. 2004); this entails that when inputs can be fully predicted based on previous experience, they do not elicit prediction errors. To test this hypothesis, we took the same study to the MEG and performed source localisation. Here, we found that priors sparsify the networks involved in processing the stimulus, such that when a prior is present only the brain areas that are most diagnostic to the stimulus features are activated (Mayer et al. in preparation). All alternative interpretations of the stimulus are thus “explained away”. Thus, consciousness and its neural correlates appear as mobile targets, which adjust their locus in the presence of expectations. This poses a further challenge to the search for the NCC, as not only the timing, but also the location of neural activation does not appear as a diagnostic feature for the NCC.

Finally, Predictive Coding also suggests that priors may be used to change the way information is sampled, as the models derived from previous experience can be used to optimise the search for the most relevant information (Friston et al. 2012). Only rarely do we keep our gaze still and wait for the world to bring novel information; instead, we scan images through rhythmic patterns of eye movements accompanied by fixations. This active sensing view implies that perception is not a passive phenomenon in which the system waits for information to hit the sensory transducers, but instead an active process that seeks information through exploratory routines (Melloni et al. 2009; Schroeder et al. 2010). To test whether and how priors affect the sampling of information we developed stimuli for which we could quantify the local information content at each point (Figure 2) and determined the efficiency of information extraction based on eye movements in the presence or absence of expectations. Figure 2 shows that when subjects have prior knowledge of the object they are trying to perceive, they can immediately orient their eyes to areas of most diagnostic information for the perception of an object. At the same time, the sampling of information becomes sparser, concentrating eye movements to maximally informative areas (Moca et al. 2011). This implies that priors direct our exploratory motor routines, thus optimising perception.

Figure 2: (a) Original images are filtered through a series of gabor wavelets, which allows the estimation of the points of maximal local information (Points of Maximal Information, POI) in the source image. (b) Dots of an elastic lattice are created by mapping the POI in the projection plane, and attracting them by the projection F0 of a gravitational force G. (c) Pattern of saccades/fixations when subjects recognise a stimulus and its underlying POI map. (d) Pattern of fixations for stimuli of different degradation levels from high degradation (0) to low degradation (0.30). Dots in blue correspond to fixations when subjects do not have an expectation of the stimuli, dots in red correspond to patterns of fixation observed in the presence of expectations. Note that in the presence of expectations, the distribution of fixations are much less scattered. From Moca et al. (2011).

Overall, these studies show that previous experience enriches the contents of consciousness and fundamentally changes the way information is processed in our brain, enhancing speed and efficiency. This raises questions for theories that propose a fixed latency or neural locus for conscious access, but also complicates the quest for the NCC, as they turn out to differ in time and location depending on the precision and accuracy of expectations. Although current formulations of Predictive Coding do not make specific predictions about consciousness, this framework may nevertheless prove to be an important starting point in trying to understand these effects. In fact, more explicit theoretical links between Predictive Coding and consciousness are now being worked out (e.g., Clark 2013; Hohwy 2013; Seth et al. 2011)—after all, Predictive Coding has been framed as a unifying theory of the brain (Friston 2010), which would fall short if consciousness was left unexplained.