17: From Nerve Cells to Cognition: The Internal Representations of Space and Action

• The Major Goal of Cognitive Neural Science Is to Understand Neural Representations of Mental Processes

• The Brain Has an Orderly Representation of Personal Space

  • The Cortex Has a Map of the Sensory Receptive Surface for Each Sensory Modality

  • Cortical Maps of the Body Are the Basis of Accurate Clinical Neurological Examinations

• The Internal Representation of Personal Space Can Be Modified by Experience

• Extrapersonal Space Is Represented in the Posterior Parietal Association Cortex

• Much of Mental Processing Is Unconscious

• Is Consciousness Accessible to Neurobiological Analysis?

  • Consciousness Poses Fundamental Problems for a Biological Theory of Mind

  • Neurobiological Research on Cognitive Processes Does Not Depend on a Specific Theory of Consciousness

  • Studies of Binocular Rivalry Have Identified Circuits That May Switch Unconscious to Conscious Visual Perception

  • Selective Attention to Visual Stimuli Can Be Studied on the Cellular Level in Nonhuman Primates

  • How Is Self-Awareness Encoded in the Brain?

• An Overall View

Neural scientists believe that a cellular approach is necessary to understand how the brain works. Considering that the brain has a hundred billion nerve cells, it is remarkable how much can be learned about mental activity by examining one nerve cell at a time. Progress is particularly good when we understand the anatomy and connections of functionally important pathways.

Cellular studies of the sensory systems, for example, provide important insight into how stimuli at the body's surface are translated by the brain into sensations and planned action. In the visual system, the sensory system most thoroughly studied at the cellular level, information arrives in the brain from the retina in parallel pathways dedicated to analyzing different aspects of the visual image—form, movement, and color. These separate inputs are eventually integrated into coherent images according to the brain's own rules, rules that are embodied in the circuitry of the visual system.

Different modalities of perception—an object seen, a face touched, or a melody heard—are processed similarly by the different sensory systems. Receptors in each system first analyze and deconstruct stimulus information. Receptors at the periphery of the body for each system are sensitive to a particular kind of physical event—light, pressure, tone, or chemical odorants. When a receptor is stimulated—when, for example, a receptor cell in the retina is excited by light—it responds with a distinct pattern of firing that represents certain properties of the stimulus. Each sensory system obtains information about the stimulus in this way and this information is conveyed along a pathway of cells leading to a specific (unimodal) region of cerebral cortex. In the cortex different unimodal regions representing different sensory modalities communicate with multimodal association areas through specific intracortical pathways, and in this network signals are selected and combined into an apparently seamless perception.

The brain thus produces an integrated perception because nerve cells are wired together in precise and orderly ways according to a general plan that does not vary greatly among normal individuals. Nevertheless, the connections are not exactly the same in all individuals. As we shall learn in later chapters, connections between cells can be altered by activity and by learning. We remember specific events because the structure and function of the connections between nerve cells are modified by those events.

Despite its success, neural scientists believe that a cellular approach alone is not sufficient for understanding how the integrative action of the brain—the simultaneous activity of discrete sets of neurons—produces cognition. For this task the brain must be studied as an information processing organ. This is the approach of cognitive neural science, which uses a combination of methods from a variety of fields—cell biology, systems neural science, brain imaging, cognitive psychology, behavioral neurology, and computational neuroscience.

Ulric Neisser, one of the pioneers of cognitive psychology, defined the challenge of this field in the following terms:

In this chapter we first discuss how cognitive neural science evolved from otherwise disparate disciplines. We illustrate the success of the approach by considering what has been learned about one complex mental state: the experience of personal and extrapersonal space, both real and imagined. We then discuss how the unconscious and conscious mental processes are modeled by cognitive neural science. A great deal of cognitive processing goes on unconsciously. Sigmund Freud likened the conscious perception of mental processes to the perception of the external world by sense organs. We also discuss the profound challenges to a scientific study of consciousness. The five major subjects of cognitive neural science—perception, action, emotion, language, and memory—are discussed in detail in the subsequent five parts of the book beginning with Chapter 21.

Until the end of the 19th century the chief method for understanding the mind was introspection. In fact, the scholarly study of the mind was a branch of philosophy. By the middle of the 19th century, however, the philosophical approach was giving way to empirical analysis and eventually the formation of the independent discipline of experimental psychology. In its early years experimental psychology was concerned primarily with the sequence of events by which an external stimulus becomes an internal sensation. By the end of the 19th century the interests of psychologists turned to how behavior is generated, how it is modified by learning and attention, and how it is stored in memory.

The discovery of simple experimental ways of studying learning and memory—first in human beings by Hermann Ebbinghaus in 1885 and a few years later in experimental animals by Ivan Pavlov and Edgar Thorndike—led to a rigorous empirical school of psychology called behaviorism. Behaviorists, notably J. B. Watson and B. F. Skinner in the United States, argued that behavior could be studied with the precision of the physical sciences, but only if psychologists abandoned speculation about what occurs in the mind and focused exclusively on the observable aspects of behavior. For example, the behaviorists argued that one cannot base a psychology on the idea that people do certain things because they believe they are the right things to do or because they want to do them.

Behaviorists regarded these unobservable mental processes, especially anything as abstract as motivations, feeling, or conscious awareness, as inaccessible to scientific study. They concentrated instead on evaluating—objectively and precisely—the relationship between specific physical stimuli and observable responses in intact animals. Their early successes in studying simple forms of behavior and learning encouraged them to treat all cognitive processes that intervene between the stimulus (input) and behavior (output) as irrelevant.

During behaviorism's most influential period, in the 1950s, many psychologists accepted the most radical behaviorist position: that observable behavior is all there is to mental life. As a result, the scientific concept of behavior largely depended on the techniques used to study it. This emphasis reduced the domain of experimental psychology to a restricted set of problems and excluded some of the most fascinating features of mental life.

By the 1960s it was not difficult for the founders of cognitive psychology—notably Edwin Tolman, Frederick Bartlett, George Miller, Noam Chomsky, Ulric Neisser, and Herbert Simon—to convince the scientific community that behaviorism was too limiting Building on earlier evidence from Gestalt psychology, psychoanalysis, and neurology, these early cognitive psychologists sought to demonstrate that our knowledge of the world is based on our biological equipment for perceiving the world, that perception is a constructive process that depends not only on the stimulus but also on the mental apparatus of the perceiver—the organization of the sensory and motor systems in the brain. We now realize that this constructive process also involves emotion, motivation, and reward.

What ultimately distinguished the cognitivists from the behaviorists was not only their conceptual approach to behavior but also the complexity of the methods they used. Cognitivists realized that only relatively few input–output relationships are stereotyped, that these relationships vary significantly because of mental states, past history, and expectations, the very factors that the behaviorists tended to ignore. Thus these variables must also be observable in behavior (or output) but are just more difficult to identify than the behavior defined by behaviorists.

This new perspective relied on neural network modeling and fortunately it coincided with the emergence of large-scale computers in the period following World War II. Computers allowed the modeling and testing of ideas about large neural networks that in principle are capable of higher mental functions. However, once psychologists acknowledged that mental activity was equivalent to computational processes in the brain, they had to face the fact that most mental processes were still largely inaccessible in living subjects. Without direct access to the brain, it would be difficult if not impossible to choose between various rival theories.

Fortunately, new tools for the empirical study of mental processes quickly became available, and significant progress was soon made in cellular analyses of the neural mediation of vision, touch, and action in intact primates engaged in ordinary behavior. Single-neuron recording and noninvasive imaging and recording techniques have allowed researchers to describe how neural activity in different sensory and motor pathways encodes sensory stimuli and planned actions. Moreover, imaging methods permit direct visualization of the brain in human subjects engaged in mental activity, allowing insight into attention and aspects of consciousness under controlled conditions (Chapter 20). Thus we can now study directly neural representations of the environment and motor action by comparing cellular recordings in primates engaged in purposeful activity with imaging of the human brain at work.

Cognitive neural science, as now practiced, emerged from four major technical and conceptual developments. First, in the 1960s and 1970s techniques were developed by Robert Wurtz and Edward Evarts at the National Institutes of Health for studying the activity of single cells in the brains of animals, including primates, engaged in controlled behavior in the laboratory. This allowed investigators to correlate the activity of specific populations of neurons with specific perceptual and motor processes. From these microelectrode studies we have been able to see that the mechanisms of perception are much the same in humans, monkeys, and even simpler animals.

These cellular studies in monkeys also made it possible to identify the importance of different combinations of areas of the brain involved in specific cognitive functions, such as attention and decision-making. These approaches changed the way the biology of behavior is studied both in experimental animals and in humans.

Second, developments in neural science and cognitive psychology stimulated a renewed interest in the behavioral analysis of patients with brain lesions that interfere with mental functioning. This area, neuropsychology, had remained a strong subspecialty of neurology in Europe but was neglected for a time in the United States. Lesions of different regions of the brain can result in quite specific cognitive deficits. The behavioral consequences of brain lesions thus tell us much about the function of specific neural pathways. Lesion studies have shown that cognition is the product of several specialized systems, each with many components. For example, the visual system has specialized pathways for processing information about color and form on the one hand and movement on the other.

Third, the development of imaging techniques such as positron emission tomography (PET) and functional magnetic resonance imaging (MRI), as well as the development of magnetoencephalography, has made it possible to relate changes in the activity of large populations of neurons to specific mental acts in living humans (see Chapter 20). This advance has been paralleled by two further developments. The use of voltage and calcium-sensitive dyes has permitted the study of neuronal activity in large ensembles of neurons, both in vitro and in the brains of behaving animals. The more recent use of light-sensitive ion channels has permitted the activation or inactivation of the activity of specific neurons or groups of neurons in the neural circuits of intact behaving animals.

Finally, improvement in computers and the emergence of a powerful subdiscipline of computational neural science has made it possible to model the activity of large populations of neurons and to test ideas about the roles of specific components of neural circuits in the brain in particular behaviors. To understand the neural organization of a complex behavior like speech we must understand not only the properties of individual cells and pathways but also the network properties of circuits in the brain. Although network properties depend on the properties of individual neurons in the network, they are not identical or even similar to those properties but are an emergent property of the way those different cells are interconnected.

When computational approaches are combined with detailed behavioral studies, for example with the psychophysical study of a specific perceptual act, the combined analysis can help to characterize the properties of a system. Thus psychophysics can describe what the system is capable of doing whereas computational modeling can describe how the properties of constituent cells could account for system properties of the neural circuits involved (see Appendices E and F).

The work of cognitive and computational neuroscientists is providing enormous insight into the workings of the brain. However, it also raises a difficult set of questions about the relationship between observed neurophysiological and mental processes, and particularly between these cellular biological processes and consciousness. The answers to these questions are yet unknown, but the mere fact that scientists are addressing them is a major advance.

To illustrate how cognitive neural science describes a mental act, in the next few sections we summarize what neural science has learned about the brain's representation of personal space (one's body) and peripersonal space, the space within arm's length ("near space"). We shall see how the brain constructs mental representations of space from external sensory input; how this representation gives rise to imagined and remembered space; and how it is selected by action, modified by normal experience, and distorted by abnormal experience such as loss of a body part. This discussion illustrates a key principle that we will consider again in Chapter 19, that action has a key role in perception. The cognitive functions of the premotor areas provide for flexibility in behavior, preventing an otherwise stereotypic relationship between sensory input and behavioral output. By interpreting sensory input based on experience and mental state, these cognitive premotor processes shape our behavior.

The neural representation of space is most clearly evident in the early stages of sensory processing—in primary and higher-order areas of somatosensory cortex—where it takes the form of a map of the tactile sensors on the body surface. We shall see how this map can be modified after the loss of a body part and how those modifications can create a phantom representation. We shall also see how representations of personal and peripersonal space differ from extrapersonal space, the space beyond arm's length ("far space"), and how representations of extrapersonal space can give rise to imagined and remembered space.

The neural representation of the body surface is a simple example of an internal representation, one that has been extensively explored in the study of touch and proprioception. Touch provides information about the properties of objects, such as their shape, texture, and solidity; proprioception provides information about the static position and movement of fingers and limbs.

An internal representation can be thought of as a certain pattern of neural activity that has at least two aspects: (1) the pattern of activation within a particular population of neurons (some cells are active and others not) and (2) the pattern of firing in individual cells.

Sensory neurons with receptors in the skin translate the mechanical energy of a stimulus into neural signals that are then conveyed along pathways that end in the somatosensory areas of the parietal lobe of the cerebral cortex (Figure 17–1). Each pathway includes one or more synaptic relays. At each relay, where thousands of afferent axons terminate on a cluster of similar neurons, the arrangement of the presynaptic fibers preserves the spatial relations of the receptors on the body surface. This somatotropic order thus creates a neural map of the body surface at each synaptic relay in the somatosensory system—information from neighboring receptors in the skin is conveyed to neighboring cells in each synaptic relay.

Neural maps of the body surface were first detected in laboratory animals using gross recording and stimulation techniques on the surface of the postcentral gyrus of the parietal cortex, the only portion of the cortex accessible with the experimental techniques available at that time. In the late 1930s Wade Marshall found that he could produce an evoked potential in the cortex by touching a specific part of the animal's body surface (Figure 17–2). Evoked potentials are electrical signals that represent the summed activity of thousands of cells and are recorded with external macroelectrodes on the brain surface. The evoked response method was used by Marshall, Clinton Woolsey, and Phillip Bard to map the neural representation of the body surface in the postcentral gyrus of monkeys (Figure 17–3).

The human somatosensory cortex was similarly mapped in the late 1940s by the Canadian neurosurgeon Wilder Penfield during operations for epilepsy and other brain disorders. Working with locally anesthetized conscious patients, Penfield stimulated various points on the surface of the postcentral gyrus and asked the patients to report what they felt. This procedure was necessary to ascertain where the epilepsy started and therefore to avoid damage to healthy brain tissue during surgery. Penfield found that activation of specific populations of cells in the postcentral gyrus reasonably simulated natural activation of these populations, producing sensations of touch and pressure in the contralateral hand or leg. From these studies Penfield constructed the neural map of the human body in the primary somatosensory cortex.

In this map the leg is represented in the most medial area of cortex followed by the trunk, arms, face, and finally, most laterally, the teeth, tongue, and esophagus. The area devoted to each part of the body reflects the relative importance of that part in sensory perception. Thus the area for the face is large compared with that of the back of the head, that of the index finger is gigantic compared with the big toe, and the torso is represented in the smallest area of all (Figure 17–4). Such differences reflect differences in innervation density throughout the body. Similar relationships are observed in other animals. In rabbits, for example, the face and snout have the largest cortical representation because they are the most important sensory surfaces a rabbit uses to explore its environment (Figure 17–5).

The Cortex Has a Map of the Sensory Receptive Surface for Each Sensory Modality

Marshall went on to find that the receptive surfaces for vision and hearing, the retina and the cochlea, are also represented topographically in the cortex. Marshall's early efforts to analyze these sensory maps of the body probed only limited areas of the cortex and used techniques with poor spatial resolution. His work in the area of touch led to the conclusion that there is a single large representation of the body surface in the cerebral cortex. Later studies based on single-neuron recordings revealed four fairly complete maps in the four areas of the primary somatosensory cortex (Figure 17–6).

Although each of the four areas has essentially the same body map, each area processes a distinct type of information. Area 3a receives information from muscles and joints, important for limb proprioception. Area 3b receives information from the skin, important for touch. This information from the skin is further processed within area 1 and then combined with information from muscles and joints in area 2. This explains why a small lesion in area 1 impairs tactile discrimination, whereas a small lesion in area 2 impairs the ability to recognize the shape of a grasped object.

Cortical Maps of the Body Are the Basis of Accurate Clinical Neurological Examinations

The existence in the brain of maps of the sensory receptive surface and a similar motor map for movement explains why clinical neurology can be an accurate diagnostic discipline, even though for many decades before brain imaging came along neurology relied on only the simplest tools: a wad of cotton, a safety pin, a tuning fork, and a rubber reflex hammer. For example, disturbances in the somatic sensory system can be located with remarkable accuracy because there is a direct relationship between the anatomical organization of the functional pathways in the brain and specific perceptual and motor behaviors.

A dramatic example of this relationship is the Jacksonian march, a characteristic sensory seizure first described by the British neurologist John Hughlings Jackson. In this type of seizure there is, in addition to a motor progression, a sensory progression. Numbness and paresthesia (inappropriate sensations such as burning or prickling) begin in one place and spread throughout the body. For example, numbness might start at the fingertips, spread to the hand, up the arm, across the shoulder into the back, and down the leg on the same side. This sequence is explained by the arrangement of inputs from the body in the somato sensory cortex (Figure 17–4); the seizure starts in the lateral region of the cortex, in the area where the hand is represented, and propagates across the cortex toward the midline.

Until recently it was thought that the sensory maps of the body surface in the cortex were hard-wired, and the pathways from the receptors in the skin to the cortex are fixed early in development. But cortical maps do change, even in adults, and details of these maps vary considerably from one individual to another.

To show that experience can account for this variability, owl monkeys were trained to touch a rotating disk with the tips of the middle fingers to obtain food. After several months of touching the disk, the area of cortex devoted to the tips of the middle fingers was greatly expanded, whereas that devoted to the adjacent proximal phalanges, which had not been used in the experiment, was correspondingly reduced (Figure 17–7). These results demonstrate that use of the fingertips can strengthen connections between neurons along the somatosensory pathway from skin to cortex.

Dramatic changes in afferent connections can also occur because of disuse. In one study of several monkeys an upper limb was rendered completely useless by severing all sensory nerves to the arm. The animals were monitored for 10 or more years. In all these monkeys the representation of the face, where innervation remained intact, expanded into the adjacent area of cortex that had represented the hand before deafferentation. As a result, stimulation of the face evoked responses in the area of cortex that normally represented the hand. These changes occurred in a wide area of cortex: A third of the entire body map was taken over by new connections from the face.

How do these changes occur? Afferent connections to neurons in the somatic sensory cortex are thought to be fine-tuned during development when the firing of pre- and postsynaptic cells is correlated. Cells that fire together are thought to strengthen their connection together. Michael Merzenich and his colleagues tested this idea by surgically connecting the skin of two adjacent fingers of a monkey. This procedure assures that the connected fingers are always used together and therefore increases the correlation of their inputs from the skin. Increasing the correlation of activity from adjacent fingers in this way abolishes the sharp discontinuity normally evident between the zones in the somatosensory cortex that receive inputs from these digits. Thus, although patterns of connections are genetically programmed, they are also modified by experience.

Magnetic encephalography, a method for recording the magnetic field produced by local electrical activity, has been used to construct cortical maps of the hand with a precision of millimeters (Figure 17–8). This technique has been used to compare the hand area in the cortex of normal adult humans with that of patients with a congenital fusion of the fingers (syndactyly). Patients with this syndrome do not have individual fingers—their hand is like a fist—so that neural activity in one part of the hand is always correlated with activity in all the other parts. The representation in the cortex of the syndactylic hand is considerably less than that of a normal person, and within this shrunken representation the neurons that receive signals from the fingers are not organized somatotopically (Figure 17–9A).

When the fingers of one syndactylic patient were surgically separated, however, each newly separate finger became individually represented in the cortex within weeks. The new neural organization occupied an area of cortex closely corresponding to that of normal individuals, with normal distances between each digit (Figure 17–9B).

These results raise an important question that is even more urgent in the study of phantom limbs. How are changes in cortical maps interpreted by the brain, and how do they shape perception? Many patients with amputated limbs continue to have a vivid sensory experience of the missing limb, a disorder known as the phantom limb syndrome. The patient senses the presence of the missing limb, feels it move around, and even feels it try to shake hands when greeting someone. Terrible pain is often felt in the phantom limb.

Phantom limb sensation and the pain associated with it have been attributed to impulses entering the spinal cord from the scar of nervous tissue in the stump. In fact, removing the scar or cutting the sensory nerves just above it may relieve pain in some cases. But imaging studies of the somatosensory cortex of patients who have lost a hand suggest that phantom limb sensations are caused by a rearrangement of cortical circuits. As the afferents from the lost hand wither, adjacent afferent fibers expand into their place, just as in the monkeys with deafferented limbs.

In several patients the area of cortex that represented a hand before amputation now receives afferents from at least one other site on the skin. This has been called remapping of referred sensations. Stimuli applied to the face and upper arm are selectively capable of eliciting referred sensations in the phantom hand; both areas are represented in the brain next to the hand area (Figure 17–10). Thus changes in the arrangement of sensory afferents force changes in the readout of the sensory map—the brain learns to interpret activity on the patch of cortex receiving information from the face and upper arm as emanating from the amputated limb.

Neurons in the primary somatosensory cortex areas 3a, 3b, and 1 project to higher-order unimodal areas of the anterior parietal lobe (Brodmann's area 2), and to multimodal association areas in the posterior parietal cortex (Brodmann's areas 5 and 7). The latter also receive input from the visual and auditory systems and from the hippocampus. The parietal association areas thus integrate somatic sensory information with other sensory modalities to form spatial percepts of objects in extrapersonal or far space.

Indeed, the connection between higher mental processes and specific nerve cells has been most dramatically demonstrated in these association areas in the posterior parietal cortex. Lesions here produce complex defects in personal or peripersonal spatial perception, visuomotor integration, and selective attention. Damage to the posterior parietal lobe produces object agnosia, a modality-specific inability to recognize certain kinds of objects even though afferent sensory pathways function normally. For example, some patients with posterior parietal damage are unable to recognize objects through touch (astereognosis). In fact, the most common agnosias result from lesions in the posterior parietal cortex.

Many patients with parietal lesions also show a striking deficit in awareness of one side of their body. For example, such patients may not dress, undress, or wash the affected side (personal neglect syndrome). They may even deny or disown their left arm or leg, going so far as to ask, "Who put this arm in bed with me?" Because the idea of having a left limb is foreign to them, patients may also deny the paralysis in this limb and attempt to leave the hospital prematurely because they feel nothing is wrong with them. Such denial about a disease or disability is referred to as anosognosia.

In some patients with right parietal lesions the sensory neglect extends from near space to far space. In these cases the ability to copy the left side of a drawing is severely disturbed. The patient may sketch the petals of a flower on the right side only. When asked to copy a clock, the patient may ignore the numbers on the left, try to cram all the numbers into the right half of the clock, or draw them on one side running off the clock face (Figure 17–11). A particularly dramatic example of spatial neglect is seen in self-portraits by a German artist who suffered a stroke that affected his right posterior parietal cortex. The portraits done in the two months after the stroke show a profound neglect of the left side of the face (Figure 17–12).

Spatial neglect can be quite selective. Some patients with neglect syndrome after injury to the right hemisphere have deficits in the perception of the form of objects. A patient may recognize an entire object but not all its parts, even though the visual pathways are intact (Figure 17–13).

Another form of spatial neglect is the neglect of one half of a remembered image, called representational neglect. This was first observed by the Italian neurologist Edoardo Bisiach in a group of patients in Milan, all of whom had injury to their right parietal lobe. Patients were asked to imagine that they were standing in the city's main public square, the Piazza del Duomo, facing the cathedral, and to describe from memory the buildings around the square (Figure 17–14). These subjects were able to identify all the buildings on the right side of the square (ipsilateral to the lesion) but could not recall the buildings on the left, even though these buildings were thoroughly familiar to them. The patients were then asked to imagine that they were standing on the steps of the cathedral, so that right and left were reversed. In this imagined position the patients were again asked to identify the buildings around the plaza. This time they identified only the buildings that they previously failed to name.

These results suggest that memory of external space is perceived in relation to the vantage point of the observer, not simply of that of objects in the environment. These Milanese patients clearly had stored a complete memory of the entire piazza and had complete access to that memory. But when they remembered the piazza they neglected the left half, depending on the vantage point of the remembered image, because they were unable to recall images associated with their left side, contralateral to the side of the lesion. Thus, Bisiach concluded, memories for each half of the visual field are accessed through the contralateral hemisphere.

Recent PET studies indicate that when normal subjects close their eyes and imagine an object such as the letter "a," the visualization recruits activity in the primary visual cortex, just as when an actual object is seen with the eyes. Patients with representational neglect presumably lack such an orienting mechanism. Thus damage to the posterior parietal cortex, which impairs real-time visual perception, can also impair remembered or imagined visual images.

In 1860 Herman Helmholtz, one of the pioneers in applying physical methods to perception, succeeded in measuring the conduction velocity of the nerve impulse to be approximately 90 m/s. He then went on to study reaction time—the time it took a subject to react to a stimulus—and found it to be much slower than the time required for the information to reach the brain by means of conduction time alone. This caused Helmholtz to realize that the brain must require a considerable amount of time to process sensory information before that information reaches conscious perception. Helmholtz proposed that this was the time the brain needed to evaluate, transform, and reroute the neural signals prior to our being aware of the significance of these signals. This unconscious inference, he argued, was required for perception and voluntary movement.

In the beginning of the 20th-century Sigmund Freud elaborated on Helmholtz's idea that much of mental activity is unconscious, pointing out that our unconscious mental life is not a single process but has at least three components: implicit, dynamic, and preconscious unconscious. Implicit unconscious is Helmholtz's unconscious inference. It includes, as we shall learn in Chapters 65 and 66, implicit memory, the type of memory that underlies learning perceptual and motor skills and which we now attribute to the striatum, the cerebellum, and the amygdala. The dynamic unconscious is that part of unconscious mental activity that involves our conflicts, repressed thoughts, and sexual as well as aggressive urges. This component of unconscious mental processes was the major focus of Freud's work. Finally, the preconscious unconscious is that part of the unconscious that is most readily accessible to consciousness. This component is concerned with organizing and planning for immediate actions, functions we now attribute to the prefrontal cortex.

The insight of Helmholtz and Freud that much of our mental life is unconscious raised the following related questions: What is left for freedom of action? What is the nature of free will? A major step in addressing this issue empirically was a study by Benjamin Libet. The study was based on an earlier finding that any voluntary movement is preceded by a readiness potential, a small electrical response recorded from the surface of the skull that occurs approximately one second before the movement. Libet asked subjects to will a movement and found to his surprise that a subject's awareness of his own willingness to move a finger followed, rather than preceded, the readiness potential and did so by as much as a full second. By recording neural activity we can predict a subject's desire to move a finger before the subject is aware of his own desire to move that finger! Thus what we consider acts of free will may have a significant unconscious step.

Consciousness Poses Fundamental Problems for a Biological Theory of Mind

Exploration of the nature of spatial neglect and free will touches on one of the great issues of cognitive neural science, and in fact of all science: the nature of consciousness. The unique character of consciousness has attracted fierce interest and debate among philosophers of mind because it is difficult to see how consciousness might be explained in reductionist physical terms.

At the beginning of this book we stated that what we commonly call mind is a set of operations carried out by the brain. Because consciousness is a fundamental property of mind, it too must be a function of the brain and in principle we should be able to identify neural circuits that give rise to it. However, before we can develop theories of consciousness that can be tested by empirical science, we must first define consciousness in operational terms.

Here we should emphasize that, in general, the concepts that neuroscientists initially use to describe mental processes—such as learning, memory, or consciousness—are those developed by philosophers. Such concepts were formed without knowledge of how mental processes are mediated by the brain. Once neuroscientists define a specific mental process in psychological terms—and we can now do so quite precisely—they then can attempt to localize and analyze the neuronal systems that mediate the process. This approach, as we shall see, can now even be applied to consciousness.

Consciousness is ordinarily thought of as a state of self-awareness. Philosophers of mind such as John Searle and Thomas Nagel have defined three essential features of self-awareness: subjectivity, unity, and intentionality.

The subjectivity of self-awareness is the characteristic that poses the greatest philosophical and scientific challenge. Each of us has an awareness of a self that is the center of experience. Each of us experiences a world of sensations that feel unique and private. Our own experience seems much more real to us than the experiences of others. Our own ideas, moods, and sensations—our successes and disappointments, joys and pains—are experienced directly, whereas we can only indirectly appreciate other people's ideas, moods, and sensations. Is the aroma of lavender that I smell identical to your experience of lavender? This is not simply a question of our sensory capability. Even when sensory capabilities are measurably identical, the aroma of lavender is not only determined by the lavender but also by our personal history—the experience we recall from memory—and since experiential history is highly individualized, lavender may not produce the same subjective sensation in each of us.

Once we know how the aroma of lavender is mediated by neural signals that announce the presence of chemical molecules, how does our sensation, the conscious awareness of an aroma, arise from other neural networks of the brain?

The fact that conscious experience is fundamentally subjective raises the question of whether it is even possible to determine objectively some characteristics of consciousness that transcend individual experience. If the senses produce only subjective experience, the argument goes, those same senses cannot be the means of arriving at an objective understanding of experience.

The unity of self-awareness refers to the fact that our experience of the world at any given moment is felt as a single unified experience. All of the various sensory modalities are blended into a single experience. When we sit down to dinner we experience the chair against our back, the sound of music, and the fruity flavor of the wine as connected and simultaneous. When we speak to our dinner partners we do so in whole sentences; we are aware that we are completing an idea but pay little if any attention to the process of constructing sentences.

Finally, self-awareness has intentionality. That is, our conscious experience connects successive moments and we have the sense that successive moments are directed to some goal.

In earlier times these features of consciousness led some philosophers to a dualistic view of mind, a view that the body and the mind are very different substances—the body being physical and the mind existing in some nonphysical, spiritual medium. Today almost all philosophers of mind agree that what we call consciousness derives from physical properties of the brain. Thinkers about consciousness fall into two groups. The first group, of which Daniel Dennett is the most prominent advocate, thinks there is no problem of consciousness. Consciousness emerges quite simply from an understanding of neuronal activity. Dennett argues, much as did the neurologist John Hughlings Jackson a century earlier, that consciousness is not a discrete operation of the brain but the outcome of the computational activity of the association areas of the brain. The second group, which includes Francis Crick, Christof Koch, John Searle, Thomas Nagel, Antonio Damasio, and Gerald Edelman, believes that consciousness is a discrete phenomenon and that the issues of subjectivity, unity, and intentionality must be confronted if we are to understand how our experience is constructed.

Because consciousness has properties that other mental functions do not, a biological explanation poses a formidable problem, a problem so inherently difficult that the philosopher Colin McGinn has argued that consciousness is simply inaccessible to empirical study because of limitations inherent in human intelligence. Just as monkeys cannot understand quantum theory, humans cannot understand consciousness, McGinn argues. Conversely, Searle and Nagel argue that consciousness is accessible to analysis but we have been unable to explain it because it is a highly subjective and complex property of the brain unlike any function of the brain we understand—indeed, unlike any other subject of scientific inquiry.

Of the three features of consciousness, subjectivity is the most difficult to analyze empirically. Nagel and Searle illustrate the precise difficulty in the following way. Assume we succeed in studying a person's consciousness by recording the electrical activity of neurons in a region known to be important for consciousness while that person carries out a particular task requiring conscious attention. How do we then analyze the results? Can we say that the firing of a group of neurons causes a private subjective experience? Does a burst of action potentials in the thalamus and somatic sensory cortex switch information into consciousness so that a person now perceives an object in his hand and perceives it as round or square, hard or malleable? What empirical grounds do we have for believing that when a mother looks at her infant child the firing of cells in the inferotemporal cortex concerned with face recognition causes conscious recognition of her child's face?

As yet we do not know even in the simplest case how the firing of specific neurons leads to conscious perception. In fact, Searle argues that we lack even an adequate theoretical model of how an ontologically objective phenomenon—electrical signals in another person's brain—can cause an ontologically subjective experience such as pain. Because consciousness is irreducibly subjective, it lies beyond the reach of science as we currently practice it.

Similarly, Nagel argues that because current science is essentially a reductionist approach to understanding phenomena it cannot address consciousness without a significant change in method, one in which the elements of subjective experience are defined. These elements are likely to be basic components of brain function much as atoms and molecules are basic components of matter. According to Nagel, object-to-object reductions are not problematic because we understand, at least in principle, how the properties of a given type of matter arise from the molecules of which it is made. What we lack are rules for extrapolating subjective experience from the physicochemical properties of interconnected nerve cells.

Nagel argues that our complete lack of insight into the elements of subjective experience should not prevent us from discovering rules that relate conscious phenomena to cellular processes in the brain. In fact, Nagel believes that the knowledge needed to think about a more fundamental type of analytical reduction—from something subjective (experience) to something objective (physical)—can be gained only through the accumulation of cell-biological information. Only after we have developed a theory of mind that supports this novel and fundamental reduction will the limitations of the current reductionism become apparent. The discovery of the elementary components of subjective consciousness, Nagel argues, may require a revolution in biology and most likely a complete transformation of scientific thought.

Neurobiological Research on Cognitive Processes Does Not Depend on a Specific Theory of Consciousness

Most neural scientists whose work touches on the question of consciousness are not necessarily working toward or anticipating a revolution in scientific thought. Although neural scientists working on issues such as sensory perception and cognition must struggle with the difficulties of defining consciousness experimentally, these difficulties do not appear to preclude productive research. The physicist Steven Weinberg perhaps best expressed this attitude:

Indeed, neural science has made considerable progress in understanding the neurobiology of sensory perception without having to account for individual experience. Understanding the neural basis of perception of color and form, for example, does not depend on resolving the question of whether each of us sees the same blue. Despite the fact that perception of an object is constructed by the brain from piecemeal sensory information, and despite individual differences caused by experience, perception of an object is not arbitrary and appears to correspond to objective physical properties of the object. What we do not understand is the step from action potentials to awareness of an object.

Although the subjectivity of consciousness makes the neurobiological study of consciousness especially difficult, in principle such a study may not be insurmountable using current methods. The subjective nature of perception does not prevent one person from objectively studying what another person perceives. We have been able to correlate some regularities of perception with specific patterns of neuronal activity in different individuals under a variety of circumstances. The correlation between a neural event and a mental event, based on rigorous criteria, should be a sufficient first approximation of the neural process mediating a mental operation by any reasonable standards of scientific explanation. For this reason Crick and Koch emphasized that the first step in the analysis of consciousness is to find the neural correlates of consciousness, the minimal set of neural events that give rise to a conscious percept.

Finding the neural systems that mediate con sciousness may not be simple. Gerald Edelman and Stanislas Dehaene have argued that the neural correlates of consciousness are unlikely to be localized but rather widely distributed throughout the cerebral cortex and thalamus. There is extensive evidence of massive feedforward broadcasting as well as, feedback or recursive connections between cortical areas, which Dehaene believes may be essential for the conversion of unconscious to conscious perception.

By contrast, Crick and Koch believed that the most elementary neural correlates of consciousness are likely to involve only a small set of neurons, and therefore one should be able to determine the neural circuits to which they belong. Crick and Koch proposed a search for the neural activity that produces specific instances of consciousness, such as perception of the movement of an object, its shape, and its color. Having done that we may eventually be in a position to meet Searle's and Nagel's higher demands: to develop a theory of the correlations we discover empirically, to state the laws of correlation between neural phenomena and subjective experience.

Because at any moment we can be conscious of one of a large variety of sounds, smells, and objects as well as actions, consciousness must involve modulatory control over a variety of neural systems. Thus consciousness is required for many aspects of mental activity: visual perception, thinking, emotion, action, and the perception of self. Because we understand the visual system best, Crick and Koch argued that our efforts should be focused on visual perception and in particular on two phenomena: binocular rivalry and selective attention.

Studies of Binocular Rivalry Have Identified Circuits That May Switch Unconscious to Conscious Visual Perception

When two different images are presented simultaneously to the two eyes—horizontal bars to one eye, vertical bars to the other—the subject's perception alternates spontaneously from one monocular view to the other. Erik Lumer and his colleagues found in functional imaging experiments that whenever an individual switches from one eye to the next—from one conscious percept to the next—three sets of cortical areas are recruited. One is the ventral visual pathway of the temporal lobe, which is concerned with perceptions of objects and people. The others are the parietal and frontal regions, which are known to be involved in visual attention to space. Lumer and his colleagues suggest that the frontal and parietal areas are critical for conscious perception and that these areas focus awareness on specific internal representations of visual images.

Nikos Logothetis has carried out similar analyses at the level of individual neurons and confirmed that the competition between rivalrous stimuli in the two halves of the visual field is resolved late in the ventral pathway, in the inferior temporal cortex and the lower layers of the superior temporal sulcus. These regions in turn project to and receive connections from the prefrontal cortex. In light of these findings Crick and Koch argued that the pathways for conscious visual perception course through the inferior temporal cortex to the prefrontal and parietal cortices.

Selective Attention to Visual Stimuli Can Be Studied on the Cellular Level in Nonhuman Primates

Selective attention in vision is another useful starting point for a cell-biological approach to the study of consciousness. At any given moment we are aware of only a small fraction of the sensory stimuli that impinge on us. As we look out on the world, we focus on specific objects or scenes that have particular interest and exclude others.

If you raise your eyes from this book to look at a person entering the room, you are no longer attending to the words on this page. Nor are you attending to the decor of the room or other people in the room. This focusing of the sensory apparatus is an essential feature of all sensory processing, as Williams James first noted in his Principles of Psychology (1890):

Cellular studies of the posterior parietal cortex in monkeys have provided important insight into the neural mechanisms of focusing attention on specific objects in the visual field. Like neurons in other visual processing centers, each parietal neuron fires when a visual stimulus enters its receptive field (see Chapter 25 for a description of the receptive fields of cortical neurons in the visual system). The strength of the neuron's response depends on whether the animal is paying attention to the stimulus. The response is moderate when the animal's gaze is directed away from the stimulus but vigorous when the monkey attends to the stimulus (Figure 17–15).

These findings are consistent with the clinical observation that the parietal cortex is involved in focusing on objects in space. The response of the neuron is independent of how the animal attends to the stimulus. The firing rate of the neuron increases by about the same amount whether the animal merely looks at the stimulus or reaches for it while continuing to look elsewhere (Figure 17–15). This independence is significant because the posterior parietal cortex makes connections with structures in the prefrontal cortex that are involved in the planning and execution of movements of the eyes and hands.

When an object induces slightly disparate images in the two retinas, we do not see double images. Instead we perceive a single object in front of or behind the plane of fixation. Three-dimensional movies and Magic Eye books take advantage of this phenomenon, displaying slightly different images to each eye to induce a conscious perception of depth. Neurons in the primary visual cortex, the first synaptic relay of the visual system in the cerebral cortex, are sensitive to this retinal disparity and could therefore provide the basis for depth perception. However, these same neurons respond differently to black and white images that are anticorrelated and disparate—images in which each black pixel presented to one eye corresponds to a white pixel in the other, and vice versa. Although the neural synapse should give rise to a conscious perception of depth, in fact such images are not perceived as a single image having depth; instead they are treated as rivalrous alternating images. One sees either a white-on-black or black-on-white image, and the perceptual switch occurs spontaneously every few seconds, without any separation of depth.

Both retinal disparity and anticorrelated images produce an ocular reflex that adjusts the eyes to a depth of field equal to the plane of the image fixated, yet anticorrelated images are not perceived as one image with a single depth of field. The signal of depth triggers a cellular response in the primary visual cortex that is not consciously perceived and therefore does not have a direct role in conscious depth perception. It is thought that later stages of visual processing are responsible for depth perception and somehow reject the depth information computed for anticorrelated images in the primary visual cortex.

The study is important because it shows how neural activity can be dissociated from conscious perception. Disparate anticorrelated images are consciously perceived as rivalrous images—you see one input or the other but you do not see them fused into one object. However, neurons in the primary visual cortex do detect the anticorrelated images as fused and compute the depth of the fused image. In addition, the eyes make automatic vergence movements to the computed depth of the fused image that the brain does not consciously perceive.

These findings reinforce the idea that sensory input alone does not give rise to consciousness; higher-level interpretation of that input is needed.

How Is Self-Awareness Encoded in the Brain?

If visual attention is presently the most tractable example of consciousness, self-awareness is probably the deepest problem. Although aspects of self-awareness are evident in nonhuman primates, self-awareness is central to human identity and has evolved in parallel with language and other forms of symbolic communication.

A more promising approach to the study of consciousness may lie in the latest advances in neural prosthetics that give people the ability to voluntarily modulate neural signals to achieve a goal (move a cursor on the screen). Similarly, some individuals can achieve great control of their breathing and heart rate. These feats suggest that studies of how people can consciously control signals that are normally unconscious may shed light on the neural processes of self-awareness.

To come to grips with the biological processes of cognition, it is necessary to move beyond the individual neuron and consider how information is processed in neural networks. This requires not only the methods and approaches of cellular and systems neuroscience but also the insights of cognitive psychology.

The anterior regions of the parietal lobe contain elementary internal representations of the body surface and peripersonal space that can be modified by experience. Analysis of such modifications in the posterior parietal association cortex indicates that selective attention is a factor in integrating the internal representation of the body with perception of extra personal space. The representation of the body is integrated with the representation of actual, imagined, or remembered visual space, and self-consciousness functions within this integrated representation. Indeed, the Russian neuropsychologist A. R. Luria suggested that portions of the parietal lobe constitute the aspect of cortical organization that is the most distinctly human.

But it is likely that just as there is more than one form of spatial experience so there is more than one form of consciousness, each with different neural representations. Thus, Edelman and Damasio distinguish between primary (or core) consciousness and higher-order (extended) consciousness. Primary consciousness is an awareness of objects in the world, of the ability to form mental images of them. Primary consciousness is not unique to humans but shared by nonhuman primates and perhaps by other vertebrate animals as well. By contrast, higher-order consciousness involves a consciousness of being conscious and is uniquely human. It allows for a concept of past and future and therefore the ability to think of the consequences of one's acts and feelings.

In their attempt to develop a coherent reductionist approach to the study of consciousness, Crick and Koch began with Sigmund Freud's view that most mental functions are unconscious, including much of thinking. We are only conscious of the sensory representation of mental activities. Freud wrote in 1923: "It dawns upon us like a new discovery that only something which has once been a perception can become conscious, and that anything arising from within [apart from feelings] that seeks to become conscious must try to transform itself into external perception."

To study consciousness one must rely on first-person reports of (subjective) perception. Thus an empirical definition of consciousness must take into account behavioral output (action), which is integral not only to the study but also to our concept of consciousness.

Intuitively we think that a conscious percept is one we can describe in words. What are words? They are sounds we associate with sensory percepts based on a set of rules for manipulating those sounds (ie, language). Thus, we might consider conscious percepts to be those percepts that can be flexibly linked with actions based on abstract rules.

If, while you are sleeping, a fly settles on your face and you wave it away, this action does not indicate consciousness. It is probably mediated by reflex pathways (similar to the pathways that mediate ocular convergence on rivalrous images). However, if asked to raise your right hand when sensing a light touch and your left hand when sensing a cold stimulus, you could perform that kind of action only if you were awake and conscious of the stimulus. Conscious percepts are those that can in principle support voluntary behavioral responses. This idea explains why correlates of consciousness show up in high-level areas that are also associated with action, such as parietal and prefrontal cortices.