21: Sensory Coding

• Psychophysics Relates the Physical Properties of Stimuli to Sensations

  • Psychophysical Laws Govern the Perception of Stimulus Intensity

  • Psychophysical Measurements of Sensation Magnitude Employ Standardized Protocols

  • Sensations Are Quantified Using Probabilistic Statistics

  • Decision Times Are Correlated with Cognitive Processes

• Physical Stimuli Are Represented in the Nervous System by Means of the Sensory Code

  • Sensory Receptors Are Responsive to a Single Type of Stimulus Energy

  • Multiple Subclasses of Sensory Receptors Are Found in Each Sense Organ

  • Neural Firing Patterns Transmit Sensory Information to the Brain

  • The Receptive Field of a Sensory Neuron Conveys Spatial Information

• Modality-Specific Pathways Extend to the Central Nervous System

  • The Receptor Surface Is Represented Topographically in Central Nuclei

  • Feedback Regulates Sensory Coding

  • Top-Down Learning Mechanisms Influence Sensory Processing

• An Overall View

Since ancient times humans have been fascinated by the nature of sensory experience. The Greek philosopher Aristotle defined five senses—vision, hearing, touch, taste, and smell—each linked to specific sense organs in the body: the eyes, ears, skin, tongue, and nose (Figure 21–1). Pain was not considered to be a specific sensory modality but rather an affliction of the soul. Intuition, often referred to colloquially as a "sixth sense," was something beyond the experience of classic sensory systems. Today neurobiologists are more likely to describe intuition as inferences derived from previous experience and thus the result of cognitive rather than sensory processes.

In this chapter we consider the organizational principles and coding mechanisms universal to all sensory systems. We define sensory information as neural activity originating from stimulation of receptor cells in specific parts of the body. These senses include the classic five senses plus a variety of modalities not recognized by the ancients but essential to bodily function: the somatic sensations of proprioception (posture and movement of our own body), pain, itch, and temperature; visceral sensations (both conscious and unconscious) necessary for homeostasis; and the vestibular senses of balance (the position of the body in the gravitational field) and head movement.

The extent to which features of sensory processing have been conserved in the course of human evolution seems nothing short of astonishing. In each of the sensory systems receptors provide the first neural representation of the external world. This information flows centrally to regions of the brain involved in cognition. The sensory pathways have both serial and parallel components, consisting of fiber tracts with thousands or millions of axons interrupted by synaptic relays comprising millions of neurons. Along the way information is transformed from relatively simple forms to the complex forms that are the basis of cognition. Sensory pathways are also recursive. The higher centers in the brain modify and structure the incoming flow of sensory signals by feeding information back to earlier stages of processing; thus percepts are shaped by internal as well as environmental factors.

In each sensory modality a specific type of stimulus energy is transformed into electrical signals by specialized receptors. The sensory information is transmitted to the central nervous system by trains of action potentials that represent particular aspects of the stimulus. The question that has intrigued philosophers and scientists alike is whether experienced sensations accurately reflect the stimuli that produce them or whether our knowledge of the world is inherently subjective and imprecise.

Modern thought about how knowledge is represented in the brain began with European philosophers of the 17th, 18th, and 19th centuries whose interest in sensation and perception was related to the question of human nature itself. The major division was between the empiricists, represented by John Locke, George Berkeley, and David Hume, and the idealists, including René Descartes, Immanuel Kant, and Georg Wilhelm Friedrich Hegel. Locke, the preeminent empiricist, advanced the idea that the mind at birth is a blank slate, or tabula rasa, void of any ideas. Knowledge is obtained through sensory experience—what we see, hear, feel, taste, and smell. In fact, Berkeley questioned whether there was any sensory reality beyond the experiences and knowledge acquired through the senses. He asked the now-famous question: Does a falling tree make a sound if no one is near enough to hear it?

The idealists responded that the human mind possesses certain innate abilities, including logical reasoning itself. The 18th-century German philosopher Immanuel Kant classified the five senses as categories of human understanding. He argued that perceptions were not direct records of the world around us but rather were products of the brain and thus depended on the architecture of the nervous system. Kant referred to these brain properties as a priori knowledge.

Thus in Kant's view the mind was not the passive receiver of sense impressions envisaged by the empiricists. Rather the human mind had evolved to conform to certain universal conditions such as space, time, and causality. These conditions were independent of any physical stimuli detected by the body. For Kant and the idealists this meant that knowledge is based not only on sensory stimulation but also on the brain's properties that organize sensory experience. If sensory experience is inherently subjective and personal, they said, it may not be subject to scientific analysis.

The modern study of sensation and perception began in the 19th century with the emergence of experimental psychology as a scientific discipline. The first psychologists—Ernst Weber, Gustav Fechner, Hermann Helmholtz, and Wilhelm Wundt—focused their experimental study of mental processes on sensation, which they believed was the key to understanding the mind. Their findings gave rise to the fields of psychophysics and sensory physiology.

Psychophysics describes the relationship between the physical characteristics of a stimulus and the attributes of the sensory experience. Sensory physiology examines the neural consequences of a stimulus—how the stimulus is transduced by sensory receptors and processed in the brain. Some of the most exciting advances in our understanding of perception have come from merging these two approaches in both human and animal studies. For example, functional magnetic resonance imaging (fMRI) and positron emission tomography (PET) have been used in controlled experiments to identify regions of the human brain involved in the perception of pain.

Psychophysical Laws Govern the Perception of Stimulus Intensity

Early scientific studies of the mind focused not on the perception of complex qualities such as color or taste but on phenomena that could be isolated and measured precisely: the size, shape, amplitude, velocity, and timing of stimuli. Weber and Fechner developed simple experimental paradigms to study how and under what conditions humans are able to distinguish between two stimuli of different amplitudes. They quantified the intensity of sensations in the form of mathematical laws that allowed them to predict the relationship between stimulus magnitude and sensory discrimination.

For example, in 1834 Weber demonstrated that the sensitivity of a sensory system to differences in intensity depends on the absolute strength of the stimuli. We easily perceive that 1 kg is different from 2 kg, but it is difficult to distinguish 50 kg from 51 kg. Yet both sets differ by 1 kg. This relationship is expressed in the equation now known as Weber's law:

where ΔS is the minimal difference in strength between a reference stimulus S and a second stimulus that can be discriminated, and K is a constant. This is termed the just noticeable difference or difference limen. It follows that the difference in magnitude necessary to discriminate between a reference stimulus and a second stimulus increases with the strength of the reference stimulus.

Fechner extended Weber's law to describe the relationship between the stimulus strength (S) and the intensity of the sensation (I) experienced by a subject:

where S ₀ is the threshold amplitude of the stimulus and K is a constant. Although Fechner's law was widely accepted for nearly a century after its publication in 1860, his assumption that the intensity of sensation could be equated with the sum of equal increments in "just noticeable differences" turned out to be incorrect.

In 1953 S. S. Stevens demonstrated that, over an extended range of stimulation, subjective experience of sensation intensity is best described by a power function rather than by a logarithmic relationship. Stevens' law states that:

For some sensory experiences, such as the sense of pressure on the hand, the relationship between the stimulus magnitude and the perceived intensity is linear, that is, a power function with a unity exponent (n = 1).

The lowest stimulus strength a subject can detect is termed the sensory threshold. Thresholds are normally determined statistically by presenting a subject with a series of stimuli of random amplitude. The percentage of times the subject reports detecting the stimulus is plotted as a function of stimulus amplitude, forming a relation called the psychometric function (Figure 21–2). By convention, threshold is defined as the stimulus amplitude detected in half of the trials. Thresholds can also be determined by the method of limits, in which the subject reports the intensity at which a progressively decreasing stimulus is no longer detectable or an increasing stimulus is detectable. This technique is widely used in audiology to measure hearing thresholds.

The measurement of sensory thresholds is a useful diagnostic technique for determining sensory function in individual modalities. An elevated threshold may signal an abnormality in sensory receptors (such as loss of hair cells in the inner ear caused by aging or exposure to very loud noise), deficits in nerve conduction properties (as in multiple sclerosis), or a lesion in sensory-processing areas of the brain. Sensory thresholds may also be altered by emotional or psychological factors related to the conditions in which stimulus detection is measured.

Psychophysical Measurements of Sensation Magnitude Employ Standardized Protocols

The lasting importance of Fechner's work was the development of formal quantitative methods for measuring sensory performance and mathematical techniques to analyze them. Three of his methods are still widely used, either exactly as he formulated them or in a modified form: (1) the method of constant stimuli, in which a fixed set of stimuli is presented repeatedly to obtain a statistical characterization of the behavior associated with each stimulus; (2) the method of limits, described earlier; and (3) the method of adjustment or reproduction, in which a subject adjusts a second stimulus to match or reproduce the intensity of the first one.

The next major methodological and conceptual developments in psychophysics came almost a century later when S. S. Stevens introduced the technique of magnitude estimation, whereby subjects use a numerical scale to rate the intensity of the sensations experienced by stimuli of different amplitude (Figure 21–3). Verbal reports of subjective experience are widely used because they are usually reliable and repeatable. Reliability is assessed by correlations between observers rating the same stimuli; repeatability is measured by correlations between responses from the same subject to similar stimuli.

Stevens generalized the method of direct verbal reporting by defining four scales of measurement—the nominal, ordinal, interval, and ratio scales—and specifying appropriate methods for analyzing data of each type. On nominal scales items have names but not rank; examples are names of colors, tastes, and smells. On ordinal scales items are ranked with a logical range and ordered relationship to each other, but the intervals between them cannot be compared meaningfully. Verbal descriptors of pain intensity are an example of an ordinal scale. When a clinician says, "On a scale from 1 to 10 in which 10 is the worst imaginable pain, how would you rate your pain?" there is no suggestion that the pain associated with an 8 is twice as intense as the pain associated with a 4 or that the difference between 5 and 6 equals the difference between 1 and 2.

On interval scales distances but not ratios between values have meaning. Counts of the number of stimuli delivered in a session or estimates of the position of an object on a grid map are examples of interval scales. In ratio scales the concepts of rank, interval, and ratio all have valid meanings. Estimates of the perceived intensity of a stimulus are treated as ratio scales. Subjects are instructed to assign a number proportional to the perceived intensity when a stimulus is detected, and to report "zero" when they feel no stimulus. Subjects typically choose their own numerical scale within a session. The values measured during an experiment are then normalized to allow comparisons of stimulus ratings between subjects. These scales permeate modern statistics and are used widely beyond the field of experimental psychology for which they were developed.

Sensations Are Quantified Using Probabilistic Statistics

Decision theory offers another approach to measurement of sensations by using statistical methods to explain the variability of subjects' responses or false reports. When subjects are pressed to detect the weakest possible stimuli, they give many false-positive responses; that is they respond affirmatively in catch trials in which no stimulus was presented. As a result, the psychophysical thresholds measured are very low (Figure 21–2B, blue curve). Conversely, when subjects are told to avoid false positives, their perceptual thresholds become quite elevated. Trials in which strict criteria are used yield higher than normal threshold values (Figure 21–2B, red curve).

In 1927 L. L. Thurstone proposed that the variability of sensations evoked by a stimulus could be represented as a normal or Gaussian probability function with a mean (m_s) and a standard deviation (σ_s) :

This allowed him to use the mathematics of probability theory and statistical tables to predict the discriminability of pairs of stimuli that differed along a physical dimension such as intensity. He proposed to equate the physical distance between the amplitudes of two stimuli to a psychological scale value of inferred loudness called the discrimination index or d ′. He equated the number of correct responses (hits) and error trials (false positives when one stimulus is confused with another) with the sensory overlap of the two stimuli. This allowed him to use statistical tables of the normal probability function to calculate d ′ values (Box 21–1).

Decision theory methods were first applied to psychophysical studies in 1954 by the psychologists Wilson Tanner and John Swets. They developed a series of experimental protocols for stimulus detection that allowed accurate calculation of d ′ as well as techniques for measurement of subjective bias during sensory testing. Their methods were initially developed for engineers studying the detection of very weak radar pulses reflected from distant airplanes. As the engineers lowered the threshold for detection they detected more radar pulses, but their apparatus gave more false positives because it was triggered more frequently by noise. Tanner and Swets hypothesized that subjects gave false-positive responses when the sensory noise exceeded their response threshold.

Signal detection theory has been widely applied in sensory discrimination tests that require the subject to make a binary choice. Threshold measurements are a good example. In a "yes-no" experiment the beginning and end of an observation interval are cued, and the subject is required to say whether or not a signal, such as a tone, was present. We can represent trial-to-trial fluctuations of the perceived stimulus intensity and that of the silent "noise" period as two overlapping Gaussian curves. The subject says "yes" when the signal exceeds a criterion (called a decision boundary) that has been set by the subject, and "no" when it does not. When the stimulus is very weak the neural signal it evokes is very small, and there is considerable overlap between the pure noise and the stimulus signal plus noise. Hence there is no decision boundary that allows error-free responses. Nevertheless, the mathematical formulation of the probability density function allows the experimenter to compensate for subjective differences in response criteria in calculations of d ′ (Box 21–1).

Sensory thresholds can also be measured using a two-alternative forced-choice protocol in which there are two observation intervals. The subject is asked whether the stimulus occurred in the first or second interval. The two-interval procedure is widely used for measuring relative intensity or sensory quality because the results obtained are more accurate than verbal judgments and the responses required are simple. Subjects can also provide nonverbal responses in such tests using levers, buttons, or other manipulanda that allow accurate measurement of decision times. Such instrumented behaviors allow neuroscientists to measure sensory processes in experimental animals by training them to use these tools to make easy sensory judgments. Such techniques can be used to probe the sensory capabilities of animal subjects as the discrimination tasks become more difficult and to investigate the underlying neural mechanisms when electrophysiological and behavioral studies are combined in the same experiment.

Decision Times Are Correlated with Cognitive Processes

Another important quantitative measure of psychophysical behavior is the reaction time, which is the time taken to perform a perceptual task. Franciscus Donders was the first, in 1865, to measure the time required to respond to stimuli. He and others found that reaction times elicited by strong stimuli are shorter than those elicited by weak stimuli. Similarly, in forced-choice tasks the time required for a decision is shorter when the stimuli are clearly distinctive in intensity or quality than when they are near the discrimination threshold. Reaction times are widely used as measures of certainty of responses in humans and animals. They are often correlated with neural activity in sensory areas of the brain and in studies of sensory-triggered motor behaviors.

Reaction times are also used to evaluate cognitive function. The tasks illustrated in Figure 21–5 were devised by Anne Treisman to investigate the mechanisms of visual pattern recognition. The subjects were asked to locate a blue cross within an array of symbols. With some patterns the blue cross seems to "pop out," but with others the array must be carefully examined to find the blue cross.

One explanation for this is that when the sought-after item differs from the other elements of the array in only one property we can scan quickly the entire array (a parallel search), but when it shares two or more properties we need to examine all of the elements one-by-one (a serial search). If this hypothesis is true, the reaction time should not depend on the number of elements in an array when the sought-after item differs in only one property, but it should increase in proportion to the number of elements if we must examine them individually. That is exactly the result obtained in such experiments. The same hypothesis also predicts that it should take twice as long to determine that an item is absent because we need to examine all elements in an array before concluding that a particular one is absent.

Subjects typically locate a sought-after item halfway through the search. The slope of the curve relating the search time to the number of elements in the array shows how long it takes to examine each element. Such experiments indicate that 30 to 50 ms is required to compare each element with the target item (Figure 21–5C). Knowing what kinds of visual features allow a parallel search and the reaction time for detecting features in a serial search provides important clues to the underlying neural mechanisms.

The psychophysical methods described in the previous section provide objective techniques for analyzing sensations evoked by particular stimuli. These quantitative measures have been combined with neurophysiological techniques to study the neural mechanisms that transform sensory signals into specific percepts.

This approach to the neural coding problem was pioneered by Vernon Mountcastle in the 1960s. He showed that neurophysiological recordings from individual sensory neurons in the peripheral and central nervous system provide a statistical description of the neural activity evoked by a physical stimulus. He then tested hypotheses to determine which quantitative aspects of the neural response might correspond to psychophysical measurements in sensory tasks, and just as important, which do not.

The study of neural coding of information is fundamental to understanding how the brain works. A neural code describes the relationship between the activity in a specified neural population and its functional consequences for the operations that follow. The sensory systems provide a useful avenue to the study of neural coding in the brain because both the input and output of these systems can be precisely defined and quantified. Experimenters can control the physical stimuli provided to sensory receptors and measure the resulting sensations evoked by them using a variety of psychophysical techniques. By recording neuronal activity at various stages of sensory processing, neuro scientists attempt to decipher the codes that convey information in peripheral nerves and in the brain, and analyze the transformation of signals along pathways in the cerebral cortex. Indeed, study of the details of neural coding may lead to insight into the coding principles that underlie cognition.

When analyzing sensory experience it is important to realize that our conscious sensations differ qualitatively from the physical properties of stimuli because, as Kant and the idealists predicted, the nervous system extracts only certain pieces of information from each stimulus while ignoring others. It then interprets this information within the constraints of the brain's intrinsic structure and previous experience. Thus we receive electromagnetic waves of different frequencies, but we see them as colors. We receive pressure waves from objects vibrating at different frequencies, but we hear sounds, words, and music. We encounter chemical compounds floating in the air or water, but we experience them as smells and tastes. Colors, tones, smells, and tastes are mental creations constructed by the brain out of sensory experience. They do not exist as such outside the brain.

The dominant research strategy in sensory neuro science is to follow the flow of sensory information from receptors toward the cognitive centers of the brain, attempting to understand the processing mechanisms that occur at each synaptic relay and how they shape our internal representation of the external world. The neural coding of sensory information is better understood at the early stages of processing than at later stages.

Sensory Receptors Are Responsive to a Single Type of Stimulus Energy

It is often said that the power of the brain lies in the millions of neurons processing information in parallel. That formulation, however, does not capture the essential difference between the brain and all the other organs of the body. The power of a kidney or a muscle lies in the parallel action of many cells, each doing the same thing; if we understand a muscle cell, we essentially understand how a whole muscle works. The power of the brain lies in the parallel action of millions of cells, each doing something different; to understand the brain we need to understand how its tasks are organized and how individual neurons carry out those tasks.

Functional differences between sensory systems arise from the different stimulus energies that drive them and the discrete pathways that comprise each system. Because of these characteristics each neuron performs a specific task, and the train of action potentials it produces has a specific functional significance for all postsynaptic neurons. This basic idea was expressed in the theory of specificity set forward by Charles Bell and Johannes Müller in the 19th century and remains one of the cornerstones of sensory neuroscience.

The richness of sensory experience begins with millions of highly specific sensory receptors. Each receptor responds to a specific kind of energy at specific locations on the body and sometimes only to energy with a particular temporal or spatial pattern. The receptor transforms the stimulus energy into electrical energy, thus establishing a common signaling mechanism in all sensory systems. The amplitude and duration of the electrical signal produced by the receptor, termed the receptor potential, are related to the intensity and time course of stimulation of the receptor. The process by which specific stimulus energy is converted into an electrical signal is called stimulus transduction.

Sensory receptors are morphologically specialized to transduce specific forms of energy, and each receptor has a specialized anatomical region where stimulus transduction occurs. Most receptors are optimally selective for a single type of stimulus energy, a property termed receptor specificity. We see particular colors, for example, because we have receptors that are selectively sensitive to photons with specific wavelengths, and we smell particular odors because we have receptors that bind specific odorant molecules (Figure 21–6).

Human sensory receptors are classified as mechanoreceptors, chemoreceptors, photoreceptors, or thermoreceptors (Table 21–1). Mechanoreceptors and chemoreceptors are the most widespread and the most varied in form and function.

Six different kinds of mechanoreceptors that sense skin deformation, motion, stretch, and vibration are responsible for the sense of touch. Muscles contain three kinds of mechanoreceptors that signal muscle length, velocity, and force, whereas other mechanoreceptors in the joint capsule signal joint angle. Hearing is based on two kinds of mechanoreceptors, inner and outer hair cells, that transduce motion of the basilar membrane in the inner ear. Other hair cells in the vestibular labyrinth sense motion and acceleration of the fluids of the inner ear to signal head motion and orientation. Visceral mechanoreceptors detect the distension of internal organs such as the bowel and bladder. Osmoreceptors in the brain, which sense the state of hydration, are activated when a cell swells. Certain mechanoreceptors report extreme distortion that threatens to damage tissue; their signals reach pain centers in the brain.

Chemoreceptors are responsible for olfaction, gustation, itch, pain, and many visceral sensations. A significant part of pain is due to chemoreceptors that detect molecules spilled into the extracellular fluid by tissue injury and molecules that are part of the inflammatory response. Several kinds of thermoreceptors in the skin sense skin warming and cooling. Another thermoreceptor, which monitors blood temperature in the hypothalamus, is mainly responsible for whether we feel warm or cold.

Vision is mediated by four kinds of photoreceptors in the retina. The light sensitivities of these receptors define the visible spectrum. The photopigments in rods and cones detect electromagnetic energy of wavelengths that span the range 390 to 670 nm (Figure 21–7A). Unlike some other species, such as birds or reptiles, humans do not detect ultraviolet light or infrared radiation because we lack receptors that detect the appropriate short or long wavelengths. Similarly, radio waves and microwave energy bands are not perceived because humans have not evolved receptors for these frequencies.

Multiple Subclasses of Sensory Receptors Are Found in Each Sense Organ

Sensory receptors are found in specialized epithelia called sense organs, principally the eye, ear, nose, tongue, and skin. The arrangement of receptors in an organized structure allows further specialization of function within each sensory system.

Each major sensory system has several constituent qualities or submodalities. For example, taste can be sweet, sour, salty, or bitter; objects that we see differ in color; and touch has qualities of temperature, texture, and rigidity. Submodalities exist because each class of receptors contains a variety of specialized receptors that respond to limited ranges of stimulus energies.

The receptor behaves as a filter for a narrow range or bandwidth of energy. For example, an individual photoreceptor is not sensitive to all wavelengths of light but only to a small part of the spectrum. We say that a receptor is tuned to an optimal or best stimulus, the unique stimulus that activates the receptor at low energy and evokes the strongest response. As a result, we can plot a tuning curve for each receptor based on physiological experiments (the white and black curves in Figure 21–7A). The tuning curve shows the range of sensitivity of the receptor, including its threshold, the minimum stimulus intensity at which the receptor is activated. For example, blue cones in the retina are most sensitive to light of 437 nm; for that reason, they are also termed S or short-wavelength receptors. Green cones, termed M receptors for their sensitivity to middle wavelengths, respond best to 533 nm; red cones, the L or long-wavelength receptors, respond most vigorously to 564 nm wavelengths. The blue, green, and red cones respond to other wavelengths of light but these responses are weaker (see Chapter 26).

The graded sensitivity of photoreceptors means that each rod and cone responds to a wide spectrum of colors yet signals a specific wavelength by the amplitude of the evoked receptor potential. However, because the tuning curve is symmetric around the best frequency, wavelengths of greater or lesser values may evoke identical responses. For example, red cones respond equally well to light of 520 and 600 nm. How does the brain interpret these signals? The answer lies with the green and blue cones. Green cones respond very strongly to light of 520 nm, as it is close to their preferred wavelength, but respond weakly to 600 nm light. Blue cones do not respond to 600 nm light and are barely activated at 520 nm. As a result, 520 nm light is perceived as green, whereas 600 nm is seen as orange. Thus we are able to perceive a spectrum of colors through varying combinations of photoreceptors.

Similarly, the complex flavors we perceive when eating are a result of combinations of chemoreceptors of varying affinities for natural ligands. The broad tuning curves of a large number of distinct olfactory and gustatory receptors afford endless combinatorial possibilities.

Neural Firing Patterns Transmit Sensory Information to the Brain

The receptor potential generated by an adequate stimulus produces a local depolarization or hyperpolarization of the sensory receptor cell. However, the sense organs are located at distances far enough from the central nervous system that passive propagation cannot suffice to convey signals there. To communicate sensory information to the brain a second step in neural coding must occur. The change in membrane potential produced by the sensory stimulus is transformed into action potentials that can be propagated over long distances.

Action potentials are generated in olfactory sensory neurons and dorsal root ganglion neurons of the somatosensory system whose axons project directly to the central nervous system. In the auditory, vestibular, and gustatory (taste) systems the receptor cells make synaptic contact with the peripheral branches of the sensory axons that form cranial nerves VIII, VII, and IX. The retina has the most elaborate neural network for processing sensory information. Photoreceptors send signals through a series of local interneurons to retinal ganglion cells that transform visual information into bursts of action potentials that travel to the brain through the optic nerve.

Sensory receptors encode the intensity of the stimulus in the amplitude of the receptor potential. This analog signal of intensity is transformed into a digital pulse code in which the frequency of action potentials is proportional to the intensity of the stimulus (see Figure 21–3A). The notion of an analog-to-digital transformation dates back to 1925 when Edgar Adrian and Yngve Zotterman discovered the all-or-none properties of the action potential in sensory neurons. Zotterman would later write:

Despite the rather crude recording instruments available at that time, Adrian and Zotterman discovered that the frequency of firing—the number of action potentials per second—varies with the strength of the stimulus and the time over which it has been in action; stronger stimuli evoked larger receptor potentials that generated a greater number and a higher frequency of action potentials.

In later years, as recording technology improved and digital computers allowed precise quantification of the timing of action potentials, Mountcastle and his colleagues demonstrated a precise correlation between sensory thresholds and neural responses, as well as the parametric relationship between neural firing rates and self-reports of the intensity of sensations (see Figure 21–3). They also found that the dynamics of the spike train conveys important information about fluctuations of the stimulus, such as the frequency of vibration or a change in rate of movement. Humans can report changes in sensory experience that correspond to alterations in the firing patterns of sensory neurons in the range of a few milliseconds.

The temporal properties of a changing stimulus are encoded as changes in the pattern of sensory neuron activity. Many sensory neurons signal the rate at which stimulus intensity changes by rapidly altering their firing rates. For example, in slowly adapting mechanoreceptors the initial spike discharge when a probe touches the skin is proportional to both the speed at which the skin is indented and the total amount of pressure (Figure 21–8A). During steady pressure the firing rate slows to a level proportional to skin indentation. Firing stops when the probe is retracted. Thus, neurons signal important properties of stimuli not only when they fire but also when they stop firing.

The instantaneous firing patterns of sensory neurons are as important to sensory perception as the total number of spikes fired over long periods. Steady rhythmic firing in nerves innervating the skin is perceived as vibration or steady pressure. Bursting patterns may be perceived as motion. If a stimulus persists unchanged for several minutes without a change in position or amplitude, the neural response diminishes and sensation is lost, a condition called receptor adaptation.

Receptor adaptation is thought to be an important neural basis of perceptual adaptation, whereby a constant stimulus fades from consciousness. Receptors that respond to prolonged and constant stimulation, known as slowly adapting receptors, encode stimulus duration by generating action potentials throughout the period of stimulation. In contrast, rapidly adapting receptors respond only at the beginning or end of a stimulus; they cease firing in response to constant amplitude stimulation and are active only when the stimulus intensity increases or decreases (Figure 21–8B).

The existence of two kinds of receptors—rapidly and slowly adapting sensors—illustrates another important principle of sensory coding. Sensory systems detect contrasts in discrete stimuli, changes in the temporal and spatial patterns of stimulation.

The intensity of a stimulus is also represented in the brain by the total number of active neurons in the receptor population. This type of population code depends on the fact that individual receptors in a sensory system differ in their sensory thresholds or in their affinity for particular molecules. Most sensory systems have low- and high-threshold receptors. When stimulus intensity changes from weak to strong, low-threshold receptors are first recruited, followed by high-threshold receptors. Parallel processing in low- and high-threshold pathways extends the dynamic range of a sensory system by overcoming the maximum firing rate of 1,000 spikes per second imposed by the absolute refractory period. For example, rod cells in the retina are activated in very dim light but reach their maximal receptor potentials in daylight. Cone cells do not respond in dim light but sense differences in brightness in daylight. The combination of the two types of photoreceptors allows us to perceive light intensity over several orders of magnitude.

As this discussion illustrates, the possibilities for information coding through temporal patterning within and between neurons in a population are enormous. For example, the timing of action potentials in the presynaptic cell can determine whether the postsynaptic cell fires. Two action potentials that arrive synchronously or nearly so will drive the postsynaptic neuron's membrane potential much further toward or away from the threshold for an action potential than would asynchronous action potentials. The timing of action potentials between neurons also has a profound effect on long-term potentiation and long-term depression at synapses (see Chapter 67).

The Receptive Field of a Sensory Neuron Conveys Spatial Information

Populations of neurons are also important for conveying the spatial properties of stimuli in a variety of modalities. The spatial attributes of visual, tactile, and auditory stimuli include the location, dimensions, shape, and tonal frequency of the stimuli. The spatial attributes of proprioceptive stimuli include the length of muscles, joint postures, and the body's orientation in the gravitational field. These properties are linked to the anatomical arrangement of receptors within each sense organ.

The position of a sensory neuron in the sense organ is a major element of the specific information conveyed by that neuron (Figure 21–9). The skin area or region of space or tonal domain in which stimuli can activate a sensory neuron is called its receptive field. The skin area or region of space from which a sensation seems to arise is called the neuron's perceptive field. The two usually coincide.

The dimensions of receptive fields play an important role in the ability of a sensory system to encode spatial information. The objects that we see with our eyes or hold in our hands are much larger than the receptive field of an individual sensory neuron, and therefore stimulate groups of adjacent receptors. The size of the stimulus therefore influences the total number of receptors that are activated. In this manner the spatial distribution of active and silent receptors provides a neural image of the size and contours of the stimulus. This pattern is called an isomorphic representation of the stimulus.

Each receptor in the active population encodes the type of energy applied to the receptive field, the local stimulus magnitude, and its temporal properties. For example, auditory codes describe the tonal frequency, loudness, and duration of sound-pressure waves hitting the ear, whereas visual codes describe the hue, brightness, and time course of light hitting the retina. The neural representation of an object or scene is therefore composed of a mosaic of individual receptors that collectively signal its size, contours, texture, color, and temperature.

A good way to visualize the neural activity of a population of neurons, and to grasp the range of possibilities for population coding, is to think of neurons as points in a visual display that flash brightly whenever an action potential occurs. If the action potentials occur at random times, one would perceive a disorganized pattern of flickering dots like the "snow" on old-style television screens without a signal. However, if groups of pixels are turned on and off synchronously, coherent spatial patterns appear. Similarly, when a horizontal bar of light stimulates a row of adjacent photoreceptors in the retina, action potentials are generated in neighboring ganglion cells. Although each photoreceptor simply registers light in its receptive field, the pattern of a bar emerges from the population of active ganglion cells. Neurons in the central nervous system decipher the image of a bar by responding preferentially to specific ensembles of active receptors.

Synchronous patterns of activity in sensory neuron populations convey the spatial dimensions of the stimulus but do not in themselves signal its intensity. The brightness and contours of a video image are created by modulation of the luminance of each pixel. Similarly, in neural codes signal strength is conveyed by the impulse rates of the individual neurons. This is called rate coding. The temporal integration of action potentials that occurs at synapses smooths the staccato on-off firing patterns into a continuous modulated signal analogous to the gray scale of a video monitor. High firing rates in this model yield white zones, intermediate rates produce gray zones, and silence gives a black region. Rate coding thereby allows the population of neurons to simultaneously transmit the spatial properties and intensity of stimuli.

The spatial resolution of a sensory system is proportional to the total number of receptor neurons and how their receptive fields are apportioned within the population (Figure 21–10). Regions of a sense organ with a high density of receptors, such as the central retina (the fovea), have small receptive fields because the terminals of each sensory neuron are confined to a local cluster of receptors. Each retinal ganglion cell in the fovea measures the average light intensity in a small spot of the visual field; but because there are so many of them, the population of cells in the fovea transmits a very detailed representation of the visual scene. Ganglion cells in the periphery of the retina have larger receptive fields because the receptor density is much lower. The dendrites of each ganglion cell receive information from a wider area of the retina, and thereby integrate light intensity over a greater portion of the visual field. This arrangement yields a less detailed image of the scene (Figure 21–10A). Similarly, the region of the body most often used to touch objects is the hand. Not surprisingly, mechanoreceptors for touch are concentrated in the fingertips, and the receptive fields on the hand are smaller than those on the arm or trunk.

Spatial coding is ubiquitous for two reasons. First, it takes advantage of the parallel architecture of the nervous system. The number of neurons in each unimodal area of sensory cortex is approximately 100 million. Thus the possible number of spatial patterns of neural activity greatly exceeds the number of atoms in the universe. Second, each neuron is a spatial as well as a temporal decoder: It fires only when many of its excitatory synapses receive action potentials and most of the inhibitory synapses do not. That is, it fires in response to some patterns of stimulation and not others. The fact that on average each cortical neuron has 10,000 synapses makes the number of spatial coding possibilities enormous.

Spatial codes are sometimes called vector codes from the mathematical idea of vector spaces. The firing rate of each neuron in a population can be plotted in a coordinate system with multiple axes such as modality, location, intensity, and time. The neural components along these axes combine to form a vector that represents the population's activity (see Figure 21–7B). The vector interpretation is useful because it makes available powerful mathematical techniques.

The fragmentation of a stimulus into components, each encoded by an individual neuron, is the initial step in sensory processing. Assembly of the components into an internal representation of an object occurs within neural networks in the brain. This process allows the brain to abstract certain features of an object, person, scene, or external event from the detailed receptor input. As a result, the internal representation formed in the brain may exaggerate some features that are important at the moment while ignoring others. In this sense our percepts are not perfect mirrors of the stimuli that evoke them but instead a creation of the mind.

Bell and Müller realized that the richness provided by the specificity of our receptors would be lost without connections to brain centers that are as rich and varied as the receptors themselves. A sensory neuron's action potentials have a specific effect on our sensory experience because of the neuron's central connections, not because of the stimulus that evokes the action potentials. Action potentials in nerve fibers of the cochlea, for example, evoke the sensation of a tone whether activated by sound waves or by electrical stimulation with a neural prosthesis.

Each class of sensory receptors makes connections with structures in the central nervous system that are dedicated to one sensory modality, at least in the early stages of information processing. Thus sight or touch is experienced because specific central nervous structures are activated. Each sensory modality is therefore represented by the ensemble of central neurons connected to a specific class of receptors. Such ensembles of neurons are referred to as sensory systems, which include the somatosensory, visual, auditory, vestibular, olfactory, and gustatory systems (see Table 21–1).

Sensory information flows through pathways dedicated to conveying stimulus information before ending in brain regions that are more clearly concerned with cognition and action than with sensory processing. However, synaptic relays in sensory pathways do more than simply pass on signals received. Each relay neuron receives convergent excitatory synaptic inputs from many neurons in the presynaptic pathway. Likewise, each receptor neuron excites a large number of postsynaptic neurons. In addition, inhibitory interneurons in the relay nucleus modulate the excitability of relay neurons, thereby regulating the amount of sensory information transmitted centrally to higher levels of the network (Figure 21–11).

Like the primary sensory neurons in the periphery, neurons in the central sensory pathways have specific receptive and perceptive fields. The specificity of a central neuron is determined by the receptive fields of the neurons that excite and inhibit it. The neuron responds optimally to stimuli that simultaneously activate a particular set of presynaptic excitatory neurons. The neuron's receptive field is also shaped by inhibitory input. The inhibitory region of a receptive field provides an important mechanism for enhancing the contrast between stimuli and thus gives the sensory systems additional power to resolve spatial detail.

The activity of sensory neurons in the brain is more variable from trial to trial than that of sensory neurons in the periphery. Central sensory neurons also fire irregularly before and after stimulation, and during periods when no stimuli are present. The variability of the evoked central responses is a result of several factors: the subject's state of alertness, whether his or her attention is engaged, previous experience of that stimulus, and recent activation of the pathway by similar stimuli. Similarly, behavioral conditions during stimulus presentation, subjective intentions, motor plans that may evoke feedback responses, or intrinsic oscillations of the neuron's membrane potential can all modify the incoming sensory information. For these reasons, neural responses to sensory stimulation or during motor behaviors are usually illustrated both by raster plots that depict the trial-to-trial variability of firing (see Figure 21–15) and by histograms that average neural activity across trials.

The Receptor Surface Is Represented Topographically in Central Nuclei

As we saw in Chapter 17, receptor axons terminate in the brain in an orderly manner forming maps of the receptor sheet. At all levels of a sensory system the stimulus sensitivity of individual neurons varies in an orderly way across a nucleus. Maps of primary specificity—the qualities to which neurons are most narrowly tuned—provide clues to the functional organization of a nucleus.

In the first relay nuclei of the somatosensory, visual, and auditory systems adjacent neurons represent adjacent areas of the body, retina, and cochlea, respectively. The organization of these nuclei is thus said to be somatotopic, retinotopic, or tonotopic. Nuclei in the auditory system are called tonotopic because the inputs from cochlear hair cells are arranged to create an orderly shift in frequency sensitivity from cell to cell, reflecting the functional organization of the hair cells in the cochlea (Figure 21–12). In other words, the firing of a particular neuron in a population signals the location of a stimulus on the receptor surface.

Neurons in the primary sensory areas of the cerebral cortex continue to represent location-specific features of a stimulus, and the functional maps of these areas are also somatotopic, retinotopic, or tonotopic. However, at higher levels within each sensory system neurons are more sharply selective of other stimulus features. Thus central auditory neurons are less selective for frequency and more selective for certain kinds of sound. For example, some neurons are specific for vocalizations by members of the same species. In each successive nucleus the spatial organization is progressively lost as neurons become less concerned with the descriptive features of stimuli and more concerned with properties of behavioral importance in that modality (Figure 21–13).

One of the most important insights into feature detection in the cortex arose from combined physiological and anatomical studies of the cortical visual pathways by Mortimer Mishkin and Leslie Ungerleider. They discovered that sensory information arriving in the primary visual areas is divided in two parallel pathways. One pathway conveys information needed for immediate action and the other information needed for classification of images. Visual features that identify what an object is are transmitted in a ventral pathway to the temporal lobe and eventually to the hippocampus and entorhinal cortex. Visual information about where the object is located and its size and shape is transmitted in a more dorsal pathway to the parietal lobe and eventually to the motor areas of frontal cortex (Figure 21–14).

The ventral and dorsal streams of sensory information are used as the basis of two major forms of memory: semantic memory, which allows us to talk about objects or persons, and procedural memory, which we use to interact with objects or persons.

Ventral and dorsal streams are also evident in other sensory systems. In the auditory system acoustic information from speech is transmitted to Wernicke's area in the temporal lobe, which has a strong role in language comprehension, and to the Broca area in the frontal cortex, which is involved in speech production. In the somatosensory system information about an object's features such as size and shape is transmitted to ventral areas of parietal cortex for object recognition. Tactile information about object size, weight, and texture is also communicated to posterior parietal and frontal motor areas, where it is needed to plan handling of the object.

Feedback Regulates Sensory Coding

The sensory systems are not simply assembly lines that reassemble initial neural representations into ones that are more appropriate for cognition. That view is at odds with our own experience of sensation and perception. We have enormous control over perception and consciousness at high and low levels. At a high level, for example, we can switch our attention from the subject matter of a painting to the painter's use of form, color, and texture.

At a much lower level we can to some extent control the sensations that reach consciousness. We may, for example, watch television to take our minds off the pain of a sprained ankle. Direct, volitional control of the sensory information that reaches consciousness can be readily demonstrated by suddenly directing your attention to a body part, such as the fingers of your left hand, to which you were oblivious as you were attending to this text. Sensations from the fingers flood consciousness until attention is redirected to the text. Neural recordings in somatosensory and visual cortex indicate that neurons change their sensitivity, as reflected in their firing rates, but not their selectivity for stimuli (Figure 21–15).

Each of the sensory systems also has feedback projections. Each primary sensory cortex has extensive projections back to its principal relay nucleus in the thalamus. In fact, the number of feedback axons exceeds the number of afferent axons from the thalamus to the cortex. These projections have an important function that is not yet clear. One possibility is that they modulate the responsiveness of certain neurons when attention and vigilance change or during motor tasks.

Higher centers in the brain are also able to modulate the responsiveness of sensory receptors. For example, neurons in the motor cortex can alter the sensitivity of sensory receptors in skeletal muscle that signal muscle length. Activation of gamma motor neurons by corticospinal pathways enhances the sensory responses of muscle spindle afferents to stretch. Neurons in the brain stem can directly modulate the frequency sensitivity of hair cells in the cochlea. So even at the level of individual sensory receptors the information sent to the brain signals properties of both the stimulus and the subject who receives the information.

Top-Down Learning Mechanisms Influence Sensory Processing

What we perceive is always some combination of the sensory stimulus itself and the memories it evokes. The late coach of the Boston Celtics, Red Auerbach, once reflected that, in motivating a team, "It's not what you say to them, but what they hear that matters."

The relationship between perception and memory was originally developed by the empiricists, particularly the associationist philosophers James Mill and his son John Stuart Mill. Their idea was that sensory and perceptual experiences that occur together or in close succession, particularly those that do so repeatedly, become associated so that one thereafter triggers the other. It is easy to see how associationism verges on a theory of knowledge, thought, intelligence, and even consciousness. Association is a powerful mechanism and much of learning consists of committing associations to memory through repeated exposure.

We understand in principle how a network of neurons can "recognize" a specific pattern of inputs from a population of presynaptic neurons. The mechanism is called template matching. Each neuron in the target population has a pattern of excitatory and inhibitory presynaptic connections. If the pattern of arriving action potentials fits the postsynaptic neuron's pattern of synaptic connections even approximately—activates many of its excitatory synapses but mostly avoids activating its inhibitory synapses—the target neuron fires.

This mechanism also contains the essential elements of association. Suppose the pattern of inputs is a representation of the letter A, and the pattern of action potentials evoked in the target population is the representation of the letter B; B then becomes associated with A. Exposure to A evokes the internal representation of B. If the representation of B is fed back to this same target population and it evokes the representation of a C, an associative chain has been established. Whether the brain uses a mechanism like this is not known, but the speed of recognition and association together with the parallel architecture of the brain suggest that something like this must occur.

This template-matching mechanism is very powerful and forms the basis of virtually all computer-based pattern-recognition schemes. Nonetheless, no artificial scheme comes close to the human ability to recognize patterns of many kinds because artificial systems cannot handle the extreme variability of sensory stimuli in the real world. In general, attempts to recognize stimuli by referring to stored records of earlier stimuli fail because of this variability; subsequent occurrences of the same or similar stimuli rarely match earlier occurrences. The general approach among computer scientists is to search for a way to separate information about the form and structure of a stimulus from information about the properties that vary from one exposure to the next.

Whether the brain solves the recognition problem in this way is uncertain. There is currently much evidence that the neural representation of a stimulus conveyed by a large population of receptor neurons in the initial pathways of sensory systems is an isomorphic representation of the stimulus. Successive synaptic regions transform these initial neural representations into abstracted representations of our environment that we are beginning to decipher. In contrast, we barely understand the top-down mechanisms by which incoming sensory information invokes memories of past occurrences and activates our subjective prejudices and opinions. These topics are difficult to analyze experimentally, particularly in animal models. When we do understand these neural codes it is likely that we will be on the verge of understanding cognition, the way in which information is coded in memory. That is what makes the study of neural coding so challenging and exciting.

Our sensory systems provide the means by which we perceive the external world, remain alert, form a body image, and regulate our movements. Sensations arise when external stimuli interact with some of the billion sensory receptors that innervate every organ of the body. The information detected by these receptors is conveyed to the brain as trains of action potentials traveling along individual sensory axons.

These messages are analyzed centrally by several million sensory neurons performing different, specific functions in parallel. Each sensory neuron extracts highly specific and localized information about the external or internal environment and in turn has a specific effect on sensation and cognition because it projects to specific places in the brain that have specific sensory functions.

All sensory systems respond to four elementary features of stimuli—modality, location, intensity, and duration. The diverse sensations we experience—the sensory modalities—reflect different forms of energy that are transformed by receptors into depolarizing or hyperpolarizing electrical signals called receptor potentials. Receptors specialized for particular forms of energy, and sensitive to particular ranges of the energy bandwidth, allow humans to sense many kinds of mechanical, thermal, chemical, and electromagnetic events. To maintain the specificity of each modality within the nervous system, receptor axons are segregated into discrete anatomical pathways that terminate in unimodal nuclei. After about a dozen synaptic steps in each sensory system, neural activity converges on neuronal groups whose function is polymodal and more directly cognitive.

The location and spatial dimensions of a stimulus are conveyed through each receptor's receptive field, the precise area in the sensory domain in which stimulation activates the receptor. The identity of the active sensory neurons therefore signals not only the modality of a stimulus, but also the place where it occurs. The intensity and duration of stimulation are represented by the amplitude and time course of the receptor potential and by the total number of receptors activated. In the brain, intensity is encoded in the frequency of firing, which is proportional to the strength of the stimulus. The temporal features of a stimulus, such as duration and changes in magnitude, are signaled by the dynamics of the spike train.

The pattern of action potentials in peripheral nerves and in the brain gives rise to sensations whose qualities can be measured directly using a variety of psychophysical paradigms such as magnitude estimation and signal detection and discrimination tasks. Reaction times to stimuli also provide a means for measuring the intensity of stimulation and the ease of sensory discrimination in both human and animal subjects.

The richness of sensory experience—the complexity of sounds in a Mahler symphony, the subtle layering of color and texture in views of the Grand Canyon, or the multiple flavors of a salsa—requires the activation of large ensembles of receptors acting in parallel, each one signaling a particular aspect of a stimulus. The neural activity in a set of thousands or millions of neurons should be thought of as coordinated activity that conveys a "neural image" of specific properties of the external world.

Sensory information in the central nervous system is processed in stages, in the sequential relay nuclei of the spinal cord, brain stem, thalamus, and cerebral cortex. Each nucleus integrates sensory inputs from adjacent receptors and, using networks of inhibitory neurons, emphasizes the strongest signals. Processing of sensory information in the cerebral cortex occurs in multiple cortical areas in parallel, and is not strictly hierarchical. Feedback connections from areas of the brain involved in cognition, memory, and motor planning control the incoming stream of sensory information, allowing us to interpret sensory stimulation in the context of past experience and current goals.