Does stimulus complexity determine whether working memory storage relies on prefrontal or sensory cortex?

How do we define the complexity of a stimulus? Readers should note that we are generally referring to the salient portion or property of a stimulus, rather than to the whole stimulus. To fully describe an auditory pure tone, we would need to give the pitch (frequency) and amplitude, but auditory STM tasks are often based on comparing only the frequencies of two stimuli. In this sense, the stimulus is multidimensional, but the salient property of the stimulus can be represented as a scalar value—just the frequency. Consider a task where subjects are presented with an image of a cat or a dog, followed by a delay period, followed by a second image of a cat or a dog, and are asked to decide whether the two images belong to the same category (i.e., are they both cats or both dogs?) or to different categories. (For research along this line, see Freedman et al., 2001, discussed below.) The stimuli (images) are obviously quite complex, but the salient aspect of the stimuli (category) is simple and scalar.

To quantify this notion, the storage of information on a computer is perhaps an appropriate analogy. To store the frequency of a simple auditory stimulus on a computer requires a single variable (in other words, it is a scalar, or unidimensional, quantity). To store the frequency of a complex auditory stimulus created by superimposing multiple sine waves requires multiple variables (one for each sine wave), making it a multidimensional quantity. Storing a visual image in memory would require a variable or variables for each pixel in the image. However, it should be remembered that we are generally referring to the salient aspects of a stimulus, as determined by task requirements. If subjects are presented with an image and asked to remember whether it contains green trees or red tomatoes, it is possible (perhaps probable) that subjects will retain a label such as “green” or “red,” rather than a representation of the entire image. Note also that the above is not intended to be a rigorous metric for stimulus complexity or dimensionality, and the development of such a measure would be of benefit.

Research into the neural correlates of scalar STM (in other words, memory for unidimensional stimulus properties, such as the pitch of an auditory pure tone or the duration of a stimulus) has produced strong evidence for information storage in PFC, rather than in task-relevant sensory cortex. Extensive single-cell research in nonhuman primates has revealed frontal coding for stimulus frequency (for both vibrotactile and auditory pure tones) during the delay period of memory tasks (see Romo & Salinas, 2003, for a review; see also Lemus, Hernández, & Romo, 2009; Romo, Brody, Hernández, & Lemus, 1999). In fact, the neural coding of these stimuli is surprisingly simple: Neuronal firing rates tend to be monotonic (positive or negative) functions of stimulus frequency (e.g., Romo et al., 1999). This simple neural code has facilitated several innovative EEG studies by Spitzer and colleagues (Spitzer & Blankenburg, 2011, 2012; Spitzer et al., 2013; Spitzer et al., 2010), who developed a method to decode the representation of a stored stimulus from beta-band EEG activity in frontal cortex during the delay period of memory tasks. This method has allowed decoding of memory for vibrotactile frequency, auditory pure tone frequency, and visual flicker frequency (Spitzer & Blankenburg, 2012; Spitzer et al., 2010), as well as stimulus amplitude and duration (Spitzer et al., 2013). Source localization suggested that prefrontal (possibly inferior frontal, see Spitzer et al., 2010) cortex was responsible for storage. Sensory cortex, on the other hand, was not shown to maintain stimulus representations during the delay period. It should be noted, however, that frequency information has been shown to be represented in primary somatosensory cortex, and Spitzer et al.’s EEG findings are generally consistent with single-cell work done by the group of Romo et al. (see Romo & Salinas, 2003, for a review). Furthermore, Zhou and Fuster (1996) have demonstrated maintenance of other types of stimulus information in primary somatosensory cortex, suggesting that PFC storage is not due to an inability to maintain information in somatosensory cortex. The lack of representation outside of PFC poses a challenge to arguments that delay-period activity in PFC does not code stimulus information but, rather, always represents attentional activity.

Other types of simple information have also been shown to be encoded in PFC. For example, Sakurai, Takahashi, and Inoue (2004) found frontal neurons that represented stimulus duration during the delay period of a memory-for-duration task. Categorical information has also been shown to be encoded in PFC. Freedman et al. (2001) found neurons that represented the category of novel forms of stimuli (computationally morphed images of cats and dogs) in PFC during the delay period of memory tasks. While these stimuli are, obviously, far more complex than the stimuli we discuss above, task demands (in this case, match-to-category) do not require storage of stimulus information beyond category, which can be represented as a scalar value (dog = 1, cat = 2, etc.). More recently, Lee, Kravitz, and Baker (2013) have demonstrated prefrontal encoding for stimulus category using fMRI decoding methods in humans. In general, these findings directly contradict one of the main tenets of the emergent-property model: PFC is not involved in memory storage.

In contrast to these results, however, are a group of recent imaging studies that have used various decoding methods (see Tong & Pratte, 2012) to examine the contents of memory and that have found memory storage in task-relevant (often sensory) cortex, and not in PFC. These decoding methods, which include multivoxel pattern analysis and classification methods based on linear support vector machines, allow a relatively direct examination of the contents of memory.

Serences et al. (2009) used multivoxel pattern analysis methods to identify visual stimulus information (orientation or color) in primary visual cortex during the delay period; furthermore, this activity was similar to that observed during sensory processing. Christophel et al. (2012) applied decoding methods to activity found during an fMRI study of the neural correlates of WM/STM for abstract colored stimuli. During the maintenance period of the task, stimulus information was shown to be represented in early visual cortex, but not in PFC. Intriguingly, they also identified stimulus information in posterior parietal cortex; as this area has retinotopic attentional maps, this may be related to attentional support of activity in visual cortex (Constantinidis & Steinmetz, 1996; Sereno, Pitzalis, & Martinez, 2001). Similar findings have been reported by other researchers (e.g., Emrich, Riggall, LaRocque, & Postle, 2013, reported similar findings for visual STM over different memory loads).

Using a task involving STM for moving visual stimuli, Riggall and Postle (2012) found storage of stimulus information in posterior visual regions when applying multivoxel pattern analysis to fMRI data. In contrast, their analyses revealed that prefrontal activity reflected task instructions, not memory storage. (Linden, Oosterhof, Klein, & Downing, 2012, also reported a failure to find stimulus-specific activity in PFC.) Perhaps most startlingly, Serences, Ester, Vogel, and Awh (2009) and Riggall and Postle (2012) were able to identify stimulus information in cortical regions that did not display elevated delay-period activity, suggesting that standard analyses of BOLD signals, even load-dependent signals, may not be sufficient to identify regions involved in WM/STM storage. Harrison and Tong (2009) also reported similar findings and were able to decode the orientation of a grating held in memory with approximately 80 % accuracy based on the application of a linear classifier to fMRI scans of early visual cortex, even though overall levels of sustained activity were low. Ester, Anderson, Serences, and Awh (2013) used similar methods to link the contents of visual cortex to subject performance.

Linke, Vicente-Grabovetsky, and Cusack (2011) applied fMRI decoding methods to sensory cortex during an auditory STM (for frequency) task. While frequency-specific increased activity was identified during stimulus encoding, it was absent during the maintenance period. Furthermore, during maintenance, activity was actually suppressed below baseline, and this suppression was frequency specific, suggesting not only does auditory cortex not serve as a storage system for auditory frequency information, but also that it is actively inhibited, potentially to protect the contents of memory in neural systems downstream.

Support for memory maintenance in sensory cortex also comes from other methodologies, such as single-cell recording. Zhou and Fuster (1996) reported object-selective delay-period activity in primary somatosensory neurons during a haptic memory task. Hayden and Gallant (2013) found stimulus-specific activity in V4 neurons during a delayed match-to-sample task. In addition, recent TMS research has supported a role for language cortex in verbal WM storage: Acheson, Hamidi, Binder, and Postle (2011) observed verbal WM deficits when applying TMS to regions of temporal cortex involved in language production. Note that we are focusing on activity-related memory in sensory cortex, not “structural” memory (i.e., plasticity; Menning, Roberts, & Pantev, 2000; Weinberger, 2004).

In an intriguing study that poses a potential problem for our argument, Harris, Miniussi, Harris, and Diamond (2002) reported negative effects on vibrotactile STM when TMS was applied over contralateral primary somatosensory cortex shortly (300 or 600 ms) after target stimulus offset, but not when applied late (900 or 1,200 ms), after stimulus offset, which could potentially be interpreted as evidence for STM storage in SI. Harris et al. suggested that the difference between this finding and Romo et al.’s (1999; Romo & Salinas, 2003) findings on PFC storage could be due to the lack of training Harris et al.’s subjects received, compared to Romo et al.’s nonhuman subjects. This seems unlikely, given Spitzer et al.’s (2010; Spitzer & Blankenburg, 2011, 2012) findings of PFC storage in untrained human subjects. We have recently provided an alternate view: TMS over early sensory cortex increases neural “noise,” which then feeds forward into storage systems in PFC; computational modeling of this hypothesis has provided results consistent with Harris et al.’s (2002) results (Bancroft, Hogeveen, Hockley, & Servos, under review).

Serences et al. (2009) reported finding information storage in early visual cortex when the salient stimulus property was the orientation of a Gabor stimulus. Since orientation (e.g., in degrees from some baseline) can be represented as a scalar quantity, we might expect to find PFC storage, rather than sensory storage. However, it is possible that task parameters made it difficult for subjects to encode a scalar representation of the salient stimulus property: The stimulus was presented for 1 s but flickered on and off at a rate of 5 Hz. A similar study by Ester et al. (2013) also flickered their grating on and off. It is possible that subjects were unable to extract a reliable scalar representation of stimulus orientation, and as such, the stimulus representation in visual cortex was maintained. It is also possible that information maintenance in visual cortex reflects the retention of spatial information; the visual displays used by Serences et al. and Ester et al. include information other than orientation (for example, color, stimulus size, etc.). As was reported by Christophel et al. (2012), areas in parietal cortex involved in visual attention also demonstrated information maintenance. Finally, neither Serences et al. nor Ester et al. reported whole-brain analyses but, rather, focused on visual cortex; as such, it is possible that a scalar representation did exist in PFC, but was not identified during analysis. While these studies pose a potential problem for our argument, we would argue that, instead, they are evidence for the influence of task parameters and demands on neural coding.

The notion that sensory (or other task-relevant) cortex can be responsible for WM/STM maintenance is well-supported by recent research. However, it is clearly not the only possible storage substrate.

How can we account for the prefrontal encoding of representations of some types of stimuli (scalar stimuli; novel forms of stimuli, etc.) but encoding in task-relevant cortex of many other forms of stimuli? Consider: Information about visual flicker rate is stored in prefrontal cortex (Spitzer & Blankenburg, 2012), but a representation of abstract, “swirled” stimuli is stored in early visual cortex (Christophel et al., 2012), and WM for faces and bodies appears to rely on higher visual areas (Linden et al., 2012). In the tactile modality, vibrotactile frequency information is found in PFC (Romo & Salinas, 2003; Spitzer et al., 2010), but information about surface texture or the orientation of striations on an object appears to be maintained in primary somatosensory cortex (Zhou & Fuster, 1996). Similar results can be found in other sensory modalities (Postle, 2006; Spitzer & Blankenburg, 2012; Spitzer et al., 2013).

Can such a disparate set of results be explained by fractionation of WM/STM into multiple systems, specialized for a specific type of stimulus or task? While this approach has been applied to memory in the past (e.g., Wilson, O Scalaidhe, & Goldman-Rakic, 1993), Postle (2006) pointed out that the increasing body of literature on memory for different types of stimuli suggests that we would need to fractionate WM/STM into potentially hundreds of different memory systems—hardly an elegant approach. It is much more parsimonious to argue that WM/STM relies on task-relevant sensory or cognitive cortex for storage, in combination with domain-general executive/attentional systems to help maintain information in task-relevant cortex, as well as a flexible, adaptable system in PFC to represent simple or novel types of stimuli.

There is a pattern in these findings, and we suggest that storage location can be determined (at least partially) by the complexity or dimensionality of stimuli. Simple, low-dimensional stimuli (in other words, those that are easily represented on a neural level) tend to be represented in PFC, while more complex stimuli tend to be represented in task-relevant cortex. The more complex a stimulus is, the more likely it is to require specialized neural circuitry to be processed and stored—neural circuitry such as is found in relevant sensory or language cortex. In contrast, simple stimuli (such as scalar representations of stimulus frequency) are less likely to require the specialized hardware of sensory cortex and can be found in PFC.

This raises an important question. If information can be stored in task-relevant cortex, why have the ability to store information in PFC? Why not rely solely on sensory cortex, language cortex, and so forth (at the very least, for information that can already by represented in those regions, if not for novel forms of stimuli)? Co-opting sensory (or other task-relevant) cortex for memory storage can interfere with the concurrent processing of sensory stimuli and may also leave the contents of memory particularly vulnerable to interference. Indeed, WM span tasks (one of the most common classes of WM task; see Conway et al., 2005, for a review) rely on a concurrent processing task during memory maintenance to increase difficulty and force the recruitment of executive/attentional resources. A recent study by Anderson, Ester, Serences, and Awh (2013; see also Ester et al., 2013) reported that the accuracy of population tuning curves in visual cortex was affected by the size of a set of visual stimuli to be attended; it appears likely that similar effects could be found for neural populations when stimuli are being held in memory, rather than simply attended. Such an effect was found by Konstantinou and Lavie (2013), who found decreased perceptual sensitivity when increasing visual STM load, but not when increasing demand on executive processes, independent of STM load. The ability to offload certain types of information onto PFC can assist with minimizing the impact of memory maintenance on concurrent tasks, as well as helping to protect the contents of memory from interference caused by concurrent processing.

The neural organization of WM and STM appears to be even more fluid and dynamic than Postle (2006) suggested. We suggest that the neural systems recruited for memory maintenance may vary not only on the basis of the sensory modality of stimuli, but also on the complexity of stimuli, and on the task-relevant aspects of stimuli. This differs from both the emergent-property model (Postle, 2006) and the traditional multiple-components model (Baddeley, 2010). For example, the emergent-property model is explicitly based on the notion that PFC does not serve a mnemonic function but, rather, is involved in WM/STM through executive and attentional functions. However, this is inconsistent with the growing body of recent experimental evidence that demonstrates WM/STM storage in PFC, and sometimes only in PFC (e.g., Freedman et al., 2001; Sakurai et al., 2004; Spitzer & Blankenburg, 2011, 2012; Spitzer et al., 2010). In contrast, the multiple-components model suggests that WM/STM relies on dedicated WM systems, but the neuroscience literature strongly argues against this view.

The neuroscientific research we have described in the present article provides strong support for recent psychological theory on WM/STM. Recent psychological approaches to WM/STM have suggested domain-specific storage and rehearsal processes and domain-general executive and attentional processes (Conway et al., 2005; Engle et al., 1999). The notion of domain-specific storage and rehearsal processes makes perfect sense, in light of the role often played by sensory (or other task-relevant) cortex in WM/STM storage. Furthermore, the reliance of both memory and sensory/cognitive processing on the same cortical systems provides a neural basis for the inverse relationship between memory storage and concurrent processing (Barrouillet, Portrat, & Camos, 2011). At the same time, recent research findings (e.g., Christophel et al., 2012) have blurred the line between memory storage processes and attentional/executive processes. As we improve our understanding of how and why stimulus information is encoded in attentional/executive systems, psychological theory will likely have to adapt.

Furthermore, the growing evidence that WM/STM relies on storage in sensory/task-relevant cortex casts serious doubt on the generality of models that suggest that WM is actually the activated contents of long-term memory (LTM; Cowan, 1999; Ruchkin, Grafman, Cameron, & Berndt, 2003). If the same sensory and cognitive systems that process stimuli are also responsible for their storage in WM/STM, what role is there for LTM? In addition, various studies (e.g., Christophel et al., 2012; Freedman et al., 2001; Spitzer & Blankenburg, 2011, 2012; Spitzer et al., 2010) have addressed WM/STM for stimulus types that we would not expect to be represented in LTM, as well as demonstrating that they are encoded in regions of PFC or sensory cortex. It is possible, of course, that LTM is involved in WM/STM for stimulus types that are represented in LTM. However, given that WM/STM can be demonstrated for many types of stimuli that would not have preexisting representations in LTM, models that equate WM with activated LTM are incompatible with a nontrivial portion of the literature. Indeed, Fuster (2003) differentiated between a dynamic basis for cognitive functions (the coordinated patterns of activity of cortical neurons) and a structural basis for cognitive functions (information stored in LTM about prior experiences with tasks, stimuli, etc.). The present discussion does not rule out a role for systems involved in LTM; if it is more efficient or simpler for WM/STM to store a “pointer” to relevant information in LTM, rather than storing the information itself, then doing so is consistent with current frameworks, such as some form of Postle’s (2006) emergent-property model. However, when dealing with novel or generic stimuli (e.g., vibrational frequency), we would not necessarily expect to find representations in LTM.

This raises the interesting possibility that some extant models of WM/STM could be considered as special cases of a more general model. Recent treatments of the multiple-components model have identified model components with task-relevant neural systems; the neural systems so identified are often the systems we might expect to be involved in a task based on the emergent-properties model (e.g., attributing the phonological output buffer to Broca’s area; Baddeley, 2010).

The present state of the theoretical literature raises some intriguing unanswered questions, and building a complete understanding of the neural systems underlying WM/STM will require the integration of information from all levels of analysis, from single-cell electrophysiology to functional neuroimaging. Recent studies (such as those of Spitzer & Blankenburg, 2012, and Spitzer et al., 2010, and computational studies such as Bancroft et al., 2013) have made significant progress in doing so, but enormous future efforts will be required. The recent application of new methods of analyzing neural data (e.g., Christophel et al., 2012) also represents a tremendous advance in our understanding of the neural correlates of WM/STM, and it seems likely that new and important findings will come out of further use of advanced analysis methods.

We do not suggest that stimulus complexity is the sole determinant of storage substrate, nor do we suggest that PFC stores only simple stimuli, and sensory/task-relevant cortex stores only complex stimuli. Indeed, there have been reports that some tasks may recruit both sensory cortex and PFC to maintain different aspects of stimuli; Raabe, Fischer, Bernhardt, and Greenlee (2013) found coding for stimulus location in both the visual pathway and the frontal eye fields during a memory task that involved retention of spatial information. Analyzing local field potentials and spiking, Siegel, Warden, and Miller (2009) found phase-dependent PFC encoding for stimuli when multiple visual stimuli were retained. In addition, parietal encoding of stimulus information (in addition to encoding in visual cortex) has been reported by Christophel et al. (2012); this possibly represents the activity of attentional systems that support maintenance in sensory cortex. However, there seems to be a fairly clear differentiation between prefrontal encoding for simple (and occasionally novel) stimuli and sensory encoding for complex stimuli for which preexisting representations can be found in sensory and task-relevant cortex.

The ability of PFC to adapt to represent novel forms of information explains why subjects are capable of WM/STM for stimulus types that are novel or have limited ecological validity (e.g., vibrotactile stimuli, artificially generated animal images). Given the increasing literature on nontraditional WM/STM tasks, a flexible, adaptive storage system will probably become a required component of cognitive models of WM/STM. A growing body of research has suggested that PFC can adapt to represent and store novel forms of information (for reviews, see Duncan, 2001; Duncan & Miller, 2002). Research into adaptive coding has demonstrated that prefrontal neurons can adapt to encode various types of information even after relatively short training periods. For example, Freedman et al. (2001) recorded from monkey PFC neurons during a delayed match-to-category task. Subjects were presented with two images, separated by a delay, that were created by morphing images of cats and dogs together. As such, each image was either more “cat-like” or more “dog-like.” Monkeys were trained to report whether the images were of the same category or different categories. After training, neurons in the lateral PFC adapted to represent the categorization of the target image during the delay period, and roughly equal numbers of neurons represented a cat-like or a dog-like categorization. Since the monkeys were naive as to images of cats and dogs, it appears unlikely that they had a preexisting code in PFC for representing cats and dogs. When the task was changed so that subjects had to fit stimuli into one of three different categories, PFC neurons adapted to encode the new categorical possibilities. In a recent study by Lee et al. (2013), subjects were presented with visual images and had to maintain the image for either a visual comparison task or a category comparison task. Using fMRI decoding methods, they identified stimulus representations in sensory cortex when subjects were performing the visual comparison task, but in PFC when subjects were performing the category comparison task. As well as being consistent with other work on PFC encoding of category information (e.g., Freedman et al., 2001), it serves as a clear demonstration that the salient (i.e., task-relevant) aspect(s) of a stimulus determines the storage system recruited.

Duncan and Miller (2002) proposed an adaptive coding model of PFC, in which prefrontal neurons are programmable or adaptable on the basis of behavioral goals and can serve mnemonic, attentional, and/or executive functions. While Duncan and Miller pointed out that this is only a framework for future research, the existence of a prefrontal system that can adapt to encode different types of information is consistent with the current experimental literature. Research into adaptive coding has answered the question of how subjects can store novel or unfamiliar forms of information for which there may not be preexisting neural representations, a property of PFC that may or may not interact with the seeming differentiation between memory storage in PFC and memory storage in task-relevant cortex.

The complexity and breadth of the literature has made it increasingly difficult to construct models of WM/STM based purely on neuroscientific or purely on behavioral data, and any new model must be able to fit both sides of the literature. It is time for a fundamental change in our understanding of WM and STM. Rather than relying on dedicated neural systems, it relies on the coordinated activity of systems that have primary purposes other than memory. Rather than being able to point to a single cortical region as the storage substrate of memory, maintenance relies on the functional integration of anatomically distinct systems, and the system(s) recruited for memory storage depend as much on the complexity of a stimulus as it does the sensory modality. Cortical systems can even rapidly adapt to represent novel stimuli.

One of the most important (even shocking) implications of the research outlined above (e.g., Riggall & Postle, 2012; Serences et al., 2009) is that even the interpretation of neuroimaging data must be reconsidered, in light of results showing memory maintenance in regions that do not display increased delay-period activity. Rather, decoding methods may be necessary to accurately interpret neuroimaging data. Indeed, various recent fMRI studies (including some of those discussed in this article) show inconsistency between activation found using standard analysis techniques and activation found using multivariate techniques. By extension, this suggests that theoretical treatments of WM/STM based on neuroimaging data should also be reconsidered. This reassessment of the literature, of course, depends on the validity of relatively novel decoding methods for fMRI and EEG/ERP data. It is possible that these methods do not fully identify stimulus representations stored in the brain and that the studies reviewed above are revealing only part of the picture. However, given the demonstrated ability of these methods and converging findings from various methodologies (fMRI, EEG/MEG, cellular recordings, etc.), we feel justified in suggesting that the WM/STM literature requires reassessment with an open mind.

Advances in empirical research are regularly revealing new and interesting aspects of working and short-term memory, and theoretical treatments of WM/STM have been slow to adapt. Concerted effort toward incorporating recent experimental results into theory is needed, and doing so offers enormous and exciting potential for our understanding of the relationship between cognitive processes and neural systems. In particular, it would be beneficial to develop or select a rigorous metric for stimulus complexity or dimensionality and formally assess the relationship between the complexity of a stimulus and the neural systems recruited for storage. One future direction may involve extending the work of Lee et al. (2013). Lee et al. found PFC storage of stimulus information when categorical information was retained and sensory encoding when visual object information was retained. Critically, when visual information was retained, subjects were asked to decide whether a visual fragment belonged to the original object. This raises an interesting question: If we varied the number of relevant visual features, would the location of storage change? In other words, if there were few relevant visual features, would we see PFC storage rather than sensory storage (as we find when all features of the object are potentially relevant)?

The advent of techniques for decoding the contents of memory using techniques such as fMRI and EEG has rendered the already complex literature on WM and STM even more convoluted. Application of those techniques, however, has helped reveal an apparent role for information complexity in determining the neural systems that are recruited for memory storage.