How do we create and manipulate internal models of our visual surroundings? Whereas conventional approaches broach this topic by characterizing storage processes in visual working memory, I place special emphasis on the “working” component of the system: active manipulation. By delineating the neurocognitive architecture of this system, I aim to identify the locus of deficits limiting storage and manipulation capacities, and develop treatments to augment cognition in healthy, aging, and vulnerable populations. To this end, I adopt an interdisciplinary and collaborative “Mind-Brain–Machine” approach:
Mind (Methods: psychophysics, development, and genetics): I have constructed a developmental model of factors shaping individual storage and manipulation abilities, and demonstrated that working memory operates over two separate –and competing– representations that may differ in their formats of representation (Pailian & Alvarez, 2018).
Brain (Methods: EEG, neurostimulation): I have identified a novel neurophysiological marker of mental manipulation that is distinct from that for storage (Pailian et al, 2017), and separately enhanced these functions using neurostimulation (Pailian et al, under review). These findings demonstrate that storage and manipulation are governed by separate neural mechanisms and resources, and present non-invasive brain stimulation as an effective tool for cognitive enhancement (Pailian et al., 2019).
Machine (Methods: convolutional neural networks, Bayesian modeling): I have demonstrated that visual working memory operates over hierarchical levels of representation (Bill et al, under review), and use neural nets to explore how map-like structures in cortical space may provide both the flexibility and constraints required for storage and manipulation (currently ongoing).
I. Augmenting Working Memory for Scientific and Clinical Gain
Augmenting Encoding/Learning: Using trans-cranial random noise stimulation (tRNS), I doubled participants’ performance in a memory task infused with statistical regularities (i.e. certain colors co-occurring with higher probability). Given that performance reached baseline levels when these regularities were removed, tRNS selectively improved participants’ abilities to learn these regularities and create efficient/compressed memory representations. I replicate these findings and demonstrate that this neuroenhancement transfers to a separate statistical learning task.
Augmenting Storage: By applying 20 minutes of anodal trans-cranial direct current stimulation (tDCS) to the right-posterior parietal cortex, I enhanced visual storage capacity by up to 26%, (Pailian et al., 2019). Performance exceeded the hallmark 4-item storage limit, suggesting that behavioral working memory limits may not reflect a fixed upper bound.
Augmenting Manipulation: I demonstrated an intriguing double dissociation. Anodal tDCS applied to the right posterior parietal cortex improved storage – but not manipulation – ability. However, applying the same protocol while targeting the right dorsolateral prefrontal cortex yielded the opposite effect: an enhancement of manipulation ability by up to 23%, with no effect on storage. This provides causal evidence that storage and manipulation capacities are determined by separate neural resources (Pailian et al., under review).
I use various forms of neurostimulation to identify brain circuitry supporting storage and manipulation, and to enhance cognition in healthy and vulnerable populations across various stages of visual information processing.
Moving Forward: I will leverage artificial neural networks and real-time EEG recordings to develop personalized neurostimulation methods (electrode montages, dosage parameters, stimulation duration) to maximally augment cognition in healthy, aging, and clinical populations (e.g. “chemo-brain” cancer patients, Alzheimer’s patients, ADHD individuals, etc.). Given the working memory system’s connections to broader cognition, such an individualized dual neurostimulation-behavioral training protocol has potential for long-range transfer to a variety of cognitive domains (e.g. verbal comprehension, STEM aptitude, decision-making, inhibition, spatial navigation, etc.).
II. "Building" the Working Memory Machine
Memory Representations are Hierarchical in Format: Using a Bayesian observer model, I demonstrate that humans actively exploit motion structure to facilitate performed in an an object-location updating paradigm. Moreover, I demonstrate that individuals are able to employ the correct or close-to-correct motion structure in a similar location prediction task, even for multiply-nested (deep) hierarchies. These results suggest that humans hierarchically represent information when updating identity-location bindings, and that they flexibly employ near-optimal priors when doing so (Bill et al., under review).
Representational Geometry of Convolutional Neural Networks can serve to Probe the Architecture of Working Memory: Given that the architecture of convolutional neural networks resembles hierarchical visual information processing in humans, I used a neural net encoding model (developed by Bashivan et al., 2019 by mapping AlexNet conv-3 activations to macaque V4 cortex) to synthesize images that maximally drive neural activity for mid-level object representations. I explored how the representational geometry of this encoding model can serve as a biologically plausible model for information represented in working memory. To this end, I computed a representational dissimilarity matrix of euclidean distances of neural site predictions for all synthetic image pairs. I subsequently presented participants with a working memory storage task, in which memoranda consisted of synthesized images that were close vs. far in this neural net representational space. In so doing, I found that storage performance was highly dependent on this representational structure: images close in this space interfered destructively (currently ongoing).
What structure, constraints, and representational formats are required to build a limited capacity processor that can flexibly store and manipulate information? Using Bayesian modeling and convolutional neural networks, I leverage artificial intelligence to gain insights into human visual working memory - arguably beyond that which can be learned from experimental testing alone.
Neural Net Similarity Structure of all Synthesized Images
(represented in multidimensional space)
Moving Forward: This neural-net guided approach may prove instrumental towards generating biologically plausible hypothesis of working memory architecture and its underlying constraints. Namely, I will use this model to investigate 1) how map like structures across various levels of cortex can support a variety of cognitive operations, and 2) how excitatory vs. inhibitory activations of neural populations can give rise to cognitive constraints. Neural networks provides a non-intrusive method for such investigations.
I. Origins and Malleability of Storage Limits
III. Architecture of Mental Manipulation
Mental manipulation enables combinatorial thought, by allowing for working memory representations to be modified by input from perception (updating) or from long-term memory (mental simulation). I identified a behavioral signature of manipulation limits: whereas 2 items can be manipulated ad infinitum with relatively no cost, manipulating larger set sizes leads to systematic errors that increase as a function of the number of manipulations performed (Pailian & Halberda, 2013). How and why is the system structured in a way that makes manipulation so difficult?
Storage and Manipulation May Rely on Separate Neural Resources: I discovered a novel electroencephalographic marker of mental manipulation: activity at frontal-central electrode sites become increasingly more negative during manipulation – but not during storage. Complementarily, the established marker of visual storage (contralateral delay activity at posterior electrodes) is unaffected by manipulation (Pailian, Störmer, & Alvarez, 2016).
Memory for Initial State is Preserved: Manipulating information does not affect memory for the initial (unmanipulated) state of that item (Pailian & Halberda, 2015b). Ongoing work suggests evidence for separate representations with distinct formats of representation.
Manipulation Failures Stem from Misbinding Errors: Computationally modeling response errors suggest that manipulation failures result from misbinding features of objects that are being manipulated. The frequency of misbinding errors increases as a function of manipulation load and interference from a lingering trace representation (Pailian & Alvarez, 2018).
Moving Forward: These findings suggest that manipulation ability is inherently constrained by the architecture of the working memory system, as it requires two separate representations that are costly to produce and interfere with each other. How do these representations differ in the ways with which they interface with broader cognition and in their formats of representation? How can such an "interference account" explain limits in non-visual manipulations and mental simulation?
IV. Origins and Malleability of Storage Limits
Storage limits in working memory render the system an information processing bottleneck, restricting input to broader cognition. My work focuses on identifying the origins of these limits and constructing a developmental model of factors determining individual differences in storage capacity. To this end, I have developed a response-time based method that separately quantifies estimates of storage capacity and executive control (Pailian & Halberda, 2015a). This paradigm reveals three important findings:
Storage Capacity is Highly Heritable (Pailian et al., in prep; Wilmer, Pailian, et al., 2017).
I tested pairs of monozygotic and dizygotic twins, and found that estimates of storage capacity (but not executive control) are more highly correlated in the monozygotic twins, suggesting a genetic predetermination to storage limits.
Individual Differences in these Upper Bounds are not Fixed from Birth. Rather visual working memory storage abilities increase across the ages of 3- to 8-years (independent of changes in executive control), after which they reach adult-like levels (Pailian et al., 2016).
Storage Capacity is Fundamentally Limited. The evolutionary roots of these limits, which I investigated by measuring storage abilities in African Gray Parrots, show capacity limits similar to human adults, suggesting homologous mechanisms (Pepperberg & Pailian, 2017).
Moving Forward: These findings suggest that individual differences in storage emerge from an interaction of genes plus some "X-factor" that is not accounted for by genetics, develops across the lifespan, and does not rely on a mechanism unique to humans. What exactly is this "X-factor"?