Confidence as an Arbiter of Attentional Allocation during Learning in Multidimensional Environments

Abstract

Reinforcement learning algorithms are notoriously inefficient in high-dimensional environments, and yet people manage to solve these complex problems with ease. One way in which our brains are thought to make such high-dimensional problems tractable is by using selective attention to reduce the dimensionality to only those features that are relevant for the task. Prior work has demonstrated that in reward-based learning, there is a bi-directional relationship between learning and attention, but how the brain decides where to employ attention over the course of learning is debated. Drawing on ideas from theoretical and experimental work, we propose that internal confidence computations may arbitrate between different attention strategies. To test this hypothesis, we used a high-dimensional reinforcement learning task in which efficient learning and concomitant maximization of reward requires narrowing of subjects' attention to the relevant dimension. Our results demonstrate a clear link between confidence and the breadth of attention during learning. We also incorporated our hypotheses into two novel computational models that predict trial-by-trial attention during the task. While neither of our models performed better than the value-based model to which we compared them, our results do not disprove the idea that confidence modulates the distribution of attention during learning. Rather, they suggest that models which allocate attention purely based on value miss an important component of how subjects actually distribute their attention, and that more thought needs to be given to the role of confidence, as well as perseverance and hypothesis-testing.