Beyond simple model-free reinforcement learning in human decision making

Abstract

Over the last two decades, there has been a large scale effort in

cognitive neuroscience to understand learning and decision making from

the perspective of simple model-free reinforcement learning

algorithms. This interest was invigorated in the mid 1990's, when it

was realized that the phasic activity of midbrain dopaminergic neurons

resembles reward prediction errors. The algorithms studied formalize

the notion of learning from past experiences through trial and

error. Although important, there are many aspects of behavior they

cannot explain. More recent work has begun to fill in some of these

gaps by borrowing yet additional ideas from computational

reinforcement learning. One line of inquiry has concentrated on

aligning goal-directed behavior, which resembles the common sense

notion of &ldquo;planning&rdquo;, with <italic>model-based</italic> reinforcement

learning. This work has aimed to understand how the brain is able to

learn the world model prescribed by the model-based framework, and to

characterize the neural correlates of the value functions it

predicts. This thesis adds to this work by offering two separate, but

related, algorithmic accounts of how the brain may be able to actually

map the world model into a decision. Existing data are examined and

new experiments are performed. A second line of inquiry has

concentrated on understanding behavior from the perspective of

<italic>hierarchical</italic> reinforcement learning. The thesis makes two

contributions to this area as well. First, it is shown that the brain

codes pseudo-reward prediction errors, a prediction error in response

to a faux reward signal that is used to train skills that are not in

themselves useful, but that may be used to achieve other means.

Second, an optimality framework is provided for understanding which

skills are most beneficial to have when confronted with an ensemble of

tasks.