Meta-Reinforcement Learning with Episodic Recall: An Integrative Theory of Reward-Driven Learning

Abstract

Research on reward-driven learning has produced and substantiated theories of model-free and model-based reinforcement learning (RL), which respectively explain how humans and animals learn reflexive habits and build prospective plans. A highly developed line of work has unearthed the role of striatal dopamine in model-free learning, while the prefrontal cortex (PFC) appears to critically subserve model-based learning. The recent theory of meta-reinforcement learning (meta-RL) explained a wide array of findings by positing that the model-free dopaminergic reward prediction error trains the recurrent prefrontal network to execute arbitrary RL algorithms -- including model-based RL -- in its activations.

In parallel, a nascent understanding of a third reinforcement learning system is emerging: a non-parametric system that stores memory traces of individual experiences rather than aggregate statistics. Research on such \textit{episodic learning} has revealed its unmistakeable traces in human behavior, developed theory to articulate algorithms underlying that behavior, and pursued the contention that the hippocampus is centrally involved. These developments lead to a set of open questions about (1) how the neural mechanisms of episodic learning relate to those underlying incremental model-free and model-based learning and (2) how the brain arbitrates among the contributions of this abundance of valuation strategies.

This thesis extends meta-RL to provide an account for episodic learning, incremental learning, and the coordination between them. In this theory of episodic meta-RL (EMRL), episodic memory reinstates activations in the prefrontal network based on contextual similarity, after passing them through a learned gating mechanism (Chapters 1 and 2). In simulation, EMRL can solve episodic contextual water maze navigation problems and episodic contextual bandit problems, including those with Omniglot class contexts and others with compositional structure (Chapter 3). Further, EMRL reproduces episodic model-based RL and its coordination with incremental model-based RL on the episodic two-step task (Vikbladh et al., 2017; Chapter 4). Chapter 5 discusses more biologically detailed extensions to EMRL, and Chapter 6 analyzes EMRL with respect to a set of recent empirical findings. Chapter 7 discusses EMRL in the context of various topics in neuroscience.