When to Explore: Guiding Deep Reinforcement Learning with State Counts and Value State Prediction Errors for Efficient Learning

Captari, Marius (2024) When to Explore: Guiding Deep Reinforcement Learning with State Counts and Value State Prediction Errors for Efficient Learning. Master's Thesis / Essay, Artificial Intelligence.

Preview	Text Thesis.pdf Download (4MB) | Preview
	Text toestemming.pdf Restricted to Registered users only Download (133kB)

Abstract

Despite the considerable attention given to the questions of how much and how to explore in Deep Reinforcement Learning (DRL), the investigation into when to explore remains relatively unexplored. While more sophisticated exploration strategies exhibit success in specific, often sparse reward environments, existing simpler approaches, such as e-greedy, persist in outperforming them across a broader spectrum of domains. The appeal of these simpler strategies lies in their stationarity, which supports the agent's learning stability and mitigates hyper parameter complexity. The downside is that these methods are essentially a blind switching mechanism, which completly disregard the agent's internal state. In this research we introduce a novel exploration strategy that combines the agent's value state prediction errors with counts of the current hashed state, providing a nuanced approach to guide the timing of exploration decisions. Experiments conducted on a range of Atari games reveal that the proposed strategy outperforms traditional methods, facilitating faster learning rates, even when compared to using each exploration signal in isolation. This study contributes to a more intricate understanding of exploration dynamics, underscoring the significance of temporal considerations in the exploration-exploitation dilemma within DRL.

Item Type:	Thesis (Master's Thesis / Essay)
Supervisor name:	Sabatelli, M.
Degree programme:	Artificial Intelligence
Thesis type:	Master's Thesis / Essay
Language:	English
Date Deposited:	08 Jan 2024 09:19
Last Modified:	08 Jan 2024 09:19
URI:	https://fse.studenttheses.ub.rug.nl/id/eprint/31796

Actions (login required)

View Item