The Primacy Bias Through The Lens of the Fisher Information Matrix

Falzari, Massimiliano (2024) The Primacy Bias Through The Lens of the Fisher Information Matrix. Master's Thesis / Essay, Artificial Intelligence.

Preview	Text mAI2024MassimilianoFalzari.pdf Download (30MB) | Preview
	Text Toestemming.pdf Restricted to Registered users only Download (177kB)

Abstract

DRL systems exhibit a significant tendency to overfit to early experiences, a phenomenon known as the PB. This bias can severely impact learning efficiency and final performance, particularly in complex environments. This thesis presents a comprehensive investigation of the PB through the lens of the FIM and introduces FGSF, a novel method for its mitigation. We first develop a theoretical framework characterizing the PB through distinctive patterns in the FIM trace, identifying critical memorization and reorganization phases during learning. Building on this understanding, we propose FGSF, which leverages the geometric structure of the parameter space to selectively modify network weights, preventing early experiences from dominating the learning process while preserving valuable knowledge. Through extensive empirical evaluation across multiple environments from the DMC, we demonstrate that FGSF consistently outperforms baseline approaches, particularly in complex, high-dimensional tasks. Our analysis reveals several key insights: (1) the PB affects critic networks more severely than actor networks, with critic-only intervention often outperforming full network scrubbing, (2) FGSF's effectiveness scales with task complexity and replay ratio, suggesting particular utility in challenging learning scenarios, (3) the method maintains robust performance across different hyperparameter settings while introducing minimal computational overhead, and (4) even simple noise injection methods can provide meaningful improvements, indicating that the PB may be fundamentally linked to optimization dynamics. These findings not only advance our understanding of the PB but also provide practical tools for its mitigation, contributing to the development of more efficient and robust DRL systems. The geometric perspective offered by our FIM-based analysis opens new avenues for understanding and addressing learning dynamics in deep neural networks.

Item Type:	Thesis (Master's Thesis / Essay)
Supervisor name:	Sabatelli, M.
Degree programme:	Artificial Intelligence
Thesis type:	Master's Thesis / Essay
Language:	English
Date Deposited:	28 Nov 2024 09:12
Last Modified:	28 Nov 2024 09:12
URI:	https://fse.studenttheses.ub.rug.nl/id/eprint/34453

Actions (login required)

View Item