Evolutionary Optimization of activation functions in deep heterogeneous networks

Schweickert, Gregory (2022) Evolutionary Optimization of activation functions in deep heterogeneous networks. Bachelor's Thesis, Artificial Intelligence.

Preview	Text bAI_2022_SchweickertG.pdf.pdf Download (815kB) | Preview
	Text toestemming.pdf Restricted to Registered users only Download (134kB)

Abstract

The effective use of deep neural networks often requires time consuming and expertise-reliant manual design decisions. Existing meta-learning methods have come a long way toward alleviating these requirements and enabling end-to-end learning. However, one important yet often overlooked facet is the selection of activation functions. Although methods exist to optimize this process, the vast majority are only concerned with homogeneous networks. That is, optimizing a single activation function for all hidden units in a network. Conversely, heterogeneous networks employ a variety of activation functions throughout. In this work, an evolutionary search method for discovering well-performing homogeneous and heterogeneous activation function setups is presented. Using random search for baseline comparison, experiments show that the developed directed search method is well-suited for the task. Indeed, hand-engineered deep CNNs tested on CIFAR-10 using ReLU and Swish are outperformed by those using discovered solutions. Furthermore, the explored heterogeneous setups result in better performance than their homogeneous counterparts. Lastly, novel solutions of both types are shown to generalize to CIFAR-100. However, transfer to an up-scaled architecture is relatively less successful. The presented methods offer a promising new approach to meta-learning in the space of deep heterogeneous neural networks.

Item Type:	Thesis (Bachelor's Thesis)
Supervisor name:	Abreu, S. and Jaeger, H.
Degree programme:	Artificial Intelligence
Thesis type:	Bachelor's Thesis
Language:	English
Date Deposited:	15 Mar 2022 15:10
Last Modified:	15 Mar 2022 15:10
URI:	https://fse.studenttheses.ub.rug.nl/id/eprint/26700

Actions (login required)

View Item