Oostwal, Elisa (2020) Phase Transitions in Layered Neural Networks: The Role of The Activation Function. Master's Thesis / Essay, Computing Science.
|
Text
Phase_Transitions_in_Layered_Neural_Networks_The_Role_of_The_Activation_Function.pdf Download (3MB) | Preview |
|
![[img]](https://fse.studenttheses.ub.rug.nl/style/images/fileicons/text.png) |
Text
toestemming.pdf Restricted to Registered users only Download (127kB) |
Abstract
One of the improvements in the model of artificial neural networks is the use of other activation functions than the conventionally used sigmoidal function. Many alternative activation functions have been proposed which are all claimed to have a superior performance. While an extensive comparison of activation functions based on their performance has been made, a theoretical foundation that explains the observed differences is lacking. In this thesis we investigate which characteristics of the activation functions determine the type of phase transition. For this, we borrow concepts from statistical physics to research the learning behaviour of artificial neural networks in the context of off-line learning. Five activation functions are studied: sigmoidal, Rectified Linear Unit, Leaky Rectified Linear Unit, Piecewise Linear Unit, and a novel activation function, dubbed Rectified Piecewise Linear Unit. Our research shows that sigmoidal and PLU activation both cause a discontinuous phase transition in networks with more than three hidden units, whereas ReLU and LReLU activation induce a continuous phase transition. RePLU causes a discontinuous phase transition for a particular range of its slope, but provokes a continuous phase transition when its slope exceeds this upper limit. We hypothesize that a continuous phase transition is established when the response of the activation function is linear and the slope on the negative domain differs from the slope on the positive domain.
| Item Type: | Thesis (Master's Thesis / Essay) |
|---|---|
| Supervisor name: | Biehl, M. and Bunte, K. and Straat, M.J.C. |
| Degree programme: | Computing Science |
| Thesis type: | Master's Thesis / Essay |
| Language: | English |
| Date Deposited: | 09 Dec 2020 13:22 |
| Last Modified: | 09 Dec 2020 13:22 |
| URI: | https://fse.studenttheses.ub.rug.nl/id/eprint/23691 |
Actions (login required)
 |
View Item |