Eilers, Pieter Jan (2021) Analysis of Feedback Alignment. Master's Thesis / Essay, Computing Science.
|
Text
WMCS901-30_2021_EilersPJ.pdf Download (1MB) | Preview |
|
![[img]](https://fse.studenttheses.ub.rug.nl/style/images/fileicons/text.png) |
Text
toestemming.pdf Restricted to Registered users only Download (128kB) |
Abstract
In the context of training neural networks, backpropagation of error with gradient descent is the most widely used method. With the backpropagation algorithm, the forward weights are reused in the feedback pass through the network. This process is not biologically plausible, as it requires neurons in hidden layers to know the synaptic weights of neurons in different layers. Recently, a new method has been suggested which shows that the feedback weights do not have to be identical and symmetrical to the forward weights. The weights used in the backward pass can be replaced by random feedback weights. The network will learn how to use these feedback weights effectively, essentially \textit{learning how to learn}. In this thesis, we use on-line learning in student-teacher scenarios to compare the effectiveness of feedback alignment with the most commonly used backpropagation. We simulate several realizable, overrealizable and unrealizable scenarios for both shallow and deep networks. These networks use either the sigmoidal erf activation function or ReLU activation in the hidden neurons. Experiments show that feedback alignment can perform at least as efficiently and accurately as backpropagation in many scenarios. In simulations of deep over-parameterized student networks with both erf and ReLU activation, feedback alignment seems to have a systematic advantage in terms of earlier escape from learning plateau states where loss slows down significantly.
| Item Type: | Thesis (Master's Thesis / Essay) |
|---|---|
| Supervisor name: | Biehl, M. and Straat, M.J.C. and Bunte, K. |
| Degree programme: | Computing Science |
| Thesis type: | Master's Thesis / Essay |
| Language: | English |
| Date Deposited: | 10 Sep 2021 09:30 |
| Last Modified: | 10 Sep 2021 09:30 |
| URI: | https://fse.studenttheses.ub.rug.nl/id/eprint/26074 |
Actions (login required)
 |
View Item |