Guest Talk: Sebastian Kassing: On the existence of minimizers in neural network optimisation landscapes

Tuesday, May 23, 2023, 10am

Location: RWTH Aachen University, Department of Computer Science - Ahornstr. 55, building E3, room 9u10

Speaker: Sebastian Kassing

 

Abstract:

We prove existence of global minima in the loss landscape for the approximation of continuous target functions using shallow feedforward artificial neural networks with ReLU activation. This property is one of the fundamental artifacts separating ReLU from other commonly used activation functions and has important consequences for the optimization dynamics. We propose a kind of closure of the search space so that in the extended space minimizers exist. In a second step, we show under mild assumptions that the newly added functions in the extension perform worse than appropriate representable ReLU networks. This then implies that the optimal response in the extended target space is indeed the response of a ReLU network. This is joint work with Steffen Dereich (Uni Münster) and Arnulf Jentzen (Uni Münster, The Chinese University of Hong Kong Shenzhen).