A Comparative Study of Sparsity Promoting Techniques in Neural Network for Modeling Non-Linear Dynamics

Abstract

Sparsity-promoting techniques show promising results in improving the generalization of neural networks. However, the literature contains limited information on how different sparsity techniques affect generalization when using neural networks to model non-linear dynamical systems. This study examines the use of sparsity-enhancing techniques to improve accuracy and reduce the divergence rate of neural networks used to simulate such systems. A range of sparsity methods, including hard and soft thresholding, pruning and regrowing, and L1-regularization, were applied to neural networks and evaluated in a complex nonlinear aluminum extraction process by electrolysis. The results showed that the most effective technique was L1 regularization, which enhanced the important connections in the network and improved the model performance. In contrast, many of the more advanced sparsity techniques resulted in significantly worse performance and higher divergence rates. Additionally, the application of Stochastic Weight Averaging during training increased performance and reduced the number of diverging simulations. These findings suggest that carefully selecting the right sparsity techniques and model structures can improve the performance of neural network-based simulations of dynamical systems.