Time-series predictions with Recurrent Neural Networks - Studying Recurrent Neural Networks predictions and comparing to state-of-the-art Sequential Monte Carlo methods

Abstract

Recurrent Neural Networks (RNNs) have shown great success in sequence-to-sequence processing due to its ability to retain memory while incrementally processing sequence elements. It has become a fundamental algorithm for processing text and speech, and is recently becoming more popular on time-series prediction as well. Recent blog posts showing RNN flexibility applied to time-series prediction combined with flexible and easy-to-use APIs such as Keras, are leading to a widespread use of RNN for time-series prediction.

The aim of this thesis is to study the performance of RNN to predict time-series under different scenarios of noise and stationarity. In order to do this, we will simulate time-series from a state-space model (SSM) with known noise and stationarity parameters. An SSM is a subclass of a Bayesian hierarchical model and simulates a hidden state and observation depending only on static parameters and state. This is a common setup found in the signal processing literature. The hidden state will act as an underlying signal while the observation is a mixture of signal and noise. RNN models will try to predict the signal based on the simulated observed values. We are going to experiment with a long short-term memory (LSTM) model, which is the most popular type of RNNs currently used.

We evaluate the quality of the LSTM predictions by computing the cumulative mean square error (CMSE) with respect to the true simulated signal. In addition, we will estimate the signal using state-of-the-art Sequential Monte Carlo methods (SMC) such as the Bootstrap filter and Particle Markov Chain Monte Carlo (PMCMC). We then also compare the performance of the LSTM models with respect to the estimated signals. The objective is to get insights into the LSTM performance level when compared to custom-tailored state-space models.

The general findings are that the LSTM models perform worse than custom-tailored SSM models in non-stationary scenarios. PMCMC outperforms LSTM in all scenarios given the custom-tailored SSM parameters. Bootstrap filtering struggles more with estimating the state in non-stationary scenarios as noise increases, but still perform better than LSTM.