On Explaining the Surprising Success of Reservoir Computing Forecaster of Chaos? The Universal Machine Learning Dynamical System with Contrasts to VAR and DMD

Abstract

Machine learning has become a widely popular and successful paradigm, including in data-driven science and engineering. A major application problem is data-driven forecasting of future states from a complex dynamical. Artificial neural networks (ANN) have evolved as a clear leader amongst many machine learning approaches, and recurrent neural networks (RNN) are considered to be especially well suited for forecasting dynamical systems. In this setting, the echo state networks (ESN) or reservoir computer (RC) have emerged for their simplicity and computational complexity advantages. Instead of a fully trained network, an RC trains only read-out weights by a simple, efficient least squares method. What is perhaps quite surprising is that nonetheless an RC succeeds to make high quality forecasts, competitively with more intensively trained methods, even if not the leader. There remains an unanswered question as to why and how an RC works at all, despite randomly selected weights. We explicitly connect the RC with linear activation and linear read-out to well developed time-series literature on vector autoregressive averages (VAR) that includes theorems on representability through the WOLD theorem, which already perform reasonably for short term forecasts. In the case of a linear activation and now popular quadratic read-out RC, we explicitly connect to a nonlinear VAR (NVAR), which performs quite well. Further, we associate this paradigm to the now widely popular dynamic mode decomposition (DMD), and thus these three are in a sense different faces of the same thing. We illustrate our observations in terms of popular benchmark examples including Mackey-Glass differential delay equations and the Lorenz63 system.