Complexity-calibrated Benchmarks for Machine Learning
Reveal When Next-Generation Reservoir Computer Predictions Succeed and Mislead

Sarah E. Marzen

W. M. Keck Science Department
Claremont McKenna, Scripps, and Pitzer College
925 N Mills Ave, Claremont, CA 91711 USA

Paul M. Riechers

School of Physical and Mathematical Sciences
Nanyang Technological University, 637371 Singapore, Singapore and

James P. Crutchfield

Complexity Sciences Center
Physics Department
University of California at Davis
Davis, CA 95616

ABSTRACT: Recurrent neural networks are used to forecast time series in finance, climate, language, and from many other domains. Reservoir computers are a particularly easily trainable form of recurrent neural network. Recently, a “next-generation” reservoir computer was introduced in which the memory trace involves only a finite number of previous symbols. We explore the inherent limitations of finite-past memory traces in this intriguing proposal. A lower bound from Fano's inequality shows that, on highly non-Markovian processes generated by large probabilistic state machines, next-generation reservoir computers with reasonably long memory traces have an error probability that is at least ~ 60% higher than the minimal attainable error probability in predicting the next observation. More generally, it appears that popular recurrent neural networks fall far short of optimally predicting such complex processes. These results highlight the need for a new generation of optimized recurrent neural network architectures. Alongside this finding, we present concentration- of-measure results for randomly-generated but complex processes. One conclusion is that large probabilistic state machines—specifically, large ε-machines&mdsah;are key to generating challenging and structurally-unbiased stimuli for ground-truthing recurrent neural network architectures.

Sarah E. Marzen, Paul M. Riechers, and James P. Crutchfield, “Complexity-calibrated Benchmarks for Machine Learning Reveal When Next-Generation Reservoir Computer Predictions Succeed and Misleads”, Scientific Reports 14 (2024) 8727.