ABSTRACT: How much information do natural systems store and process? In this work we attempt to answer this question in multiple ways. We first establish a mathematical framework where natural systems are represented by a canonical form of edge-labeled hidden Markov models called ε-machines. Then, utilizing this framework, a variety of measures are defined and algorithms for computing them from an ε-machine are described.
The first two measures defined are related to the length of time a system remembers. The first, the Markov order, is a well-known measure of the time one must observe a system in order to make accu- rate predictions. Despite its statistical nature, it is shown to be a topological property of the process's ε-machine. The second, the recently defined cryptic order, quantifies the ability to retrodict a system's internal dynamics. It is also shown to be a topological property of the ε-machine, and efficient algorithms for computing both quantities are given.
The second batch of metrics quantify information generation and storage in a system by partitioning the observations. By considering the role of both the past and the future behavior of a system, a semantic understanding of information generation emerges, labeling some information generation as ephemeral, having no lasting effects on the system, and the rest as bound, playing a role temporal structure. Following through with this decomposition, other quantities of less straight-forward interpretation are also defined. This is followed by a thorough discussion of these quantities and other derived quantities.
Lastly the decomposition of the entropy rate into ephemeral and bound components is applied to several standard chaotic systems through a duality between the entropy rate and the Lyapunov exponent. This exposes new structural behaviors hitherto unknown in these systems. These revolutions hint at a method for tuning natural or engineered systems so as to maximize the ability to harness their intrinsic computing abilities.