paraafPeter A. van der Helm Demo link
About Teaching Research Publications Presentations


Structural information theory and its applications



Theory

In the 1960s, Emanuel "Maan" Leeuwenberg initiated structural information theory (SIT). SIT began as a quantitative coding model of visual pattern classification (demo) which, in interaction with empirical research, developed into a competitive theory of perceptual organization. Nowadays, SIT includes empirically-successful quantitative models of amodal completion (demo) and symmetry perception (demo). Furthermore, in object perception, SIT proposes an integration of viewpoint-independent and viewpoint-dependent factors quantified in terms of descriptive complexities (demo).

Central to SIT is the simplicity principle, which implies that the visual system is assumed to prefer the simplest interpretation among all possible interpretations of a stimulus. To make predictions, the interpretations are represented by symbol strings, and the symbol string with the overall simplest code is taken to specify the preferred interpretation. A simplest code is a symbol representation that enables the reconstruction of the stimulus using a minimum number of descriptive parameters; it is obtained by capturing a maximum amount of regularity, and it implies a hierarchical stimulus organization in terms of wholes and parts. These wholes and parts then are predicted to be the perceived objects.
Emanuel Leeuwenberg
Emanuel Leeuwenberg

See Acta Psychologica 2003 for a special issue in his honor, on the occasion of his 65th birthday.

Historically, SIT's simplicity principle is a descendant of Hochberg and McAlister's (1953) minimum principle, and both are information-theoretic translations of the law of Prägnanz. This Gestalt law was proposed, in the early 20th century, by Wertheimer (1912, 1923), Köhler (1920), and Koffka (1935), and it refers to the natural tendency of physical systems to settle into relatively stable minimum-energy states to which, in the case of the human visual system in the brain, preferred interpretations might correspond. Hochberg and McAlister casted this Gestalt idea in terms of representational compactness by replacing minimum energy with minimum descriptive information. SIT added a concrete coding language specifying the so-called transparent holographic regularities to be captured in descriptive codes. For the mathematical foundation of SIT's coding language, see Journal of Mathematical Psychology 1991, and for the psychological foundation of the transparent holographic nature of visual regularity, see Psychological Review 1996 and Psychological Review 2004.

SIT's simplicity principle concurs with the minimum description length principle in the mathematical domain of algorithmic information theory (AIT) that, also in the 1960s, was initiated by Kolmogorov and Solomonoff. Until the 1990s, SIT and AIT evolved independently, and indeed, there are differences between SIT and AIT:
For the rest, however, SIT and AIT share many starting points and objectives. SIT's and AIT's modern information-theoretic approaches can be said to present viable alternatives to Shannon's (1948) classical (or selective) information-theoretic approach and to the classical Helmholtzian likelihood principle which, in vision, assumes that the preferred interpretation of a proximal stimulus is the one most likely to be true in the world, that is, the one with the highest probability of specifying the actual distal stimulus.

Just as Shannon's approach, the Helmholtzian likelihood principle presupposes knowledge about probabilities in terms of, for instance, frequencies of occurrence of things in the world. Such probabilities, however, are often hardly quantifiable, if at all (demo). In SIT and AIT, this problem is circumvented by turning to precisals, that is, artificial probabilities derived from the length of shortest descriptive codes. AIT has shown that precisals might well be reliable alternatives for the often unknown real probabilities. In vision, this paved the way for a more detailed comparison of SIT's simplicity principle and Bayesian implementations of the likelihood principle (demo).

This comparison revealed that precisals and real probabilities may be far apart for viewpoint-independent factors (Bayesian priors) but seem close for viewpoint-dependent factors (Bayesian conditionals) which are decisive in the everyday perception by a moving observer. This implies that both the simplicity principle and the likelihood principle may have guided the evolution of the human visual system, the difference being that the likelihood principle assumes that the human visual system is a special-purpose system in that it is highly adapted to one specific world, whereas the simplicity principle assumes it is a general-purpose system in that, as an emergent side-effect of the preference for simplest interpretations, it is fairly adaptive to many different worlds. For an extensive discussion on these issues, see Psychological Bulletin 2000, and for an updated brief discussion, see Acta Psychologica 2011.

In Marr's (1982) terms, SIT began as a theory at the computational level of description (demo). Just as Bayesian implementations of the likelihood principle, for instance, SIT models vision as if it, somehow, considers all possible interpretations before it selects a preferred interpretation. In both cases, this implies a modeling (of the nature) of process outcomes rather than of process mechanisms. Nowadays, however, SIT also includes process models. For instance, see Psychological Review 1999 for the so-called holographic bootstrapping mechanism in the detection of visual regularity (demo), and see Proceedings of the National Academy of Sciences USA 2004 for the so-called transparallel processing mechanism in the selection of simplest interpretations.

This transparallel processing mechanism relies on so-called hyperstrings (demo), which are distributed representations that enable the processing of an exponential number of codes as if only one code were concerned (demo). This goes beyond the parallel distributed processing (PDP) mechanism in "neural" network models as considered in connectionism (demo), and has an additional storage advantage: PDP models presuppose that all possible outcomes for all possible inputs are represented beforehand in a network, whereas SIT's transparallel processing mechanism performs an on-the-fly construction of all possible outcomes for only the input at hand (demo).

In cognitive (neuro)science, the latter suggests that the relatively rigid neural network in the brain allows for flexible cognitive networks which change with changing input. This agrees with the finding, in neuroscience, of transient neural assemblies which seem to be involved in binding similar features in the input, and which signal their presence by synchronous firing of the neurons involved (a topic which is also studied in dynamic-systems approaches to cognition). This intriguing connection between transparallel processing (a form of cognitive processing) and neuronal synchronization (a form of neural processing) has been explored in Cognitive Processing 2012, leading to a concrete picture of flexible cognitive architecture (constituted by hyperstring-like "gnosons" or "fundamental particles of cognition") in between the relatively rigid level of neurons and the still elusive level of consciousness (demo).

For a book on these ideas, see Simplicity in Vision
For a pdf-presentation on these ideas, see Visual regularity or its printable handout
For more details on methodological principles guiding these ideas, see Marr's levels, Research cycles, and Metaphors of cognition



Applications

The conglomerate of ideas within SIT has found societal applications in art science and visual ergonomics. This led, for instance, to traffic reconstructions yielding safer roads, bridges, and tunnels (demo). Furthermore, during the past decades, it has been applied to a wide range of topics in vision science. See the book Structural Information Theory for details about these topics which include: