On Lattices and the Dualities of Information Measures

Abstract

Measures of dependence among variables, and measures of information content and shared information have become valuable tools of multi-variable data analysis. Information measures, like marginal entropies, mutual and multi-information, have a number of significant advantages over more standard statistical methods, like their reduced sensitivity to sampling limitations than statistical estimates of probability densities. There are also interesting applications of these measures to the theory of complexity and to statistical mechanics. Their mathematical properties and relationships are therefore of interest at several levels. Of the interesting relationships between common information measures, perhaps none are more intriguing and as elegant as the duality relationships based on Mobius inversions. These inversions are directly related to the lattices (posets) that describe these sets of variables and their multi-variable measures. In this paper we describe extensions of the duality previously noted by Bell to a range of measures, and show how the structure of the lattice determines fundamental relationships of these functions. Our major result is a set of interlinked duality relations among marginal entropies, interaction information, and conditional interaction information. The implications of these results include a flexible range of alternative formulations of information-based measures, and a new set of sum rules that arise from path-independent sums on the lattice. Our motivation is to advance the fundamental integration of this set of ideas and relations, and to show explicitly the ways in which all these measures are interrelated through lattice properties. These ideas can be useful in constructing theories of complexity, descriptions of large scale stochastic processes and systems, and in devising algorithms and approximations for computations in multi-variable data analysis.