Directed Information Flow and Causality in Neural Systems

Definition

In the human experience, information typically flows from one place to another. By contrast, the notion of mutual information introduced by Shannon (1948) is perfectly symmetric in its arguments and does not distinguish between “input” and “output.” In this sense, it is perhaps surprising that this very measure of information indeed captures the capacity of any communication channel – though we should recall that the proof of this fundamental fact is not merely a simple consequence of Shannon’s definition.

In spite of Shannon’s strong and fundamental results, it has been tempting to define a notion of directed information. This was first proposed in (Marko 1973) for stationary processes. The more general and useful definition was given in the brief and beautiful note by Massey (1990). Moreover, Massey (1990), Kramer (1998) and subsequent work revealed that directed information has a natural place in the study of information transmission with feedback from the output to the...