In the thesis "Structural Complexity and its Implications for Design of Cyber-Physical Systems," the author, K. Sinha, defines the $C$ complexity of a dynamical system in the following way: \begin{equation} C=C_1+C_2*C_3 \end{equation} where the term $C_1$ represents the sum of the complexity of the individual system responses, while the second term $C_2$ corresponds to the sum of the complexity of each pairwise interaction between the system responses. The third term $C_3$ refers to the energy of the graph representing the system. Since the system described by Sinha is described by a response represented by a random variable, the author proposes to calculate the first term by summing the entropy associated with each system response. The second term, $C_2$, is calculated using the mutual information. More precisely, if the mutual information is nonzero, the second term is equal to the reciprocal of the mutual information. Details of the calculations and an explanation of Sinha's proposed formula can be found here: Sinha thesis. More specifically, in Chapter 7 of the thesis on page 226. What confuses me is this: Sinha's formula adds a term corresponding to entropy ($C_1$) to a term that is the product of the reciprocal of the mutual information ($C_2$, which is the size of the reciprocal of entropy) multiplied by the normalized energy of the graph corresponding to the system ($C_3$). This seems a bit odd to me, since Shannon entropy and mutual information are different concepts. Shannon entropy measures the uncertainty or disorder in a single random variable, while mutual information measures the amount of information that two random variables have in common. Shannon entropy is usually expressed in information theory in "units" of bits or nats. Mutual information can be written as a function of Shannon entropy and consequently measured in bits or nats. How is it possible that its reciprocal, multiplied by $C_3$, has the size of bits or nats?

  • $\begingroup$ You are right to be skeptical - his approach seems a bit ad hoc. $\endgroup$
    – HohO
    Nov 8, 2023 at 13:51


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.