Whilst studying Shannon entropy, I came up with a singular idea, which surely someone already analysed, about which I cannot find anything though. I'm going to explain myself better.
Let's suppose we deal with a string of $N$ characters, written with an alphabet of $k$ letters $\{a_1, a_2, \ldots, a_k\}$, each one with a probability $p(a_i)$ to appear. Of course we need
$$\sum_{k = 1}^{k}\ p(a_i) = 1$$
Then here is a simple application: suppose we have a binary alphabet in which "$1$" has a probability of $p$ to appear, and "$0$" has a probability $1-p$ to appear. Given a message of $N$ characters, for large $N$,it'll have $N(1-p)$ characters "$0$" and $Np$ characters "$1$".
Now, the number of the total possible messages is given by
$$\binom{N}{Np}$$
And taking the $\log$ (in base two in this case)m using Stirling we get
$$\log\binom{N}{Np} \approx n\ H(p)$$
Where
$$H(p) = -(p\log_2 p + (1-p)\log_2 p)$$
Now it says: if we deal with $k$ letters, then what we get is the well known Shannon Entropy
$$H(X) = -\sum_{x = 1}^{k}\ p(x)\log p(x)$$
associated to the distribution $X = \{x, p(x)\}$.
Question
Is it possible to apply this, somehow, in order to find the entropy of a conversation? Or the entropy of a single phrase like ?
$$\text{THE PEN IS ON THE TABLE}$$
In this case we would deal with an alphabet of $26$ letters, but how to deal with more than one word? (Namely more than one string, if I understood well. Unless it use string for a complete phrase...)