Skip to main content

Subsection 0.6.3 Entropy in Information Theory

Definition 0.6.4.

The Shannon entropy of a discrete random variable with possible outcomes \(i\in\{1,2,\ldots,n\}\) where \(n\in\mathbb{N}\) and corresponding probabilities \(\mathbb{p}_i\) is defined as:
\begin{equation*} H_\text{Shannon} := -\sum_{i=1}^n{p_i \log_2{p_i}} . \end{equation*}