6:  Basic elements of information theory



- Entropy

  o  Intuition 

  o  Formal definition

  o  Digression: properties of the logarithm

  o  More intuitions: entropy and data compression


- Other issues

  o  Entropy and predictability (conditional entropy)

  o  Entropy and mutual information

  o  Relative entropy and cross-entropy

  o  Entropy and perplexity