Objectives
The scope of this course is provide an introduction to basic concepts of information theory, with emphasis on data compression and coding, and to provide a connection between information theory and signal processing techniques investigated in previous courses of digital communication.
Course program
The concept of information and its measure. Entropy, divergence, mutual information, and their properties. Jensen's inequality. Data processing theorem. Fano's inequality. Asymptotic equipartition property and its implications. Entropy rate and Shannon-McMillan-Breiman theorem. Markov processes. Data compression principles. Source coding: instantaneous and uniquely decodable codes. Kraft's inequality. Huffman code, Shannon-Fano-Elias code, and arithmetic coding. Introduction to the channel capacity: definition, properties, and examples. Channel capacity and code rate. Jointly typical sequences. Channel coding theorem (Shannon). Joint source-channel coding theorem. Method of types, universal coding, Lempel-Ziv algorithm. Differential entropy, divergence, and mutual information for continuous random variables. The Gaussian channel the channel coding theorem with transmit power constraint. Parallel Gaussian channel channels and channels with feedback. Rate distortion theory. Rate distortion function and rate distortion theorem. Rate distortion function for binary and Gaussian variables.
