Symbolbild
Ein Symbolbild, das die AG repräsentiert
Login |
 
 

Introduction to Information Theory

Information theory is the science of operations on data such as compression, storage, and communication. This course is a basic introduction to information theory at the graduate or advanced undergraduate level. Prerequisites include a basic knowledge of elementary probability, which, however, will be reviewed in the first weeks. The first central notions covered are entropy and variable length encoding, leading to the noiseless coding theorem of Claude Shannon (1948) which relates the average codeword length for an encoding of a source to its entropy. However, when information must be transmitted over a noisy channel, one must reconsider how to encode information efficiently while minimizing the probability of uncorrected errors in the transmission. This leads to the notion of channel capacity and, finally, the famous noisy coding theorem. Measures of information and uncertainty like entropy, mutual information, and relative entropy have applications in many diverse fields, e.g., pattern recognition, computer vision, and plagiarism detection, some of which will be studied at the end of the course. References include S. Roman, Coding and Information Theory, Graduate Texts in Mathematics, Springer, 1994, and T. M. Cover and J. A. Thomas, Elements of Information Theory, Second Edition, Wiley, 2006.

A part of the book on information theory: index.php?eID=tx_nawsecuredl&u=0&g=0&t=1511718272&hash=a6bfc1b2faa2d12b55a134e336f95bca4633c946&file=fileadmin/informatik/ag-saupe/Images/Coding_and_Information_Theory.pdfPDF

A short review of basic (discrete) probability theory: index.php?eID=tx_nawsecuredl&u=0&g=0&t=1511718272&hash=fd986f344778c81246bf1c35d8cda3cbf74ff5e5&file=fileadmin/informatik/ag-saupe/Images/Pattern_Classification.pdfPDF

List of homework assignments (weeks are calendar weeks):
Week 16 / 2012: PDF
Week 17 / 2012: PDF
Week 18 / 2012: PDF
Week 19 / 2012: Problems 10, 11 on page 29 of the book
Week 20 / 2012: Problems 8, 9 on page 29 and problem 3, page 37 of the book. Finally, try problem 8, page 38.
                       This is a hard problem: do the easy case q = 0 first, using the equalities in exercise 1, then do the general case.
Week 21 / 2012: problems 1, 2, 3, 15, 16, 17 on pages 50, 51
Week 22 / 2012: lecture and problem session cancelled
Week 23 / 2012: problems 4, 10, 12 on page 61
Week 24 / 2012: problems 13, 14 on page 61
Week 25 / 2012: problems 7, 9 on page 66
Week 26 / 2012: problems 6, 15, 17 on page 79
Week 27 / 2012: problems 2, 3, 6, 16 on page 86
Week 28 / 2012: lecture and problem session postponed, new date t.b.a. 

Class and Problem Session, Wed 9:00 - 12:15, Z 613

Exam (oral) Tuesday, July 31, 2012, 14:00h
Exam (makeup) Tuesday, October 16, 2012, 14:00h