Information Theory

Faculty: Faculty of Engineering
Department(s): Computer Engineering
Course: Information Theory
Weekly hours: Theory: 2 Exercises: 1
ECTS Credits: 5
Semester: Fall

Lecture Schedules:

Lecturer: Dr. Hiqmet Kamberaj
Room Number: 409
Phone Number of the lecturer: +389 (0)23174010 (ext. 123)
E-mail address of the lecturer: km.ude.ubi|jarebmakh#km.ude.ubi|jarebmakh

Course Objectives:

The aim of the course is to give an introduction to Information Theory for students of computer science, communication theory and statistics at senior and first-year graduate-level course in Information Theory. In this course student will learn about entropy and mutual information, nature algebraic structure of information theoretic, and problems in data compression and communications.

Available topics for Master Diploma Thesis

  • Symbolic Mutual Information For Estimation of Nonlinear High-order Correlated Fluctuations of Different Dynamical Systems.
  • Enhancement of Transition Path Sampling using Biased Potential Energy Functions.
  • Other topics are available upon request.
Skill outcomes Necessary ( + ) Not Necessary ( –)
Written communication skills +
Oral communication skills +
Computer skills +
Working in laboratory +
Working team +
Preparing projects +
Knowledge of foreign language +
Scientific and professional literature analysis +
Problem solving skills +
Management skills +
Presentation skills +

Course Textbooks:

  1. T. M. Cover and J. A. Thomas, Elements of Information Theory, Second Edition, John Wiley & Sons, 2006.
  2. D.J.C. MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press, 2003.

Weekly Study Plan

Weeks Topics
1 Introduction. Entropy. Jansen's inequality, data processing theorem, Fanos's inequality.
2 Asymptotic Equipartition Property.
3 Entropy rate of a Stochastic Process.
4 Data compression.
5 Channel capacity.
6 Differential entropy.
7 Joint and conditional differential entropy.
8 Relative entropy and mutual information.
9 Gaussian Channel.
10 Rate distortion.
11 Preparatory week.
12 Final exam week.
13 Preparatory week.
14 Preparatory week.
15 Make up 1 Exam.
In June Make up 2 Exam.


Students are obliged to attend at least 60% of lectures.


  1. Achieved success in a course shall be evaluated through a final exam and seminar work.
  2. The topic of the seminar work for each course shall be chosen in the first two weeks of lecture. The seminar work shall be delivered at the last (10th) week of lecture. It shall consist from 7,500-10,000 words (tables, graphs are excluded).
  3. The maximum number of credit points (a) for final exam is 60% (b) for seminar work is 40% of the total number of points.
  4. The student who has not passed the exam may enter the exam 2 (two) more times during the make-up exam sessions.

Student workload:

Please calculate the “Total Student Work Load” and then distribute that figure to the different engagements. For calculating the Total Student Work Load we multiply the course ECTS credits with standard figure 30. (ECTS Credit: 5) x 30 = 150 hours.

Activities Hours
Lecture hours for 10 weeks: 20
Laboratory and class exercises for 10 weeks: 10
Student Mentoring for 10 weeks: -
Consultation for 10 weeks: 4
Exam preparations and exam hours (final, Makeup) : 20
Individual reading work for 10 weeks (Reading assignments/expectations for reading and comprehension is 5 pages per hour. Example: If a book 300 pages, total Individual reading work for 10 weeks 300:5 = 60 hours. 58
Homework and work practice for 10 weeks: 38
Preparation of diploma work, for 10 weeks: -
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License