Information Theory

Faculty: Faculty of Engineering
Department(s): Computer Engineering
Number of Students:
Course: Information Theory
Weekly hours: Theory: 2 Exercises: 1
ECTS Credits: 5
Semester: Fall

---

Lecture Schedules:

---

Lecturer: Dr. Hiqmet Kamberaj
Room Number:
Phone Number of the lecturer:
E-mail address of the lecturer: km.ude.ubi|jarebmakh#km.ude.ubi|jarebmakh

---

Course Objectives:

The aim of the course is to give an introduction to Information Theory for students of computer science, communication theory, and statistics in senior and first-year graduate-level courses in Information Theory. In this course, students will learn about entropy and mutual information, the nature of the algebraic structure of information-theoretic, and problems in data compression and communications.

---

Available topics for Master Diploma Thesis

• Symbolic Mutual Information for Estimating Nonlinear High-order Correlated Fluctuations of Different Dynamical Systems.
• Enhancement of Transition Path Sampling using Biased Potential Energy Functions.
• Other topics are available upon request.

Skill outcomes	Necessary ( + ) Not Necessary ( –)
Written communication skills	+
Oral communication skills	+
Computer skills	+
Working in laboratory	+
Working team	+
Preparing projects	+
Knowledge of foreign language	+
Scientific and professional literature analysis	+
Problem solving skills	+
Management skills	+
Presentation skills	+

---

Course Textbooks:

1. H. Kamberaj, Lecture Notes in Information Theory, International Balkan University, 2020.

Additional Literature:

1. T. M. Cover and J. A. Thomas, Elements of Information Theory, Second Edition, John Wiley & Sons, 2006.
2. D.J.C. MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press, 2003.

---

Study Plan —- International Balkan University - Academic Calendar

Weeks	Topics
1	Introduction. Entropy. (https://youtu.be/vL4Mu4VAOZg YouTube] )
2	Relative entropy. Mutual information. Jansen's inequality. (https://lnkd.in/dJJStEZ2)
3	Data-processing theorem. Fanos's inequality. Asymptotic equipartition Property. (https://lnkd.in/d88AaayB)
4	Entropy rate of a Stochastic Process. (https://www.youtube.com/watch?v=aaaph2Em0YM)
5	Data compression. (https://www.youtube.com/watch?v=DFX06MgYOoQ)
6	Channel capacity.
7	Differential entropy.
8	Relative entropy and mutual information.
9	Gaussian Channel.
10	Transfer entropy. (Alpha, q) - Information Theoretic Measures.
11	Preparatory week.
12	Final exam week.
13	Preparatory week.
14	Preparatory week.
15	Make up 1 Exam.
In June	Make up 2 Exam.

---

Attendance

Students are obliged to attend at least 60% of lectures.

Exams

---

1. Achieved success in a course shall be evaluated through a final exam and seminar work.
2. The topic of the seminar work for each course shall be chosen in the first two weeks of the lecture. The seminar work shall be delivered in the lecture's last (10th) week. It shall consist of 7,500-10,000 words (tables and graphs are excluded).
3. The maximum number of credit points (a) for the final exam is 60%, and (b) for seminar work is 40% of the total number of points.
4. The student who has yet to pass the exam may enter the exam 2 (two) more times during the make-up exam sessions.

---

Student workload:

For calculating the Total Student Workload, we multiply the course ECTS credits with standard figure 30. (ECTS Credit: 5) x 30 = 150 hours.

Activities	Hours
Lecture hours for 10 weeks:	20
Laboratory and class exercises for 10 weeks:	10
Student Mentoring for 10 weeks:	-
Consultation for 10 weeks:	4
Exam preparations and exam hours (final, Makeup):	20
Individual reading work for 10 weeks (Reading assignments/expectations for reading and comprehension is 5 pages per hour. Example: If a book is 300 pages, the total Individual reading work for 10 weeks is 300:5 = 60 hours.	58
Homework and work practice for 10 weeks:	38
Preparation of diploma work, for 10 weeks:	-