MINDLAB READING

This page is created to be the (temporary?) main information page of the MIND lab reading group. Contents in our reading group can be any old or new (but interesting or important to many members) research works.

Plan

First NOTE that, although there will be a leader in a reading group, a reading group is NOT a talk. A leader might be the one who is most familiar with the selected topic, but he/she does not necessarily know everything. In order to keep a reading group survive, everyone should try quite hard to understand the selected material so that the discussions in a reading group will be fruitful.

For the first 2-3 week, Jung will take the lead. He will cover three topics which are frequently mentioned in literatures and he is most familiar with.

After that, if it successes, we hope that we can continue the process and change the leader week by week. The leader should have time at least 2 weeks to read the topic.

Date and Time

At this moment, I plan to arrange the reading group in every friday afternoon (1pm - 2pm), but let us talk for more convenient dates and times.

May 11, 2007: Mixture Models and EM algorithm

Bishop's PRML book

This topic is a nice introduction for Bayesian paradigm in machine learning. After this meeting, we should be able to answer the following questions:

• What is the Bayesian machine learning paradigm?

• The Bayes equation

• Why being Bayesian is a good idea? What are advantages of the Bayesian paradigm over the classical paradigm?

• It is intuitive, easy to understand (but might not easy to do)
• It can solve the model selection problem

• How can we train the learner in Bayesian paradigm?

• This talk illustrates one Bayesian toolkit: the EM algorithm.

Main paper

• Chris Bishop's PRML book chapter 9. ( I have one copy left; anyone who do not have this book please feel free to borrow me)

Supplementary

• David Mackay's book chapters 2 (prerequisite), 20 and 22 get it here
• Zoubin Ghahramani's slide lecture2 lecture3 (DO NOT read PCA and FA)
• Chris Bishop's slide click (DO NOT read variational inference)

Prerequisite

In order to understand how EM work, we have to understand a bit on Information entropy, the Jensen's inequality and Kullback-Leibler divergence. So please take a look at its properties before reading group: David Mackay's book chapter 2 is a very good introduction.

May 18, 2007: The Monte Carlo Methods#1

David Mackay's ground-breaking book

Last meeting illustrated one way to do Bayesian inference: the EM algorithm. However, it can give us only the maximum likelihood of the parameter which is not we exactly want in Bayesian inference. This meeting will introduce more general Bayesian toolkit, namely, the monte carlo methods.

The topics we will cover are rejection sampling, uniform sampling, importance sampling, and introduction to Markov chain.

Prerequisite

You should know what is the eigen-decomposition of a matrix.

Introduction to Monte Carlo

• David Mackay's Chapter 29

Introduction to Markov Chain

• Eric Vigoda's lecture.

May 25, 2007: The Monte Carlo Methods#2

This meeting will cover the Markov chain Monte Carlo (MCMC) method, especially the Metropolis and the Gibbs algorithms. If there is enough time, we may go further to slice sampling, simulated annealing, MCMCMC and asymmetric Metropolis-Hasting methods.

prerequisite

• You should know what is an ergodic Markov chain
• You should know what is a detailed-balance property of a Markov chain

Interesting papers (please VOTE)

Classics Works

• The Strength of Weak Learnability by Rob Schapire click.
• Adaboost paper [1]

These two papers are the origin of all boosting methods.

• A Theory of the Learnable by L. G. Valiant click

• Shannon's source coding and channel capacity theorems

See Mackay's textbook.

• PCA, SVD and Regularization

• See also Toronto's ML reading group click