• The Hong Kong University of Science and Technology

• MSBD 501

L01-1 Probability and Bayes’

Parameter learning, long-term relative freq.

先验后验 Prior before E/posterior given E

L(H|E) = P(E|H) Model~H, Data~E

Likelihood of hypothesis H given evidence E

-MLE 最大似然估计， θfix

BinomialL

LogL ,

-Bayes’ 后~前* L

-贝估计,θ random

L01-2 Info: Divergence, Entropy

Jensen’s Ineq ,

-Entropy (H below)

Condition En > En

KL Div. (Relative En), how Q(X) differs from true distribution P(X), 大于 0, P=Q 时为 0

Cross En

-Unsupervised,

Min KL ó min Cro En ó min log Q(D)

H (P, Q) = , D data

-Supervised,

Min ConCro En ó max

H (P, Q) ≈

互信息 =

L02 Linear Regression

Mean Squared Error 均方误差(确定权重 w

min MSE, let gradient = 0