Linear Regression
Logistic Regression For Classification
Find a linear hyperplane to separate the data, better that output the probability of class
- Linear model:
$$ z(\boldsymbol{w},\boldsymbol{x}) = \boldsymbol{w} \cdot \boldsymbol{x} $$
- Link Function:
$$ \hat{p}(z) = \frac{1}{1 + exp(-z)} $$
- Cross entropy loss:
$$ l(y,\hat{p}) = ylog\hat{p} + ( 1 - y )log(1 - \hat{p} ) $$
- Cost Function:
$$ L(\boldsymbol{w},\{\boldsymbol{x}_i,y_i\}^m_{i=1} )= \sum^m_{i =1} log(1 + exp(\boldsymbol{w} \cdot \boldsymbol{x}_i)) - y_i\boldsymbol{w} \cdot \boldsymbol{x}_i $$
- Gradient:
$$ \nabla_w L(\boldsymbol{w},\{\boldsymbol{x}_i,y_i\}^m_{i=1} ) = (\frac{1}{1 + exp(-\boldsymbol{w} \cdot \boldsymbol{x}_i)}- y_i)\boldsymbol{x}_i $$
The backpropagation algorithm works through the layers of deeper neural networks to calculate error gradients w.r.t to weights
