Linear Regression

Logistic Regression For Classification

Find a linear hyperplane to separate the data, better that output the probability of class

• Linear model:

$$z(\boldsymbol{w},\boldsymbol{x}) = \boldsymbol{w} \cdot \boldsymbol{x}$$

• Link Function:

$$\hat{p}(z) = \frac{1}{1 + exp(-z)}$$

• Cross entropy loss:

$$l(y,\hat{p}) = ylog\hat{p} + ( 1 - y )log(1 - \hat{p} )$$

• Cost Function:

$$L(\boldsymbol{w},\{\boldsymbol{x}_i,y_i\}^m_{i=1} )= \sum^m_{i =1} log(1 + exp(\boldsymbol{w} \cdot \boldsymbol{x}_i)) - y_i\boldsymbol{w} \cdot \boldsymbol{x}_i$$

• Gradient:

$$\nabla_w L(\boldsymbol{w},\{\boldsymbol{x}_i,y_i\}^m_{i=1} ) = (\frac{1}{1 + exp(-\boldsymbol{w} \cdot \boldsymbol{x}_i)}- y_i)\boldsymbol{x}_i$$

The backpropagation algorithm works through the layers of deeper neural networks to calculate error gradients w.r.t to weights