I am having trouble to understand the simplification of this log-likelihood function.
$$ \tag{1} \sum[ y \log{\left(p(W_TX) \right)} + (1-y)\log(1-p(W_TX)) ] $$
$$ \tag{2} \sum [ y \frac{ \log (p(W_TX) }{ 1-p(W_T X))} + \log \left(\frac{ 1 }{ 1 + e ^{W_T X}} \right) $$
$$ \tag{3} \sum [ y w x -\log\left(1 + e ^{W_T X} \right) ] $$ Where,
$W_T$ = Weights transpose of the equation
$X$ = Input variables
$y$ is the output(0 or 1)
As there are three steps in reaching the function's simplified version, I am unable to understand the transition from 2nd to 3rd step or rather 1st to 3rd step. Also I find 1st equation to result a different answer from 3rd on solving an example. Log_likelihood Formula Image link (Kindly Refer the image for the formula, apologies for uploading an image instead of math function) The source of the image is from the following Slides http://www.cs.cmu.edu/~guestrin/Class/10701-S05/slides/LogRegress-1-24-05.pdf
Kindly help by breaking the equation even further(a basic logarithmic transformation that I am not able to figure out). Thanks in advance
EDIT
Also I find 1st equation to result a different answer from 3rd on solving an example.
Checking again with an example(given below) the equations (1) and (3) return the same values
So, they are equal.But I still don't get the logarithmic transformation from (1) to (3)
y = 0
W = [-2.9469674, 0.03791044, 0.01467374]
X = [1, 34.623659624517, 78.0246928153624]
z = WTX = -0.489455173271831
h = sigmoid(z) = 0.380021923306988
log(h) = -0.9675
log(1- h) = -0.478
(3)
y * z - log(1 + e^z)
0 * -0.489 - log(1+ e^-0.489) = -0.47824416197
(1)
y * log(h)+(1-y) * log(1-h)
0 * -0.967 + 1 * -0.478 = -0.47824416197
EDIT 2
Checking wiht y=0 is not enough as it makes the first term of your equation disapear try with y=0.2 or y=1 and you will see that equation (1) and (3) are different.
Example 2
y = 1
W = [-2.9469674, 0.03791044, 0.01467374]
X = [1, 60.1825993862098, 86.3085520954683]
z = WTX = 3.6010506763003
h = sigmoid(z) = 0.973430194479833
log(h) = -0.026929162462656
log(1- h) = -3.62797983876421
(3)
y * z - log(1 + e^z)
1 * 3.6010 - log(1+ e^-3.6010) = -0.02692916246392
(1)
y * log(h)+(1-y) * log(1-h)
1 * -0.02692 + 0 * -3.627 = -0.02692916246392
You can translate line (2) to $$ \sum_i y_i \log (\frac{1 + \exp( W_T X_i )}{1 + \exp( - W_T X_i )}) - \log (1 + \exp(W_T X_i))$$
Then you factor the numerator of the fraction in the log by $\exp(W_T X_i)$ and you get
$$ \sum_i y_i \log (\exp(W_T X )\frac{1 + \exp(- W_T X_i )}{1 + \exp( - W_T X_i )}) - \log (1 + \exp(W_T X_i))$$
$$ \sum_i y_i \log (\exp(W_T X_i )) - \log (1 + \exp(W_T X_i))$$ $$ \sum_i y_i W_T X_i - \log (1 + \exp(W_T X_i))$$